To satisfy the demands of data intensive grid applications it is necessary to move to far more synergetic relationships between applications and networks. Currently, even the most complex scientific applications are simply passive users of the existing network infrastructure. The main objective of the VINCI project is to enable users' applications, at the LHC and in other fields of data-intensive science, to effectively use and coordinate shared, hybrid network resources, to correlate them with available processing power in order to dynamically generate optimized workflows in complex distributed system.

VINCI dynamically estimates and monitors the achievable performance along a set of candidate (shared or dedicated) network paths, and correlates these results with the CPU power and storage available at various sites, to generate optimized workflows for grid tasks. The VINCI system is implemented as a dynamic set of collaborating Agents in the MonALISA framework [1], exploiting MonALISA's ability to access and analyze in-depth monitoring information from a large number of network links and grid sites in real-time.

This should significantly improve the overall performance and reduce the effective costs of global-scale grids.

We use the unique infrastructure offered by the UltraLight [2] Project to develop these network services, to test them and to integrate them in real data processing activities.

Our strategy in integrating data intensive applications with geographically distributed complex services that may join and existing ones leave is the use of a multi-agent systems. The VINCI system is implemented as a set of collaborating agent based services in the MonALISA framework. Agents will act on behalf of applications or users describing their environment and requirements, locating services, agreeing information, and receiving feedback from services and presenting results. Agents will be required to engage in interactions, to negotiate, and to make pro-active run-time decisions while responding to changes in the environment. In particular, agents will need to self organize, and dynamically collaborate for effective decisions.

The MonALISA framework has been designed as a set of autonomous agent-based dynamic services that collect and analyze real-time information from a wide variety of sources (grid nodes, network routers and switches, running jobs, video devices, etc. ). MonALISA's multi-threaded, self-describing agents collaborate to process and analyze the information they gather in a distributed way, enabling them to perform a wide range of monitoring and/or control tasks. Higher level functions in the agents provide support for automated control decisions and global optimization of workflows in complex grid systems, and in other large-scale distributed applications. Agents in the MonALISA servers self-organize and collaborate with each other to manage access to distributed resources, to make effective decisions in planning workflow, to respond to problems that affect multiple sites, or to carry out other globally-distributed tasks.

A schematic view of the flow diagram and the communication between major services is presented in Figure 1. For the application that use the Network Services APIs, and End User Agent is used to provide authorization, discovery of services and secure communication with the entire system of services and at the same time to provide complete local monitoring. The Scheduling system orchestrate the request coming from different applications using the network services APIs as well as the information collected by the monitoring system in how the network is used ( traffic, topology errors) by other applications that do not use the directly the APIs capable to drive high performance network requests.

Figure1. Schematic view of the main services in the VINCI Architecture

The Global Scheduler is implemented as set of collaborating agents running on each MonALISA service. Each scheduling agent is using priority queues based on the local policies. These agents are capable to negotiate for an end to end connection using a set of cost functions. A lease mechanism should be implanted for each offer and agent is making to its peers to avoid multiple booking of the same segment for request that may appear very close in time. A two phase commit should be used in allocating a remote path and the agents supervising any established path must periodically renew the lease for the connection to all the agents involved. In this way a path can be released when certain components may fail or the application that requests it is not anymore active. When network errors are detected, the supervising agents must release the other segments involved in a global path, and inform the scheduler to try to provide an alternative path as soon as possible. The time to generate an alternative path should remain less that the typical TCP timeout for connections and in this way ongoing transfers will continue without interruptions.

The main functional units necessary for building Integrated Network Services are:

Monitoring.We use the MonALISA system, which is already being integrated into several GRID projects and R&E networks to keep a real-time picture of the end-system states (CPU load, memory, network traffic in and out, I/O load, page fault rates, etc.) and network performance on every link. In addition it is used to monitor jobs execution, the way it evolves through different states and provide information about the IO activity performed by each task and the data set used. The monitoring system capabilities are:

(1) Provide the current state of the network links connecting a VO. (2) Measure/estimate bandwidth available along a given path. (3) Monitor which networks are congested or oversubscribed. (4) Maintain historical usage of links and problems. (5) Extend monitoring to include the endhosts (most network problems are actually end-host or LAN issues)

End User Agent. It will provide local detection for hardware and software, complete monitoring of the system and end to end performance measurements. It will also provide service discovery and will act as an active listener for all the events related with the requests generated by its local applications. The End User Agent implements the APIs the applications may use to generate requests and to interact with the system. This is done using secure communication channels. It will receive time estimations to complete the request and will get continuous updates on the progress. When it is possible, will help to provide correct network configuration to the user system or to adjust it based on specific type of connection the service system will setup. This will be critical for quickly diagnosing the correct location of any performance problem within the network fabric.

We will extent the LISA [1] agent to incorporate the specific functionalities for network services. LISA is currently being used with VRVS to provide dynamically the best reflector to be used from the user perfective and it also provides a global load balancing.

Authentication, Authorization and Accounting for Users. We will use external AAA services supported by different Virtual Organizations. Loadable plug-in interface modules to support different protocols and services will provide the necessary flexibility to work with different system. For our own development and testing we may implement simple a AAA service.

Service Registration and Discovery. A reliable mechanism to dynamically register services, their dynamic sets of attributes, is an essential part in any Service Oriented System. We will continue to use the JINI Lookup Discovery Service [4].

Topology Discovery. Specialized agents will be used to discover the connection topology for each service, to keep a dynamic map of how they are allocated and used, as well as to get information on traffic for each segment. These agents will provide the basic information to the scheduling system. This service will use the information from MPLS/GMPLS agents were the infrastructure provide this functionality.

Examples of targeted capabilities for Topology Discovery are: (1) Determine which options exist between two locations in the network (2) List components in the path that are manageable. (3) Given two replicas of a data source, discover (in conjunction with monitoring) the estimated bandwidth and reliability of each to a given destination. (4) Locate network resources and services which have agreements with a given VO.

Services to create dynamically an optical path. The current prototype implemented in the MonALISA framework can be used to generate on demand an end to end optical path when optical switches [5] are used. This functionality is implemented as a set of distributed agents that cooperate in generating a end to end path or a multicast tree based on their own connectivity maps. The distributed set of the MonALISA agents are used to monitor and control the system. They use the discovery mechanism to find each other and they communicate reliable between them using the Proxy System [1]. The agent system is used to create a global path or tree as it  knows the state of each link,  inter-site  connections and the cross connections.  The routing algorithm provides global optimization and can be extended to handle priorities and pre reservations. Each created connection is continuously monitored (optical power on each link) and in case an error is detected on one segment, it automatically tries to establish and alternative path. In Figure 2 we present the mechanism used by the MonALISA agents to generate on demand an end to end optical path.

Figure2. The agent system used to create on demand an end to end optical path. A LISA agent runs on each system, and it provides access control, complete local monitoring and can dynamically reconfigure the system. MonALISA services are used to monitor and control optical switches.

The system is integrated in a reliable and secure way with the end user applications and provides simple shell-like commands to map global connections and to create an optical path / tree on demand for any data transfer application.. When an optical path is required by an user or application the agent system establish it in less than 1 second, independent of how many switches are involved and were they are located.. The entire system is monitored in real-time and it presents the state and transitions for each port and link .

Services for managing MPLS /GMPLS requests. We need to develop agents capable to interact with MPLS and GMPLS controllers [6], to provide in near real-time topology maps for other services and to generate connection requests. These agents will act more as MPLS/ GMPLS clients. They should provide the functionality to continuously monitor and supervise the connections they created.

The MPLS agents can be used together with the optical path agents to create an end to end network configuration. For networks where GMPLS is supported, the MonALISA agents only need to interface with the head-end devices, not the rest of the network. The GMPLS standard protocols topology discovery (LMP), routing (OSPF) and provisioning (RSVP) and allow interoperability across domains. For example, special routing can be done with MonaLISA agents if not included in OSPF, but the topology discovery and provisioning can be done with GMPLS. In another example, a network may not give you access to topology and will only let provide him a request with parameters and it will do

the routing itself. In this class of services we may need to also provide agents capable to configure routers or switches using SNMP or TL1 or interface with local services that provide this functionality. In the near future we should provide agents capable to interact with the DRAGON [7] system.

Scheduler and connections planning. This is a key part in developing the functionality we described for the network services. The scheduler should provide a near optimal use of resources at a global level, by combining different requests generated by clients.

We propose a 'market model' scheduler based on a distributed (agent based) implementation. A distributed implementation of the scheduler is significantly more complex than a centralized system, but provides major advantages:

• is not a single point of failure

• provides complete autonomy to each provider of resources who can implement its own policy

• allows to dynamically add or remove sites from the system

The scheduler will combine the information from the monitoring system and will analyze the requests made by users using the dedicated APIs. The scheduler implements several priority queues for describing the local infrastructure, the background traffic and the request made by applications for high throughput traffic. It should provide global optimization in how hybrid network resources are allocated. The optimization of resource allocations is in general a difficult NP complete problem.

The Global Scheduler is implemented as set of collaborating agents running on each MonALISA service. Each scheduling agent is using priority queues based on the local policies. These agents are capable to negotiate for an end to end connection using a set of cost functions. A lease mechanism should be implanted for each offer and agent is making to its peers to avoid multiple booking of the same segment for request that may appear very close in time. A two phase commit should be used in allocating a remote path and the agents supervising any established path must periodically renew the lease for the connection to all the agents involved. In this way a path can be released when certain components may fail or the application that requests it is not anymore active. When network errors are detected, the supervising agents must release the other segments involved in a global path, and inform the scheduler to try to provide an alternative path as soon as possible. The time to generate an alternative path should remain less that the typical TCP timeout for connections and in this way ongoing transfers will continue without interruptions.

Learning and Prediction. It is very difficult if not impossible to assume that we could predict all possible events in a complex environment like the GRID, and encode all the knowledge about those events in advance. Learning is the only practical approach in which agents can acquire what it needs to know about their environments. They should be able to learn as they adapt to their external environment, so that their performance improves over time. We approach this multi-agent learning task from two different levels, the local level of individual learning agents, and the global level of inter-agent operations. We need to ensure that each agent can be optimized from local knowledge, while the globally monitoring mechanism acts as a driving force to evolve the agents collectively based on the global pooled knowledge. The end goal of this learning scheme is to produce a new generation of agents that benefit from the learning experiences of individual parent agents and the collective learning experiences of all previous generations.

A schematic view of the system functionality is presented in Figure 3. Resources at each site are monitor and controlled by MonALISA services. Each MonALISA service hosts a set of Agents to perform Topology Discovery manage MPLS/GMPLS controllers where they are available, to create optical paths on pure optical switches and the agents involved in scheduling and learning. The VINCI system is optimizing the path allocation using as much as possible Layer 1 or Layer 2 segments. The applications using the VINCI APIs will drive the way network resources are allocated by specifying the type of services they need. The system will provide for all the requests an estimation of the time to complete and will continuously update the applications with a progress report. At the global level, the optimization on how resources are shared and the dynamic allocation of different path connections will be done by a Distributed Scheduler. The Scheduler will use priority queues to correlate near concurrent requests and pre-reservations, to optimize the way it allocates shared resources in hybrid networks. In case of errors on a segment used by an application, the system will try to find an alternative path and to notify the application. The MonALISA services provide also support for interaction with end user agents and to continuously notify applications about the progress in data transfers. Based on the monitoring information the system is aware of the background traffic performed by standard application on Level3 or 2 and it can dynamically decide to select alternative paths for global optimization.

Figure 3 . A schematic view of the functionality to provide dynamically an efficient end to end path to data intensive applications. The VINCI system is optimizing the path allocation using as much as possible Layer 1 or Layer 2 segments.

Our strategy is to use as much as possible existing tools and protocols like GMPLS, MPLS or DRAGON, to manage the network infrastructure. The system aims to use in a near optimal way hybrid networks, to dynamically adjust to different traffic patterns our applications will create, and to provide simple, but powerful APIs for data intensive tasks. The learning and prediction part is a key element in developing procedures to allocate shared, hybrid network resources in optimal way at the global level

We will develop services to analyze the flows in major segments and to generate correlations and predictions. This information will be used to further optimize the resource allocations. Learning algorithms (like Self Organizing Neural Networks [3]) will be used to evaluate the traffic created by other applications to identify major patters and to dynamically setup effective connectivity maps.