To
satisfy the demands of data intensive grid applications it is
necessary to move to far more synergetic relationships between
applications and networks. Currently, even the
most complex scientific applications are simply passive users
of the existing network infrastructure. The main objective of
the VINCI project is to enable users' applications, at the LHC and
in other fields of data-intensive science, to effectively use and
coordinate shared, hybrid network resources, to
correlate them with available processing power in order to
dynamically generate optimized workflows in complex distributed
system.
VINCI
dynamically estimates and monitors the achievable performance along a
set of candidate (shared or dedicated) network paths, and correlates
these results with the CPU power and storage available at various
sites, to generate optimized workflows for grid tasks. The VINCI
system is implemented as a dynamic set of collaborating Agents in
the MonALISA framework [1], exploiting MonALISA's ability to access
and analyze in-depth monitoring information from a large number of
network links and grid sites in real-time.
This
should significantly improve the overall performance and reduce the
effective costs of global-scale grids.
We use the unique infrastructure offered by the UltraLight [2]
Project to develop these network services, to test them and to
integrate them in real data processing activities.
2. A Multi Agent Architecture
Our
strategy in integrating data intensive applications with
geographically distributed complex
services that may join and existing ones leave is the use of a
multi-agent systems. The
VINCI system is implemented as a set of collaborating agent based
services in the MonALISA framework. Agents will act on behalf
of applications or users describing their environment and
requirements, locating services, agreeing information, and receiving
feedback from services and presenting results. Agents will be
required to engage in interactions, to negotiate, and to make
pro-active run-time decisions while
responding to changes in the environment. In particular,
agents will need to self organize,
and dynamically collaborate for effective decisions.
The MonALISA framework has been designed as a set of autonomous
agent-based dynamic services that collect and analyze real-time
information from a wide variety of sources (grid nodes, network
routers and switches, running jobs, video devices, etc. ). MonALISA's
multi-threaded, self-describing agents collaborate to process and
analyze the information they gather in a distributed way, enabling
them to perform a wide range of monitoring and/or control tasks.
Higher level functions in the agents provide support for automated
control decisions and global optimization of workflows in complex
grid systems, and in other large-scale distributed applications.
Agents in the MonALISA servers self-organize and collaborate with
each other to manage access to distributed resources, to make
effective decisions in planning workflow, to respond to problems that
affect multiple sites, or to carry out other globally-distributed
tasks.
The
main functional units necessary for building Integrated Network
Services are:
Monitoring.We
use the MonALISA system, which is already being integrated into
several GRID projects and R&E networks to keep a real-time
picture of the end-system states (CPU load, memory, network traffic
in and out, I/O load, page fault rates, etc.) and network performance
on every link. In addition it is used to monitor jobs execution, the
way it evolves through different states and provide information about
the IO activity performed by each task and the data set used. The
monitoring system capabilities are:
(1) Provide the current state of the network links connecting a VO.
(2) Measure/estimate bandwidth available along a given path. (3)
Monitor which networks are congested or oversubscribed. (4) Maintain
historical usage of links and problems. (5) Extend monitoring to
include the endhosts (most network problems are actually
end-host or LAN issues)
End
User Agent. It will provide local detection for hardware and
software, complete monitoring of the system and end to end
performance measurements. It will also provide service discovery and
will act as an active listener for all the events related with the
requests generated by its local applications. The End User Agent
implements the APIs the applications may use to generate requests and
to interact with the system. This is done using secure communication
channels. It will receive time estimations to complete the request
and will get continuous updates on the progress. When it is possible,
will help to provide correct network configuration to the user system
or to adjust it based on specific type of connection the service
system will setup. This will be critical for quickly diagnosing the
correct location of any performance problem within the network
fabric.
We will extent the LISA [1] agent to incorporate the specific
functionalities for network services. LISA is currently being used
with VRVS to provide dynamically the best reflector to be used from
the user perfective and it also provides a global load balancing.
Authentication,
Authorization and Accounting for Users. We will use external AAA
services supported by different Virtual Organizations. Loadable
plug-in interface modules to support different protocols and services
will provide the necessary flexibility to work with different system.
For our own development and testing we may implement simple a AAA
service.
Service
Registration and Discovery. A reliable mechanism to dynamically
register services, their dynamic sets of attributes, is an essential
part in any Service Oriented System. We will continue to use the JINI
Lookup Discovery Service [4].
Topology
Discovery. Specialized agents will be used to discover the
connection topology for each service, to keep a dynamic map of how
they are allocated and used, as well as to get information on traffic
for each segment. These agents will provide the basic information to
the scheduling system. This service will use the information from
MPLS/GMPLS agents were the infrastructure provide this functionality.
Examples of targeted capabilities for Topology Discovery are: (1)
Determine which options exist between two locations in the network
(2) List components in the path that are manageable. (3) Given
two replicas of a data source, discover (in conjunction with
monitoring) the estimated bandwidth and reliability of each to a
given destination. (4) Locate network resources and services which
have agreements with a given VO.
Services
to create dynamically an optical path. The current prototype
implemented in the MonALISA framework can be used to generate on
demand an end to end optical path when optical switches [5] are used.
This functionality is implemented as a set of distributed agents that
cooperate in generating a end to end path or a multicast tree based
on their own connectivity maps. The distributed set of the
MonALISA agents are used to monitor and control the system. They use
the discovery mechanism to find each other and they communicate
reliable between them using the Proxy System [1]. The agent system is
used to create a global path or tree as it knows the state of
each link, inter-site connections and the cross
connections. The routing algorithm provides global optimization
and can be extended to handle priorities and pre reservations. Each
created connection is continuously monitored (optical power on each
link) and in case an error is detected on one segment, it
automatically tries to establish and alternative path. In Figure 2 we
present the mechanism used by the MonALISA agents to generate on
demand an end to end optical path.
Figure2.
The agent system used to create on demand an end to end optical path.
A LISA agent runs on each system, and it provides access control,
complete local monitoring and can dynamically reconfigure the system.
MonALISA services are used to monitor and control optical switches.
The
system is integrated in a reliable and secure way with the end user
applications and provides simple shell-like commands to map
global connections and to create an optical path / tree on demand for
any data transfer application.. When an optical path is required by
an user or application the agent system establish it in less than 1
second, independent of how many switches are involved and were they
are located.. The entire system is monitored in real-time and it
presents the state and transitions for each port and link .
Services
for managing MPLS /GMPLS requests. We need to develop agents
capable to interact with MPLS and GMPLS controllers [6], to provide
in near real-time topology maps for other services and to generate
connection requests. These agents will act more as MPLS/ GMPLS
clients. They should provide the functionality to continuously
monitor and supervise the connections they created.
The
MPLS agents can be used together with the optical path agents to
create an end to end network configuration. For networks where GMPLS
is supported, the MonALISA agents only need to interface with the
head-end devices, not the rest of the network. The GMPLS standard
protocols topology discovery (LMP), routing (OSPF) and provisioning
(RSVP) and allow interoperability across domains. For example,
special routing can be done with MonaLISA agents if not included in
OSPF, but the topology discovery and provisioning can be done with
GMPLS. In another example, a network may not give you access to
topology and will only let provide him a request with parameters and
it will do
the
routing itself. In this class of services we may need to also provide
agents capable to configure routers or switches using SNMP or TL1 or
interface with local services that provide this functionality. In the
near future we should provide agents capable to interact with the
DRAGON [7] system.
Scheduler
and connections planning. This is a key part in developing the
functionality we described for the network services. The scheduler
should provide a near optimal use of resources at a global level, by
combining different requests generated by clients.
We propose a 'market model' scheduler based on a distributed
(agent based) implementation. A distributed implementation of the
scheduler is significantly more complex than a centralized system,
but provides major advantages:
is not a single point of failure
provides
complete autonomy to each provider of resources who can implement
its own policy
allows
to dynamically add or remove sites from the system
The scheduler will combine the information from the monitoring system
and will analyze the requests made by users using the dedicated APIs.
The scheduler implements several priority queues for describing the
local infrastructure, the background traffic and the request made by
applications for high throughput traffic. It should provide global
optimization in how hybrid network resources are allocated. The
optimization of resource allocations is in general a difficult NP
complete problem.
The Global Scheduler is implemented as set of collaborating agents
running on each MonALISA service. Each scheduling agent is using
priority queues based on the local policies. These agents are capable
to negotiate for an end to end connection using a set of cost
functions. A lease mechanism should be implanted for each offer and
agent is making to its peers to avoid multiple booking of the same
segment for request that may appear very close in time. A two phase
commit should be used in allocating a remote path and the agents
supervising any established path must periodically renew the lease
for the connection to all the agents involved. In this way a path can
be released when certain components may fail or the application that
requests it is not anymore active. When network errors are detected,
the supervising agents must release the other segments involved in a
global path, and inform the scheduler to try to provide an
alternative path as soon as possible. The time to generate an
alternative path should remain less that the typical TCP timeout for
connections and in this way ongoing transfers will continue without
interruptions.
Learning
and Prediction. It is very difficult if not impossible to assume
that we could predict all possible events in a complex environment
like the GRID, and encode all the knowledge about those events in
advance. Learning is the only practical approach in which agents can
acquire what it needs to know about their environments. They should
be able to learn as they adapt to their external environment, so that
their performance improves over time. We approach this multi-agent
learning task from two different levels, the local level of
individual learning agents, and the global level of inter-agent
operations. We need to ensure that each agent can be optimized from
local knowledge, while the globally monitoring mechanism acts as a
driving force to evolve the agents collectively based on the
global pooled knowledge. The end goal of this learning scheme is to
produce a new generation of agents that benefit from the learning
experiences of individual parent agents and the collective
learning experiences of all previous generations.
5. The System Functionality
A
schematic view of the system functionality is presented in Figure 3.
Resources at each site are monitor and controlled by MonALISA
services. Each MonALISA service hosts a set of Agents to perform
Topology Discovery manage MPLS/GMPLS controllers where they are
available, to create optical paths on pure optical switches and the
agents involved in scheduling and learning. The VINCI system is
optimizing the path allocation using as much as possible Layer 1 or
Layer 2 segments. The applications using the VINCI APIs will drive
the way network resources are allocated by specifying the type of
services they need. The system will provide for all the requests an
estimation of the time to complete and will continuously update the
applications with a progress report. At the global level, the
optimization on how resources are shared and the dynamic allocation
of different path connections will be done by a Distributed
Scheduler. The Scheduler will use priority queues to correlate near
concurrent requests and pre-reservations, to optimize the way it
allocates shared resources in hybrid networks. In case of errors on a
segment used by an application, the system will try to find an
alternative path and to notify the application. The MonALISA services
provide also support for interaction with end user agents and to
continuously notify applications about the progress in data
transfers. Based on the monitoring information the system is aware of
the background traffic performed by standard application on Level3 or
2 and it can dynamically decide to select alternative paths for
global optimization.
Figure
3 . A schematic view of the functionality to provide dynamically an
efficient end to end path to data intensive applications. The VINCI
system is optimizing the path allocation using as much as possible
Layer 1 or Layer 2 segments.
Our strategy is to use as much as possible existing tools and
protocols like GMPLS, MPLS or DRAGON, to manage the network
infrastructure. The system aims to use in a near optimal way hybrid
networks, to dynamically adjust to different traffic patterns our
applications will create, and to provide simple, but powerful APIs
for data intensive tasks. The learning and prediction part is a key
element in developing procedures to allocate shared, hybrid network
resources in optimal way at the global level
We will develop services to analyze the flows in major segments and
to generate correlations and predictions. This information will be
used to further optimize the resource allocations. Learning
algorithms (like Self Organizing Neural Networks [3]) will be used to
evaluate the traffic created by other applications to identify major
patters and to dynamically setup effective connectivity maps.