Page1    Page126                                                                        Page128  Page177

Figure 77: Rack layout at the Caltech Booth, SC07

This demonstration, following the team's record memory-to-memory transfer rate of 151 Gbps

using 22 10-Gbps links last year at SuperComputing 2005, and 17 Gbps sustained throughput

over one fiber link in SuperComputing 2006, represented a milestone in providing practical,

widely deployable applications. These applications exploit advances in state-of-the-art TCP-based

data transport, servers (Intel Quad Xeon systems) and the Linux kernel over a period of one year.

One of the key advances in this demonstration was the use of the now mature "Fast Data

Transport" (FDT) tool, a Java application developed at Caltech that runs on all major platforms.

FDT uses the Java NIO libraries to achieve stable disk reads and writes coordinated with smooth

data flow across the long-range network. It can stream sets of files, so that a large data set

composed of thousands of files (typical in high-energy physics applications) can be sent or

received at full speed, without the network transfer restarting between files.

By using FDT in conjunction with FAST TCP, developed by Steven Low of Caltech's computer

science department, together with an optimized Linux kernel provided by S. McKee of Michigan

known as the "UltraLight kernel," the team reached unprecedented throughput levels, limited only

by the speeds of the disks. This represents a clear advance in basic data transport capability over

wide-area networks - 20 Gbps could be sustained in a few streams memory-to-memory over long

distances very stably for many hours, using a single 10-Gigabit Ethernet link very close to full

capacity in full duplex. FDT is currently being deployed across the Ultralight testbed to be used

as a transfer tool for both "simple" storage systems involving nodes with local disks, and more

complex storage systems such as dCache.

The integration of FDT with dCache, one of the major storage systems used by ATLAS and CMS

will allow for much higher speed transfers of multi-Terabyte datasets as part of the experiments'

major data distribution, processing and analysis operations at Tier1 and Tier2 centers. FDT will

be able to transfer many small or large files in a pipeline mode, avoiding connection and transfer

startup times that are common with current tools. Initial prototypes of this integration have

shown very promising results, reaching 3-5 Gbps speeds over long distances. Further work on an

optimized integration approach, that should yield greater stability and higher long-term

throughput, without substantial changes to the dCache software, is currently underway.

127