Page1    Page125                                                                        Page127  Page177

Appendix 23. HEP Demonstration at SC07 of High

Throughput Data Transfer

Submitted by Azher Amin(azher@hep.caltech.edu), Iosif Legrand

(Iosif.Legrand@cern.ch), Artur Barczyk(Artur.Barczyk@cern.ch), Julian Bunn

(Julian.Bunn@caltech.edu, Harvey Newman(Harvey.Newman@cern.ch), Kamran

Soomro (kamran@hep.caltech.edu), Michael Thomas (thomas@hep.caltech.edu)

Caltech

January 2008

In order to help meet its present and future needs for reliable, high performance networks, our

community has engaged in leading edge network R&D over the last few years. In 2005-6, we

made substantial progress in the development and use of networks in the 10-100 Gbps range over

multiple long distance links and in the production use of data transfers among grid clusters at

speeds up to and beyond the 10 Gbps range (storage to storage).

At SC07 in Reno, Nevada the HEP team, led by Caltech, demonstrated sustained data transfer

rates of 80+ Gbps for hours between a single rack of servers at the show floor (of the type being

used at the LHC Tier1 and Tier2 centers), and similar equipment located at Caltech, CERN,

University of Michigan, KISTI and other partner institutes.

The record-setting demonstration was made possible through the use of seven 7 Gbps links to

SC07 provided by SCInet, CENIC, National Lambda Rail, and Internet2, together with a fully

populated Cisco 6509E Cisco switch, 10 gigabit Ethernet network interfaces provided by Intel

and Myricom, and a fiber channel disk array provided Data Direct Networks equipped with 4

Gbps host bus adapters from QLogic. The server equipment consisted of 36 widely available

Supermicro systems using dual quad-core Intel Xeon processors, and Western Digital SATA

disks. The rack layout is show in Figure 77. This setup is equivalent to a "mini" Tier2 center in

terms of computing and storage, but with greater network capability, as shown in Figure 78. The

server configuration included approximately 62 Quad Core CPUs, 164 Terabytes of disk, 44 10G

Ethernet switch ports in a single Cisco 6509E switch and a like number of 10 Gbps interfaces on

servers along with many gigabit Ethernet ports. The layout of the server rack and disk arrays is

show below.

126