SuperComputing 2010
Moving Towards Terabit/Sec Transfers

Detailed report for Caltech


Michael Thomas (

I've been running short tests all day to gauge the performance of Caltech <-> SC transfers.  Here's what I've found:

We can do Storage (hadoop) -> disk at ~2.4Gbps.  This rate is both the rate that the disks at SC can write, and about the rate that we can get data out of Hadoop.  As was the case last year, the rates taper off slightly over time, down to 1.5-2Gbps.  Running multiple transfers (4 hosts at Caltech -> 4 hosts at SC) seems to also negatively affect the transfer rates, though I can't give a quantitative answer as to how much until I run more tests.

SC Disk -> Caltech Storage (hadoop) is not quite as good as we saw last year.  I can consistently get an average rate of ~1.1Gbps writing into Hadoop.  Changing the number of streams and reader/writer threads has not helped much.  4 streams/threads works better than 12, probably due to the disk read contention on the source.

The older 2.6.30-9UL1 kernel likes to cause the dual-10GbE adapters on our latest Tier2 FDT server to lock up, but the latest 2.6.36-UL4 appears to be much more stable.