Page History

...

Overview

bbcp is a point-to-point network file copy application developed at SLAC. It was initially used by the BaBar Collaboration to transfer massive amounts of data between numerious sites of the Collaboration. The protocol has been specifically designed for WAN. In particular, it allows sending data in multiple simultaneous streams.

This document gives only some tips that are intended to help readers to begin using the tool to tranfer LCLS data. To find the complete documentation on the tool please visit the official site: http://www.slac.stanford.edu/~abh/bbcp/.

...

Installation

bbcp is easy to install. Installation basically involves placing the bbcp executable in your path on all the systems you want to use it on. All standard methods of authentication can be used: passwords and certificates. Keep in mind, though, that we're only going to support the password-based authentication based on SLAC UNIX (Kerberos) accounts.

...

Make sure bbcp executable is available in the executable search path at both sides.

...

Usage

First tests

The first test would be to transfer something on the same host. In the following example, you'll be transferring data from the special device (generating zeroes) into the null sink:

...

The "-P" option would tell the tool to print the statistics every 2 seconds. The tool may ask a user for his/her password on the "local" machine because it will be making SSH login (similar to 'ssh user@localhost') and launching the bbcp executable on the server side.

Transfering data in multiple streams

By default bbcp will use 4 simultaneous streams to send data over the network. You can increase (or decrease) that number by using the "-s" opting and specified the desired number of streams:

...

The optimal number of streams depends on many factors. For instance, there is no improvement to have more than 1 stream when transferring data over LAN. And that is exactly the situation which is seen in the last example. For WAN it often helps to increas the number of streams above the default number of 4. In our experience we could get the best (in a specific setup of transferring data between SLAC and IN2P3/Lyon/France) with 32 streams. In that case the transfer speed was nearing 38 MBytes/sec.

Increasing the TCP window size

Another parameter worth to consider is so called TCP window size. By default bbcp would set it to 64 KBytes. The corresponding option for bbcp is called "-w". In the following example, the desired window size is set to 2 MBytes:

...

The best performance would be achieved by setting optimal values for both TCP window size and a number of simultaneous streams. One would need to experiment with these paramaters for a specific network/system setup.

Using bbcp to transfer LCLS data

Please use one of the following two data mover nodes pool to transfer data to your home institution:

psimport.slac.stanford.edu
psexport.slac.stanford.edu

...

% bbcp -P 15 -s 32 -w 2M psimportpsexport.slac.stanford.edu:<srcfilename> <dstfilename>

...

Troubleshooting

No bbcp executable available at a remote site

...

% bbcp -z -P 2 user@SomeRemoteHost:/dev/zero /dev/null
user@SomeRemoteHost's password:
bbcp: Creating /dev/null/zero
bbcp: At 090414 02:59:48 copy 0% complete; 5461.3 KB/s
bbcp: At 090414 02:59:50 copy 0% complete; 13639.7 KB/s
..

...

TCP Tuneup

One cane find useful tips on how to tun up TCP parameters on machines involved into the data transfer to get the highest data transfer rate with bbcp:

...