Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin

Transferring large files from CERN to SLAC (or SLAC to CERN, etc.)

See the relevant discussion on hypernews

bbcp is much faster than scp (~15 MB/s compare to ~0.3 MB/s), but bbcp is not installed at CERN.
Copy it from SLAC (/usr/local/bin/bbcp) to your own CERN area (in my xrootd directory in the following examples):

Code Block
scp /usr/local/bin/bbcp

Note, you can find the Mac binary at /afs/

To transfer a single file from CERN to SLAC:

Code Block
~/xrootd/bbcp  -s 64 -P 2 -f notes.txt

To transfer a whole directory:

Code Block
~/xrootd/bbcp  -s 64 -P 2 -f -r data

To transfer from SLAC to CERN:

Code Block
~/xrootd/bbcp -z -T /usr/local/bin/bbcp -s 64 -P 2 -f notes.txt
Some notes from Wei:

1. bbcp uses ssh. So one can avoid typing password by adding your CERN ssh public key to SLAC's authorized key file (~/.ssh/authorized_keys)

2. I am using 64 streams, which is the maximum.

3. -P 2 is not necessary unless you want to see the process bar.

Another note from Gary Buhrmaster, regarding the number of streams:

The number of "optimal" streams varies, and is influenced by the size of the tcp buffers. More streams can actually hurt performance.

There are some tuning suggestion available at:

Usually the default number of streams of ~4 is reasonable, but you want to increase the window size(i.e. eliminate the '-s 64' and instead add in '-w 2M').

Copying into castor

Once you have the files at CERN (probably on pcphuat27), you can also put them to/from CASTOR.

For instance, to copy a whole directory on pcphuat27 into my castor area (you need an area with write permission!):

Code Block
 xrdcp /u1/ahaas/Zmumu/ root://castoratlas//castor/