Transferring large files from CERN to SLAC (or SLAC to CERN, etc.)
See the relevant discussion on hypernews
...
| Code Block |
|---|
~/xrootd/bbcp -z -T /usr/local/bin/bbcp -s 64 -P 2 -f grenier@noric09.slac.stanford.edu:notes.txt notes.txt |
...
Some notes from Wei:
...
1. bbcp uses ssh. So one can avoid typing password by adding your CERN ssh public key to SLAC's authorized key file (~/.ssh/authorized_keys)
2. I am using 64 streams, which is the maximum.
3. -P 2 is not necessary unless you want to see the process bar.
...
Another note from Gary Buhrmaster, regarding the number of streams:
...
The number of "optimal" streams varies, and is influenced by the size of the tcp buffers. More streams can actually hurt performance.
There are some tuning suggestion available at: http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm
Usually the default number of streams of ~4 is reasonable, but you want to increase the window size(i.e. eliminate the '-s 64' and instead add in '-w 2M').
Copying into castor
Once you have the files at CERN (probably on pcphuat27), you can also put them to/from CASTOR.
https://twiki.cern.ch/twiki/bin/view/Atlas/CastorPools
For instance, to copy a whole directory on pcphuat27 into my castor area (you need an area with write permission!):
| Code Block |
|---|
xrdcp /u1/ahaas/Zmumu/ root://castoratlas//castor/cern.ch/atlas/atlascerngroupdisk/proj-sit/pileup/ahaas/
|