You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

1 Overview

BBCP is a point-to-point network file copy application developed at SLAC. It was initially used by the BaBar Collaboration to transfer massive amounts of data between numerious sites of the Collaboration. The protocol has been specifically designed for WAN. In particular, it allows sending data in multiple simultaneous streams.

This document gives only some tips that are intended to help readers to begin using the tool. To find the complete documentation on the tool please visit the official site: http://www.slac.stanford.edu/~abh/bbcp/.

2 Installation

BBCP is easy to install. Installation basically involves placing the bbcp executable in your path on all the systems you want to use it on. All standard methods of authentication can be used: passwords and certificates. Though, keep in mind that we're only going to support the password-based authentication based on SLAC UNIX (Kerberos) accounts.

Versions of BBCP are available for most major flavors of UNIX, including Linux and Solaris.

  • NOTE: BBCP is NOT available for Windows. Should this be a problem consider other methods of data exportation which are presented in the parent document.

Since BBCP is a peer-to-peer application then no permenently running server process is required - you just invoke the bbcp executabe on a source machine and in response another bbcp executable is started on the target machine. A more sophisticated scenario involves using the three-party scheme: the source and target machines do not need to be the same (third) machine that you initiate the file transfer from. Please, consult the official BBCP documentation for further detail on that subject. At the mean time, all examples presented in the current document will be based on a simple two-party mode.

You can either download a pre-buit version of BBCP or build it yourself from source. See building instruction at http://www.slac.stanford.edu/~abh/bbcp/.

Make sure bbcp executable is available in an binary search path at both sides.

3 Usage

First tests

The first test would be to transfer something on the same host. In the following example, you'll be transferring data from the special device (generating zeroes) into the null sink:

%  bbcp -P 2 localhost:/dev/zero /dev/null
user@localhost's password:
bbcp: Creating /dev/null/zero
bbcp: At 090413 17:37:56 copy 0% complete; 811858.2 KB/s
bbcp: At 090413 17:37:58 copy 0% complete; 819277.7 KB/s
bbcp: At 090413 17:38:00 copy 0% complete; 827783.2 KB/s
..

The "-P" option would tell the tool to print the statistics every 2 seconds. The tool may ask you for a password on the "local" machine because it will be making SSH login and launching the bbcp executable on the server side. By defdault BBCP will use 4 simultaneous streams to send data over the network. You can increase that number by using the "-s" opting and specified the desired number of streams. In the next example, this is also combined with

Troubleshooting

No bbcp executable available at a remote site

If the bbcp executable isn't available in the binary search path of user 'user' at remote host 'SomeRemoteHost' then this will be reported as follows:

% bbcp -P 2 user@SomeRemoteHost:/dev/zero /dev/null
bbcp: Command not found.
bbcp: Unable to allocate more than 0 of 4 data streams.

A firewal blocking ports

Sometimes a firewall (either at SLAC or at a remote site) may block certain (ranges of) ports. Unfortunatelly, it's hard to diagnose this problem directly. BBCP would report this typically as follows:

% bbcp -P 2 user@SomeRemoteHost:/dev/zero /dev/null
bbcp: Accept timed out on port 5031
bbcp: Unable to allocate more than 0 of 4 data streams.
Killed by signal 15.

Note, that the actual port number may vary in each specific case because bbcp would attempt to dynamically allocate the next available (at the remote server machine) port. A workaround for this problem is to use option "-z" which would use reverse connection protocol (i.e., sink to source):

% bbcp -z -P 2 user@SomeRemoteHost:/dev/zero /dev/null
user@SomeRemoteHost's password:
bbcp: Creating /dev/null/zero
bbcp: At 090414 02:59:48 copy 0% complete; 5461.3 KB/s
bbcp: At 090414 02:59:50 copy 0% complete; 13639.7 KB/s
..

4 Tuneup

Usefull links

  • No labels