Page History

...

There is a lot of data so you will need a lot of space.  I suggest you use the /tmp ditrectory as intermediate storage, do one metric at a time (e.g. average_rtt), one size (e.g. 100) and one by (e.g. by-node). If this does not work send an email to unix-admin temporary requesting space in /afs/slac/public/users/cottrell (this is accessible from anonymous FTP). Note you will need space for the copied directory and for the tar's and zipped file.   I requested 100Gbytes.

For 1 metric (average_rtt)

...

$cp -v /nfs/slac/g/net/pinger/pingerreports/hep/average_rtt/average_rtt-100-by-node* /afs/slac/public/users/cottrell/average_rtt-100-by-node
#There are about 6500 files per metric. Copy takes about 20 mins per metric. 

$tar -cvzf /afs/slac/public/users/cottrell /archive-average_rtt-100-by-node.tar /afs/slac/public/users/cottrell/average_rtt-100-by-node/average_rtt-100-by-node
#A  metric takes about 6 minutes to tar and compress and each tar file occupies ~ 1.5GBytes.

For all metrics with 100Byte pings by node

...

 $tar -cvzf /afs/slac/public/users/cottrell /archive-average_rtt-100-by-node.tar /afs/slac/public/users/cottrell/average_rtt-100-by-node/average_rtt-100-by-nodemkdir

However:

 $ls /nfs/slac/g/net/pinger/pingerreports/hep/*/*-100-by-node*

...

function large_cp ()
{       while read line1; do
                cp -v directory/$line1 /afs/slac/public/users/cottrell/metrics-100-by-node
        done
}i
ls -1 directory/ | large_cp

To provide maximum flexibility we decided to write a script (pinger-tar.pl) to copy and tar the data.

The compression ratio is about 3.2. 

To increase the number of files we could change the ping size from 100 to 1000 (in average_rtt-100-by-node* for example) and the by-node to by-site, i.e. a factor of 4.

Retrieving

See http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html