...
/bin/ls: Argument list too long.
Exit 1
So following the guidance in http://www.linuxjournal.com/article/6060 we could try to write a script:
function large_cp () { while read line1; do cp -v directory/$line1 /afs/slac/public/users/cottrell/metrics-100-by-node done }i ls -1 directory/ | large_cp
...
To provide maximum flexibility we decided to write a script (pinger-tar.pl) to copy and tar the data.
The directory looks as follows:
[cottrell@pinger ~]$ ls -l /afs/slac.stanford.edu/public/users/cottrell/*.tar
-rw-rw-r-- 1 cottrell sf 512297529 Sep 11 16:33 /afs/slac.stanford.edu/public/users/cottrell/MOS-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 755387730 Sep 11 17:00 /afs/slac.stanford.edu/public/users/cottrell/alpha-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 1640854797 Sep 11 13:58 /afs/slac.stanford.edu/public/users/cottrell/average_rtt-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 293583749 Sep 11 18:59 /afs/slac.stanford.edu/public/users/cottrell/conditional_loss_probability-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 259801856 Sep 11 19:40 /afs/slac.stanford.edu/public/users/cottrell/duplicate_packets-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 1018523005 Sep 11 14:27 /afs/slac.stanford.edu/public/users/cottrell/ipdv-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 751314261 Sep 11 16:16 /afs/slac.stanford.edu/public/users/cottrell/iqr-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 1896982390 Sep 11 18:03 /afs/slac.stanford.edu/public/users/cottrell/maximum_rtt-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 297380470 Sep 11 19:53 /afs/slac.stanford.edu/public/users/cottrell/minimum_packet_loss-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 1464424282 Sep 11 13:16 /afs/slac.stanford.edu/public/users/cottrell/minimum_rtt-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 240217457 Sep 11 19:24 /afs/slac.stanford.edu/public/users/cottrell/out_of_order_packets-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 425757104 Sep 11 15:54 /afs/slac.stanford.edu/public/users/cottrell/packet_loss-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 1995419214 Sep 11 15:33 /afs/slac.stanford.edu/public/users/cottrell/throughput-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 587342674 Sep 11 18:39 /afs/slac.stanford.edu/public/users/cottrell/unpredictability-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 273680921 Sep 11 18:16 /afs/slac.stanford.edu/public/users/cottrell/unreachability-100-by-node.tar
-rw-rw-r-- 1 cottrell sf 269081359 Sep 11 19:15 /afs/slac.stanford.edu/public/users/cottrell/zero_packet_loss_frequency-100-by-node.tar
The compression ratio is about 3.2.
To increase the number of files we could change the ping size from 100 to 1000 (in average_rtt-100-by-node* for example) and the by-node to by-site, i.e. a factor of 4.
Retrieving
See http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html