...
As of 9/13/2014 there are about 20GBytes of compressed data and 100,000 files.
Nb the following files do not gunzip:
[root@sc2u0n0 afs]# find . -name "*.gz"
./slac/By 6/24/2019 there are about 33GBytes (do an ls -l /afs/slac.stanford.edu/public/users/cottrell/*.tar, import into Excel and use space to divide into columns and sum up the Bytes column).
Multiply by say 3.2 to get uncompressed data and it is about 100GBytes.
Nb the following files do not gunzip:
[root@sc2u0n0 afs]# find . -name "*.gz"
./slac/publicconditional_loss_probability-100-by-node/conditional_loss_probability-100-by-node-1998-02-20.txt.gz./slac/public/users/cottrell/minimumconditional_packetloss_lossprobability-100-by-node/minimumconditional_packetloss_lossprobability-100-by-node-20041998-0702-1820.txt.gz
I get:
268cottrell@pinger:/afs./slac/public/users/cottrell/conditionalminimum_packet_loss_probability-100-by-node/minimum_packet_loss-100-by-node-2004-07-18.txt.gz
I get:
268cottrell@pinger:/afs/slac/public/users/cottrell/conditional_loss_probability-100-by-node$gunzip /tmp/node$gunzip /tmp/conditional_loss_probability-100-by-node-1998-02-20.txt.gz
...
The raw data is the data initially recorded every 30 minutes by the pinger2.pl measurement script as opposed to the analyzed and aggregated into hourly interval data above. The hourly analyzed data has the advantage that it is cleaner since it has had filtering for some bad data, and the FTP archive for the analyzed data is updated daily etc. Gathered raw data for 1998-01 - 2007-07 is saved in For the record unzipped raw data records look like:
| Code Block |
|---|
pinger.slac.stanford.edu 134.79.240.30 www.eldjazair.net.dz 193.194.64.80 100 1178841602 10 10 195.572 196.680 198.257 0 1 2 3 4 5 6 7 8 9 196 195 198 195 197 196 197 195 196 196
|
The hourly analyzed data has the advantage that it is cleaner since it has had filtering for some bad data, and the FTP archive for the analyzed data is updated daily etc. Gathered raw data for 1998-01 - 2007-07 is saved in
| Code Block |
|---|
/nfs/slac/g/net/pinger/pingerdata/hep/data/<YYYY>/<host>/
e.g.
ls2007/brunsvigia.tenet.ac.za/
With files of the form:
-rw-rw-r-- 1 pinger iepm 584920 Jan 2 2007 ping-2007-01-01.txt.gz |
Gathered data from 2007-07 onwards is saved in:
| Code Block |
|---|
/nfs/slac/g/net/pinger/pingerdata/hep/data/<YYYY>/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/2007/brunsvigia.tenet.ac.za/ With files of the form: -rw-rw-r-- 1 pinger iepm 584920 Jan 2 2007 ping-2007-01-01.txt.gz |
Gathered data from 2007-07 onwards is saved in:
| Code Block |
|---|
/pinger.slac.stanford.edu/ping-2011-03-22.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>pcgiga.cern.ch/ping-<YYYY>2006-<MM>09-<DD>28.txt.gz e.g. |
...
ls -l /nfs/slac/g/net/pinger/pingerdata/hep/data/
...
* >! junk; wc junk), occupying 300GBytes.
.
| Code Block |
|---|
/nfs/slac/g/net/pinger/pingerdata/hep/data.unite/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz with files of the form: -rw-rw-r-- 1 cottrell sf 52464462 Jul 20 18:11 ping-2007-04.txt |
...