...
As of 9/13/2014 there are about 20GBytes of compressed data and 100,000 files.
Nb the following files do not gunzip:
[root@sc2u0n0 afs]# find . -name "*.gz"
./slac/By 6/24/2019 there are about 33GBytes (do an ls -l /afs/slac.stanford.edu/public/users/cottrell/*.tar, import into Excel and use space to divide into columns and sum up the Bytes column).
Multiply by say 3.2 to get uncompressed data and it is about 100GBytes.
Nb the following files do not gunzip:
[root@sc2u0n0 afs]# find . -name "*.gz"
./slac/publicconditional_loss_probability-100-by-node/conditional_loss_probability-100-by-node-1998-02-20.txt.gz./slac/public/users/cottrell/minimumconditional_packetloss_lossprobability-100-by-node/minimumconditional_packetloss_lossprobability-100-by-node-20041998-0702-1820.txt.gz
I get:
268cottrell@pinger:/afs./slac/public/users/cottrell/conditionalminimum_packet_loss_probability-100-by-node/minimum_packet_loss-100-by-node-2004-07-18.txt.gz
I get:
268cottrell@pinger:/afs/slac/public/users/cottrell/conditional_loss_probability-100-by-node$gunzip /tmp/node$gunzip /tmp/conditional_loss_probability-100-by-node-1998-02-20.txt.gz
...
gzip: /tmp/conditional_loss_probability-100-by-node-1998-02-20.txt.gz: invalid compressed data--length error
Updating Analyzed data in the archive
The FTP directory appears as:
...
The raw data is the data initially recorded every 30 minutes by the pinger2.pl measurement script as opposed to the analyzed and aggregated into hourly interval data above. For the record unzipped raw data records look like:
Code Block |
---|
pinger.slac.stanford.edu 134.79.240.30 www.eldjazair.net.dz 193.194.64.80 100 1178841602 10 10 195.572 196.680 198.257 0 1 2 3 4 5 6 7 8 9 196 195 198 195 197 196 197 195 196 196
|
The hourly The hourly analyzed data has the advantage that it is cleaner since it has had filtering for some bad data, and the FTP archive for the analyzed data is updated daily etc. Gathered raw data for 1998-01 - 2007-07 is saved in is cleaner since it has had filtering for some bad data, and the FTP archive for the analyzed data is updated daily etc. Gathered raw data for 1998-01 - 2007-07 is saved in
Code Block |
---|
/nfs/slac/g/net/pinger/pingerdata/hep/data/<YYYY>/<host>/
e.g.
ls2007/brunsvigia.tenet.ac.za/
With files of the form:
-rw-rw-r-- 1 pinger iepm 584920 Jan 2 2007 ping-2007-01-01.txt.gz |
Gathered data from 2007-07 onwards is saved in:
Code Block |
---|
/nfs/slac/g/net/pinger/pingerdata/hep/data/<YYYY>/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/2007/brunsvigia.tenet.ac.za/ With files of the form: -rw-rw-r-- 1 pinger iepm 584920 Jan 2 2007 ping-2007-01-01.txt.gz |
Gathered data from 2007-07 onwards is saved in:
Code Block |
---|
pinger.slac.stanford.edu/ping-2011-03-22.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>pcgiga.cern.ch/ping-<YYYY>2006-<MM>09-<DD>28.txt.gz e.g. |
...
ls -l /nfs/slac/g/net/pinger/pingerdata/hep/data/
...
* >! junk; wc junk), occupying 300GBytes.
.
Code Block |
---|
/nfs/slac/g/net/pinger/pingerdata/hep/data.unite/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz with files of the form: -rw-rw-r-- 1 cottrell sf 52464462 Jul 20 18:11 ping-2007-04.txt |
...