...
The output appears as:Code Block cottrell@pinger $ cp /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz /tmp/ cottrell@pinger $ zcat /tmp/minimum_rtt-100-by-node-2006-09-28.txt.gz cottrell@pinger $ head /tmp/minimum_rtt-100-by-node-2006-09-28.txt 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 pinger.slac.stanford.edu manila.global.net.pg 547.877 548.021 547.934 548.611 548.680 548.069 549.023 548.124 548.687 547.573 548.639 549.181 547.998 548.302 547.858 547.464 548.470 548.251 547.849 547.708 548.463 547.473 547.651 547.732 pinger.slac.stanford.edu manila.global.net.pg pinger.slac.stanford.edu lobelia.physics.wisc.edu 61.089 60.958 61.037 61.332 60.958 61.262 61.073 61.093 61.040 61.086 60.974 60.979 60.956 61.070 60.942 61.035 61.994 61.022 60.927 60.957 61.091 61.009 61.024 60.974 pinger.slac.stanford.edu lobelia.physics.wisc.edu
By default the above file is created once thus the directory appears as:
Code Block 57cottrell@pinger:~>ls -l /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05* -rw-rw-r-- 1 pinger iepm 492144 May 2 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz -rw-rw-r-- 1 pinger iepm 545968 May 3 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-02.txt.gz -rw-rw-r-- 1 pinger iepm 561661 May 4 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-03.txt.gz -rw-rw-r-- 1 pinger iepm 566550 May 5 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-04.txt.gz -rw-rw-r-- 1 pinger iepm 537127 May 6 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-05.txt.gz -rw-rw-r-- 1 pinger iepm 538830 May 7 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-06.txt.gz -rw-rw-r-- 1 pinger iepm 488360 May 8 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-07.txt.gz -rw-rw-r-- 1 pinger iepm 499020 May 9 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-08.txt.gz -rw-rw-r-- 1 pinger iepm 563840 May 10 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-09.txt.gz -rw-rw-r-- 1 pinger iepm 583454 May 11 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-10.txt.gz -rw-rw-r-- 1 cottrell iepm 577949 May 12 22:08 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-11.txt.gz -rw-rw-r-- 1 cottrell iepm 102 May 12 17:25 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-12.txt.gz
Example output format. Following the 1st line in the file there is 1 line like the following per day/per host pair. Between the initial and final src_name and tgt_name tokens there are 24 tokens one for each hour of the day, missing data is identified by a dot followed by a space (. ), e.g.:
Code Block icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 . . . . . . . . 108.892 . . . . . . . . . . . . . 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu
The first line in the file contains a label for each of the time slots (e.g. hours):Code Block 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Example output filename for the minimum_rtt metric:The remaining analyze scripts (wrap-analyze-daily.pl, wrap-analyze-monthly.pl, wrap-analyze-allmonths.pl, and wrap-analyze-allyears.pl) take as input the data from wrap-analyze-hourly.pl, wrap-analyze-daily.pl, and wrap-analyze-allmonths.pl and create files of the form:
Code Block /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>(-<YYYY>?)(-<mm>?)(-<dd>?).txt.gz /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<60|120|365>days.txt.gz /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<allmonths|allyears>.txt.gz
There are ~ 16 metrics:
<option value="MOS">Mean Opinion Score</option> <option value="alpha">Directivity</option> <option value="average_rtt" selected>Average Round Trip Time</option> <option value="conditional_loss_probability">Conditional Loss Probability</option> <option value="duplicate_packets">Duplicate Packets</option> <option value="ipdv">Inter-Packet Delay Variation</option> <option value="iqr">Inter-Quartile Range</option> <option value="maximum_rtt">Maximum Round Trip Time</option> <option value="minimum_packet_loss">Minimum Packet Loss</option> <option value="minimum_rtt">Minimum Round Trip Time</option> <option value="out_of_order_packets">Out of Order Packets</option> <option value="packet_loss">Packet Loss</option> <option value="throughput">TCP Throughput (kbits/s)</option> <option value="unpredictability">Ping Unpredictability</option> <option value="unreachability">Ping Unreachability</option> <option value="zero_packet_loss_frequency">Zero Packet Loss Frequency</option> Information on these can be found at http://www.slac.stanford.edu/comp/net/wan-mon/tutorial.html.
- More information can be found on running getdata.pl and the analyze scripts to recover missing data.
- PRM
- /afs/slac/package/pinger/prm/prm.pl
- The script prm.pl, is used to create csv files for the analyzed data which can then be used by PingER motion charts or PingER Executive Plots.
The script requires a configuration file, which contains entries for all the reports prm should create. These entries are of the form:
Code Block <metric name> <monitoring site> <country||continent> <tick> (<filter>?)
- It gets its data from pingtable.pl either by calling it locally or by accessing it via the web.
- The reports created by prm are available online.
- The reports are used by the metric motion plots, the metric maps and the executive plots.
An example of the output is seen below:
allyearly,?,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010
EDU.SLAC.STANFORD.N3-to-Afghanistan,.,.,.,.,.,.,.,.,.,.,12.504,4.481,9.257,5.284
EDU.SLAC.STANFORD.N3-to-Albania,.,.,.,.,.,.,.,.,.,.,.,4.949,7.133,4.590
later it is atomically copied to files in the permanent directory
A period(.) means there was no data.
There is a directory with lots of these analyses, see https://www-iepm.slac.stanford.edu/pinger/prmout/
Data
Data flow
- The raw data is measured by pinger2.pl and gathered from the MAs using getdata_all.pl
...
- . The
...
- data is
...
- saved in /nfs/slac/g/net/pinger/
...
- pingerdata/hep/data/
...
- <MA>/
...
...
...
...
An example record is: pinger.slac.stanford.edu 134.79.240.30 ping.slac.stanford.edu 134.79.18.21 100 1152489601 10 10 0.255 0.341 0.467 0 1 2 3 4 5 6 7 8 9 0.287 0.380 0.467 0.391 0.327 0.387 0.291 0.332 0.255 0.299The hourly data is created from the raw data in /nfs/slac/g/net/pinger/pingerdata/hep/data/<site> (e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2021-03-06.txt.gz) by /afs/slac/package/pinger/analysis/analyze-all.pl and saved in the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with rhe file name <metric>-<size>-<by>-<yyyy>-<mm>-<dd>.txt.gz. E.g. output filename: /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz. There are 24 hour lines per day/per host pair that appear as e.g.: icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 ... 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu
- The daily data is created from the hourly data by /afs/slac/package/pinger/analysis/wrap-analyze-daily.pl. It creates files for 30, 120, 180 and 365 days, that are saved infiles of the form: /nfs/slac/g/net/pinger/pingerreports/hep/data/[metric]/[metric]-[size]-by-[site|node]-YYYY-MM.txt.gz, e.g. /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2007-05.txt.gz
/nfs/slac/g/net/pinger/pingerreports/hep/throughput/throughput-100-by-node-120days.txt.gz - The monthly data is created from the daily data by /afs/slac/package/pinger/analysis/wrap-analyze-monthly.pl and the oputput goes to the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the filename <metric>-<size>-<by>.txt.gz. e.g. /nfs/slac/g/net/pinger/pingerreports/hep/packet_loss/packet_loss-100-by-node.txt.gz
The yearly data is created from the monthly data by /afs/slac/package/pinger/analysis/wrap-analyze-allyears.pl and it creates files such as /nfs/slac/g/net/pinger/pingerreports/hep/average_rtt/average_rtt-100-by-node-allyears.txt.gz. The output file lines appear as:
Code Block 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 121.52.146.180 adl-a-ext1.aarnet.net.au . . . . . . . . . . . . . . . . . 314.0620 338.4232 281.2957 . . . . 121.52.146.180 adl-a-ext1.aarnet.net.au 121.52.146.180 airuniversity.seecs.edu.pk . . . . . . . . . . . . . . . . . . 5.8450 . . . . . 121.52.146.180 airuniversity.seecs.edu.pk ...
| Code Block |
|---|
/nfs/slac/g/net/pinger/pinger_mon_data/ping-<YYYY>-<MM>.txt contains raw data from 2005-2010, just for SLAC. /nfs/slac/g/net/pinger/pinger2/data/ping-<YYYY>-<MM>.txt contains raw data from 2009-present, just for SLAC /nfs/slac/g/net/pinger/pingerdata/<data> contains data from 1997-2003 contains raw data /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ contains raw data 1997-200 |
...