...
- TimePing:** /afs/slac/package/pinger/timeping.pl
- Represents the old script to perform ping measurements and store raw data.
Stores raw data on nfs at:
Code Block /nfs/slac/g/net/pinger/pinger_mon_data/ping-<YYYY>-<MM>.txt
The following files were moved from /nfs/slac/g/net/pinger/pinger_mon_data to /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/pinger_mon_data:
Code Block -rw-rw-r-- 1 pinger iepm 3271 Sep 28 2006 countries.txt -rw-r--r-- 1 pinger iepm 5149872 Jan 3 2013 gl-2008 -rw-r--r-- 1 pinger iepm 1211063 Apr 1 2010 ping-1970-01.txt -rw-rw-r-- 1 pinger iepm 99615499 Jan 31 2005 ping-2005-01.txt -rw-r--r-- 1 pinger iepm 138793772 Feb 28 2005 ping-2005-02.txt -rw-r--r-- 1 pinger iepm 158497470 Mar 31 2005 ping-2005-03.txt -rw-r--r-- 1 pinger iepm 155739263 Apr 30 2005 ping-2005-04.txt -rw-r--r-- 1 pinger iepm 159409884 May 31 2005 ping-2005-05.txt -rw-r--r-- 1 pinger iepm 88631442 Jun 30 2005 ping-2005-06.txt -rw-r--r-- 1 pinger iepm 174529497 Jul 31 2005 ping-2005-07.txt -rw-r--r-- 1 pinger iepm 174095994 Aug 31 2005 ping-2005-08.txt -rw-r--r-- 1 pinger iepm 168049168 Sep 30 2005 ping-2005-09.txt -rw-r--r-- 1 pinger iepm 176094644 Oct 31 2005 ping-2005-10.txt -rw-r--r-- 1 pinger iepm 176201647 Nov 30 2005 ping-2005-11.txt -rw-r--r-- 1 pinger iepm 180734057 Dec 31 2005 ping-2005-12.txt -rw-r--r-- 1 pinger iepm 181186778 Jan 31 2006 ping-2006-01.txt -rw-r--r-- 1 pinger iepm 162866844 Feb 28 2006 ping-2006-02.txt -rw-r--r-- 1 pinger iepm 180516079 Mar 31 2006 ping-2006-03.txt -rw-r--r-- 1 pinger iepm 173759442 Apr 30 2006 ping-2006-04.txt -rw-r--r-- 1 pinger iepm 181055493 May 31 2006 ping-2006-05.txt -rw-r--r-- 1 pinger iepm 179011290 Jun 30 2006 ping-2006-06.txt -rw-r--r-- 1 pinger iepm 174613487 Jul 31 2006 ping-2006-07.txt -rw-r--r-- 1 pinger iepm 182841805 Aug 31 2006 ping-2006-08.txt -rw-rw-r-- 1 pinger iepm 180770346 Sep 30 2006 ping-2006-09.txt -rw-r--r-- 1 pinger iepm 189730786 Oct 31 2006 ping-2006-10.txt -rw-r--r-- 1 pinger iepm 220799957 Nov 30 2006 ping-2006-11.txt -rw-r--r-- 1 pinger iepm 229571478 Dec 31 2006 ping-2006-12.txt -rw-r--r-- 1 pinger iepm 227199839 Jan 31 2007 ping-2007-01.txt -rw-r--r-- 1 pinger iepm 214317728 Feb 28 2007 ping-2007-02.txt -rw-r--r-- 1 pinger iepm 245744520 Mar 31 2007 ping-2007-03.txt -rw-r--r-- 1 pinger iepm 238642470 Apr 30 2007 ping-2007-04.txt -rw-r--r-- 1 pinger iepm 243478362 May 31 2007 ping-2007-05.txt -rw-r--r-- 1 pinger iepm 240197802 Jun 30 2007 ping-2007-06.txt -rw-r--r-- 1 pinger iepm 240797168 Jul 31 2007 ping-2007-07.txt -rw-r--r-- 1 pinger iepm 229320024 Aug 31 2007 ping-2007-08.txt -rw-r--r-- 1 pinger iepm 223024643 Sep 30 2007 ping-2007-09.txt -rw-r--r-- 1 pinger iepm 237230240 Oct 31 2007 ping-2007-10.txt -rw-r--r-- 1 pinger iepm 222792696 Nov 30 2007 ping-2007-11.txt -rw-r--r-- 1 pinger iepm 230369872 Dec 31 2007 ping-2007-12.txt -rw-r--r-- 1 pinger iepm 280679672 Jan 31 2008 ping-2008-01.txt -rw-r--r-- 1 pinger iepm 268660800 Feb 29 2008 ping-2008-02.txt -rw-r--r-- 1 pinger iepm 284800144 Mar 31 2008 ping-2008-03.txt -rw-r--r-- 1 pinger iepm 254738571 Apr 30 2008 ping-2008-04.txt -rw-r--r-- 1 pinger iepm 283045082 May 31 2008 ping-2008-05.txt -rw-r--r-- 1 pinger iepm 253567530 Jun 30 2008 ping-2008-06.txt -rw-r--r-- 1 pinger iepm 277043096 Jul 31 2008 ping-2008-07.txt -rw-r--r-- 1 pinger iepm 265867437 Aug 31 2008 ping-2008-08.txt -rw-r--r-- 1 pinger iepm 272576870 Sep 30 2008 ping-2008-09.txt -rw-r--r-- 1 pinger iepm 290541757 Oct 31 2008 ping-2008-10.txt -rw-r--r-- 1 pinger iepm 290687870 Nov 30 2008 ping-2008-11.txt -rw-r--r-- 1 pinger iepm 303514542 Dec 31 2008 ping-2008-12.txt -rw-r--r-- 1 pinger iepm 302877695 Jan 31 2009 ping-2009-01.txt -rw-r--r-- 1 pinger iepm 273710166 Feb 28 2009 ping-2009-02.txt -rw-r--r-- 1 pinger iepm 306043767 Mar 31 2009 ping-2009-03.txt -rw-r--r-- 1 pinger iepm 293984550 Apr 30 2009 ping-2009-04.txt -rw-r--r-- 1 pinger iepm 307293039 May 31 2009 ping-2009-05.txt -rw-r--r-- 1 pinger iepm 297975758 Jun 30 2009 ping-2009-06.txt -rw-r--r-- 1 pinger iepm 315918646 Jul 31 2009 ping-2009-07.txt -rw-r--r-- 1 pinger iepm 321719374 Aug 31 2009 ping-2009-08.txt -rw-r--r-- 1 pinger iepm 302208942 Sep 30 2009 ping-2009-09.txt -rw-r--r-- 1 pinger iepm 85141878 Oct 15 2009 ping-2009-10-before-merge.txt -rw-r--r-- 1 pinger iepm 139176298 Oct 15 2009 ping-2009-10-merged.txt -rw-r--r-- 1 pinger iepm 289926170 Oct 31 2009 ping-2009-10.txt -rw-r--r-- 1 pinger iepm 76875864 Oct 14 2009 ping-2009-10.txt.bak -rw-r--r-- 1 pinger iepm 286196134 Nov 30 2009 ping-2009-11.txt -rw-r--r-- 1 pinger iepm 294000614 Dec 31 2009 ping-2009-12.txt -rw-r--r-- 1 pinger iepm 275647748 Jan 31 2010 ping-2010-01.txt -rw-r--r-- 1 pinger iepm 260805183 Feb 28 2010 ping-2010-02.txt -rw-r--r-- 1 pinger iepm 295723517 Mar 31 2010 ping-2010-03.txt -rw-r--r-- 1 pinger iepm 32801520 Apr 4 2010 ping-2010-04.txt -rw-rw-r-- 1 pinger iepm 126894 Sep 28 2006 pingtable.txt
- PingER2:
- /afs/slac/package/pinger/pinger2/share/pinger/pinger2.pl
- Represents the new Tool to perform ping measurements and store raw data.
- See http://confluence.slac.stanford.edu/display/IEPM/PingER2+at+SLAC]
At SLAC stores raw data on nfs at:
Code Block /nfs/slac/g/net/pinger/pinger2/data/ping-<YYYY>-<MM>.txt e.g. /nfs/slac/g/net/pinger/pinger2/data/ping-2011-02.txt
- Ping_Data
- /afs/slac/g/www/cgi-wrap-bin/net/offsite_mon/ping_data/ping_data.pl
- Represents the web interface(cgi script) used to fetch data from remote monitoring node.
- zips the data before transfer.
- GetData
- /afs/slac/package/pinger/getdata.pl
- queries ping_data http://slac.stanford.edu/cgi-wrap/ping_data.pl to fetch data.
Since 2004 it stores old files as zipped File at
Code Block /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2011-03-22.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/pcgiga.cern.ch/ping-2006-09-28.txt.gz
- There is also a wrapper to get data from multiple days. It is at /afs/slac/package/pinger/getdata_all.pl
- See Restoring PingER data for more details.
Data from 1997-2003 can be found in /nfs/slac/g/net/pinger/pingerdata/. The files are zipped and compressed and contain the gathered data for 100 and 1000 byte pings for each day for all monitors.
Code Block 157cottrell@pinger:~$ls -l /nfs/slac/g/net/pinger/pingerdata/ total 11 drwxrwsr-x 2 6995 iepm 512 Dec 2 2011 1997/ drwxrwsr-x 2 6995 iepm 1024 Jan 25 2005 1998/ drwxrwsr-x 2 6995 iepm 1024 Jan 25 2005 1999/ drwxrwsr-x 2 6995 iepm 1024 Aug 3 2007 2000/ drwxrwsr-x 2 6995 iepm 512 Jan 25 2005 2001/ drwxrwsr-x 2 6995 iepm 512 Jan 25 2005 2002/ drwxrwsr-x 38 6995 iepm 1536 Feb 8 2007 2003/ drwxrwsr-x 6 iepm iepm 512 Jul 20 15:19 hep/ drwxrwsr-x 3 pinger iepm 512 Mar 8 2012 new/ drwxrwsr-x 4 6995 iepm 512 Jan 25 2005 oldftp/ 158cottrell@pinger:~$ls -l /nfs/slac/g/net/pinger/pingerdata/2002/ total 3844288 -rw-r--r-- 1 6995 iepm 349112320 Jan 25 2005 data-2002-01.tar -rw-r--r-- 1 6995 iepm 316467200 Jan 25 2005 data-2002-02.tar -rw-r--r-- 1 6995 iepm 332656640 Jan 25 2005 data-2002-03.tar -rw-r--r-- 1 6995 iepm 326103040 Jan 25 2005 data-2002-04.tar -rw-r--r-- 1 6995 iepm 347064320 Jan 25 2005 data-2002-05.tar -rw-r--r-- 1 6995 iepm 324648960 Jan 25 2005 data-2002-06.tar -rw-r--r-- 1 6995 iepm 319150080 Jan 25 2005 data-2002-07.tar -rw-r--r-- 1 6995 iepm 320245760 Jan 25 2005 data-2002-08.tar -rw-r--r-- 1 6995 iepm 336117760 Jan 25 2005 data-2002-09.tar -rw-r--r-- 1 6995 iepm 335669760 Jan 25 2005 data-2002-10.tar -rw-r--r-- 1 6995 iepm 303858176 Jan 25 2005 data-2002-11.tar -rw-r--r-- 1 6995 iepm 323323904 Jan 25 2005 data-2002-12.tar 168cottrell@pinger:~$cp /nfs/slac/g/net/pinger/pingerdata/2002/data-2002-01.tar /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/ $cd /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/ $tar -xvf /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/data-2002-01.tar cache01.ansp.br/ping-2002-01-01.txt.gz cache01.ansp.br/ping-2002-01-02.txt.gz ... yumj2.kek.jp/ping-2002-01-30.txt.gz yumj2.kek.jp/ping-2002-01-31.txt.gz 182cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$ls 172.23.52.7/ monitor.seecs.edu.pk/ pinger.cdacmumbai.in/ pingerlhr-pu.pern.edu.pk/ aup.seecs.edu.pk/ moore.ece.rice.edu/ pinger.cemb.edu.pk/ pingerpwr.pern.edu.pk/ ... 177cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$cp yumj2.kek.jp/ping-2002-01-31.txt.gz /tmp/ 178cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$gunzip /tmp/ping-2002-01-31.txt.gz 180cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$tail /tmp/ping-2002-01-31.txt yumj2.kek.jp 130.87.34.37 ultra.edu.uy 164.73.128.70 100 1012520610 10 10 379.815 381.366 384.206 0 1 2 3 4 5 6 7 8 9 380.758 380.938 382.205 380.480 380.878 384.206 379.815 381.087 382.547 380.743 yumj2.kek.jp 130.87.34.37 frcu.eun.eg 193.227.1.1 100 1012520610 10 10 364.870 399.626 446.021 0 1 2 3 4 5 6 7 8 9 398.877 395.008 388.069 396.147 416.314 392.862 398.349 399.744 364.870 446.021
Note that after the above, the raw input data to wrap-analyze-hourly.pl (see below) for 1998..2003 comes from /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/ rather than /nfs/slac/g/net/pinger/pingerdata/hep/data/.
There is a script /afs/slac/package/pinger/pre2004.pl that will take the data in /nfs/slac/g/net/pinger/pingerdata/<1998..2003> copy it, unzip, and untar into /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/.
There is a second script /afs/slac/package/pinger/pre2004-hourly.pl that using /afs/slac/package/pinger/analysis/analyze-all.pl reads the raw data from /afs/slac/g/net/pinger/pingerdata/hep/data.unite/ and analyzes and stores the hourly data for the selected years.
- Data formats for timeping.pl, pinger2.pl, ping_data.pl, getdata.pl and getdata_all.pl
- Check the data has been gathered:
- /afs/slac/package/pinger/checkdata_gif.pl calls /afs/slac/package/pinger/checkdata.pl to see if there is gathered data, and then writes out http://www-iepm.slac.stanford.edu/monitoring/checkdata/
- Analyze
- There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.
- Note that to be included in the analyzed data the host must be in NODEDETAILS.
The first script to be executed is wrap-analyze-hourly.pl (which is executed by calling the wrapper analyze-all.pl --date 1days from the trscronjob) which takes as input data the output of getdata.pl and from this aggregates the data to by day and writes the latest to the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the file name <metric><size><by><yyyy><mm>-<dd>.txt.gz. The analyze-hourly.pl script is run daily from the trscrontab on pinger and by default analyzes the data gathered for yesterday.
Example output filename for the minimum_rtt metric:Code Block /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz
By default the above file is created once thus the directory appears as:
Code Block 57cottrell@pinger:~>ls -l /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05* -rw-rw-r-- 1 pinger iepm 492144 May 2 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz -rw-rw-r-- 1 pinger iepm 545968 May 3 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-02.txt.gz -rw-rw-r-- 1 pinger iepm 561661 May 4 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-03.txt.gz -rw-rw-r-- 1 pinger iepm 566550 May 5 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-04.txt.gz -rw-rw-r-- 1 pinger iepm 537127 May 6 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-05.txt.gz -rw-rw-r-- 1 pinger iepm 538830 May 7 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-06.txt.gz -rw-rw-r-- 1 pinger iepm 488360 May 8 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-07.txt.gz -rw-rw-r-- 1 pinger iepm 499020 May 9 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-08.txt.gz -rw-rw-r-- 1 pinger iepm 563840 May 10 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-09.txt.gz -rw-rw-r-- 1 pinger iepm 583454 May 11 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-10.txt.gz -rw-rw-r-- 1 cottrell iepm 577949 May 12 22:08 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-11.txt.gz -rw-rw-r-- 1 cottrell iepm 102 May 12 17:25 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-12.txt.gz
Example output format. Following the 1st line in the file there is 1 line like the following per day/per host pair. Between the initial and final src_name and tgt_name tokens there are 24 tokens one for each hour of the day, missing data is identified by a dot followed by a space (. ), e.g.:
Code Block icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 . . . . . . . . 108.892 . . . . . . . . . . . . . 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu
The first line in the file contains a label for each of the time slots (e.g. hours):Code Block 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
The remaining analyze scripts (wrap-analyze-daily.pl, wrap-analyze-monthly.pl, wrap-analyze-allmonths.pl, and wrap-analyze-allyears.pl) take as input the data from wrap-analyze-hourly.pl, wrap-analyze-daily.pl, and wrap-analyze-allmonths.pl and create files of the form:
Code Block /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>(-<YYYY>?)(-<mm>?)(-<dd>?).txt.gz /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<60|120|365>days.txt.gz /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<allmonths|allyears>.txt.gz
There are ~ 16 metrics:
<option value="MOS">Mean Opinion Score</option> <option value="alpha">Directivity</option> <option value="average_rtt" selected>Average Round Trip Time</option> <option value="conditional_loss_probability">Conditional Loss Probability</option> <option value="duplicate_packets">Duplicate Packets</option> <option value="ipdv">Inter-Packet Delay Variation</option> <option value="iqr">Inter-Quartile Range</option> <option value="maximum_rtt">Maximum Round Trip Time</option> <option value="minimum_packet_loss">Minimum Packet Loss</option> <option value="minimum_rtt">Minimum Round Trip Time</option> <option value="out_of_order_packets">Out of Order Packets</option> <option value="packet_loss">Packet Loss</option> <option value="throughput">TCP Throughput (kbits/s)</option> <option value="unpredictability">Ping Unpredictability</option> <option value="unreachability">Ping Unreachability</option> <option value="zero_packet_loss_frequency">Zero Packet Loss Frequency</option> Information on these can be found at http://www.slac.stanford.edu/comp/net/wan-mon/tutorial.html.
- More information can be found on running getdata.pl and the analyze scripts to recover missing data.
- There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.
- PRM
- /afs/slac/package/pinger/prm/prm.pl
- The script prm.pl, is used to create csv files for the analyzed data which can then be used by PingER motion charts or PingER Executive Plots.
The script requires a configuration file, which contains entries for all the reports prm should create. These entries are of the form:
Code Block <metric name> <monitoring site> <country||continent> <tick> (<filter>?)
- It gets its data from pingtable.pl either by calling it locally or by accessing it via the web.
- The reports created by prm are available online.
- The reports are used by the metric motion plots, the metric maps and the executive plots.
...