...
- TimePing:** /afs/slac/package/pinger/timeping.pl
- Represents the old script to perform ping measurements and store raw data.
Stores raw data on nfs at:
Code Block /nfs/slac/g/net/pinger/pinger_mon_data/ping-<YYYY>-<MM>.txt
The following files were moved from /nfs/slac/g/net/pinger/pinger_mon_data to /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/pinger_mon_data:
Code Block -rw-rw-r-- 1 pinger iepm 3271 Sep 28 2006 countries.txt -rw-r--r-- 1 pinger iepm 5149872 Jan 3 2013 gl-2008 -rw-r--r-- 1 pinger iepm 1211063 Apr 1 2010 ping-1970-01.txt -rw-rw-r-- 1 pinger iepm 99615499 Jan 31 2005 ping-2005-01.txt -rw-r--r-- 1 pinger iepm 138793772 Feb 28 2005 ping-2005-02.txt -rw-r--r-- 1 pinger iepm 158497470 Mar 31 2005 ping-2005-03.txt -rw-r--r-- 1 pinger iepm 155739263 Apr 30 2005 ping-2005-04.txt -rw-r--r-- 1 pinger iepm 159409884 May 31 2005 ping-2005-05.txt -rw-r--r-- 1 pinger iepm 88631442 Jun 30 2005 ping-2005-06.txt -rw-r--r-- 1 pinger iepm 174529497 Jul 31 2005 ping-2005-07.txt -rw-r--r-- 1 pinger iepm 174095994 Aug 31 2005 ping-2005-08.txt -rw-r--r-- 1 pinger iepm 168049168 Sep 30 2005 ping-2005-09.txt -rw-r--r-- 1 pinger iepm 176094644 Oct 31 2005 ping-2005-10.txt -rw-r--r-- 1 pinger iepm 176201647 Nov 30 2005 ping-2005-11.txt -rw-r--r-- 1 pinger iepm 180734057 Dec 31 2005 ping-2005-12.txt -rw-r--r-- 1 pinger iepm 181186778 Jan 31 2006 ping-2006-01.txt -rw-r--r-- 1 pinger iepm 162866844 Feb 28 2006 ping-2006-02.txt -rw-r--r-- 1 pinger iepm 180516079 Mar 31 2006 ping-2006-03.txt -rw-r--r-- 1 pinger iepm 173759442 Apr 30 2006 ping-2006-04.txt -rw-r--r-- 1 pinger iepm 181055493 May 31 2006 ping-2006-05.txt -rw-r--r-- 1 pinger iepm 179011290 Jun 30 2006 ping-2006-06.txt -rw-r--r-- 1 pinger iepm 174613487 Jul 31 2006 ping-2006-07.txt -rw-r--r-- 1 pinger iepm 182841805 Aug 31 2006 ping-2006-08.txt -rw-rw-r-- 1 pinger iepm 180770346 Sep 30 2006 ping-2006-09.txt -rw-r--r-- 1 pinger iepm 189730786 Oct 31 2006 ping-2006-10.txt -rw-r--r-- 1 pinger iepm 220799957 Nov 30 2006 ping-2006-11.txt -rw-r--r-- 1 pinger iepm 229571478 Dec 31 2006 ping-2006-12.txt -rw-r--r-- 1 pinger iepm 227199839 Jan 31 2007 ping-2007-01.txt -rw-r--r-- 1 pinger iepm 214317728 Feb 28 2007 ping-2007-02.txt -rw-r--r-- 1 pinger iepm 245744520 Mar 31 2007 ping-2007-03.txt -rw-r--r-- 1 pinger iepm 238642470 Apr 30 2007 ping-2007-04.txt -rw-r--r-- 1 pinger iepm 243478362 May 31 2007 ping-2007-05.txt -rw-r--r-- 1 pinger iepm 240197802 Jun 30 2007 ping-2007-06.txt -rw-r--r-- 1 pinger iepm 240797168 Jul 31 2007 ping-2007-07.txt -rw-r--r-- 1 pinger iepm 229320024 Aug 31 2007 ping-2007-08.txt -rw-r--r-- 1 pinger iepm 223024643 Sep 30 2007 ping-2007-09.txt -rw-r--r-- 1 pinger iepm 237230240 Oct 31 2007 ping-2007-10.txt -rw-r--r-- 1 pinger iepm 222792696 Nov 30 2007 ping-2007-11.txt -rw-r--r-- 1 pinger iepm 230369872 Dec 31 2007 ping-2007-12.txt -rw-r--r-- 1 pinger iepm 280679672 Jan 31 2008 ping-2008-01.txt -rw-r--r-- 1 pinger iepm 268660800 Feb 29 2008 ping-2008-02.txt -rw-r--r-- 1 pinger iepm 284800144 Mar 31 2008 ping-2008-03.txt -rw-r--r-- 1 pinger iepm 254738571 Apr 30 2008 ping-2008-04.txt -rw-r--r-- 1 pinger iepm 283045082 May 31 2008 ping-2008-05.txt -rw-r--r-- 1 pinger iepm 253567530 Jun 30 2008 ping-2008-06.txt -rw-r--r-- 1 pinger iepm 277043096 Jul 31 2008 ping-2008-07.txt -rw-r--r-- 1 pinger iepm 265867437 Aug 31 2008 ping-2008-08.txt -rw-r--r-- 1 pinger iepm 272576870 Sep 30 2008 ping-2008-09.txt -rw-r--r-- 1 pinger iepm 290541757 Oct 31 2008 ping-2008-10.txt -rw-r--r-- 1 pinger iepm 290687870 Nov 30 2008 ping-2008-11.txt -rw-r--r-- 1 pinger iepm 303514542 Dec 31 2008 ping-2008-12.txt -rw-r--r-- 1 pinger iepm 302877695 Jan 31 2009 ping-2009-01.txt -rw-r--r-- 1 pinger iepm 273710166 Feb 28 2009 ping-2009-02.txt -rw-r--r-- 1 pinger iepm 306043767 Mar 31 2009 ping-2009-03.txt -rw-r--r-- 1 pinger iepm 293984550 Apr 30 2009 ping-2009-04.txt -rw-r--r-- 1 pinger iepm 307293039 May 31 2009 ping-2009-05.txt -rw-r--r-- 1 pinger iepm 297975758 Jun 30 2009 ping-2009-06.txt -rw-r--r-- 1 pinger iepm 315918646 Jul 31 2009 ping-2009-07.txt -rw-r--r-- 1 pinger iepm 321719374 Aug 31 2009 ping-2009-08.txt -rw-r--r-- 1 pinger iepm 302208942 Sep 30 2009 ping-2009-09.txt -rw-r--r-- 1 pinger iepm 85141878 Oct 15 2009 ping-2009-10-before-merge.txt -rw-r--r-- 1 pinger iepm 139176298 Oct 15 2009 ping-2009-10-merged.txt -rw-r--r-- 1 pinger iepm 289926170 Oct 31 2009 ping-2009-10.txt -rw-r--r-- 1 pinger iepm 76875864 Oct 14 2009 ping-2009-10.txt.bak -rw-r--r-- 1 pinger iepm 286196134 Nov 30 2009 ping-2009-11.txt -rw-r--r-- 1 pinger iepm 294000614 Dec 31 2009 ping-2009-12.txt -rw-r--r-- 1 pinger iepm 275647748 Jan 31 2010 ping-2010-01.txt -rw-r--r-- 1 pinger iepm 260805183 Feb 28 2010 ping-2010-02.txt -rw-r--r-- 1 pinger iepm 295723517 Mar 31 2010 ping-2010-03.txt -rw-r--r-- 1 pinger iepm 32801520 Apr 4 2010 ping-2010-04.txt -rw-rw-r-- 1 pinger iepm 126894 Sep 28 2006 pingtable.txt
- PingER2:
- /afs/slac/package/pinger/pinger2/share/pinger/pinger2.pl
- Represents the new Tool to perform ping measurements and store raw data.
- See http://confluence.slac.stanford.edu/display/IEPM/PingER2+at+SLAC]
At SLAC stores raw data on nfs at:
Code Block /nfs/slac/g/net/pinger/pinger2/data/ping-<YYYY>-<MM>.txt e.g. /nfs/slac/g/net/pinger/pinger2/data/ping-2011-02.txt
- Ping_Data
- /afs/slac/g/www/cgi-wrap-bin/net/offsite_mon/ping_data/ping_data.pl
- Represents the web interface(cgi script) used to fetch data from remote monitoring node.
- zips the data before transfer.
- GetData
- /afs/slac/package/pinger/getdata.pl
- queries ping_data http://slac.stanford.edu/cgi-wrap/ping_data.pl to fetch data.
Since 2004 it stores old files as zipped File at
Code Block /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2011-03-22.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/pcgiga.cern.ch/ping-2006-09-28.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/2001:da8:270:2018:f816:3eff:fef3:bd3/ping-2018-09-09.txt.gz
At the Measurement Agents (MAs) it is typically stored at: /usr/local/share/pinger/data
- There is also a wrapper to get data from multiple days. It is at /afs/slac/package/pinger/getdata_all.pl
- See Restoring PingER data for more details.
Data from 1997-2003 can be found in /nfs/slac/g/net/pinger/pingerdata/. The files are zipped and compressed and contain the gathered data for 100 and 1000 byte pings for each day for all monitors.
Code Block 157cottrell@pinger:~$ls -l /nfs/slac/g/net/pinger/pingerdata/ total 11 drwxrwsr-x 2 6995 iepm 512 Dec 2 2011 1997/ drwxrwsr-x 2 6995 iepm 1024 Jan 25 2005 1998/ drwxrwsr-x 2 6995 iepm 1024 Jan 25 2005 1999/ drwxrwsr-x 2 6995 iepm 1024 Aug 3 2007 2000/ drwxrwsr-x 2 6995 iepm 512 Jan 25 2005 2001/ drwxrwsr-x 2 6995 iepm 512 Jan 25 2005 2002/ drwxrwsr-x 38 6995 iepm 1536 Feb 8 2007 2003/ drwxrwsr-x 6 iepm iepm 512 Jul 20 15:19 hep/ drwxrwsr-x 3 pinger iepm 512 Mar 8 2012 new/ drwxrwsr-x 4 6995 iepm 512 Jan 25 2005 oldftp/ 158cottrell@pinger:~$ls -l /nfs/slac/g/net/pinger/pingerdata/2002/ total 3844288 -rw-r--r-- 1 6995 iepm 349112320 Jan 25 2005 data-2002-01.tar -rw-r--r-- 1 6995 iepm 316467200 Jan 25 2005 data-2002-02.tar -rw-r--r-- 1 6995 iepm 332656640 Jan 25 2005 data-2002-03.tar -rw-r--r-- 1 6995 iepm 326103040 Jan 25 2005 data-2002-04.tar -rw-r--r-- 1 6995 iepm 347064320 Jan 25 2005 data-2002-05.tar -rw-r--r-- 1 6995 iepm 324648960 Jan 25 2005 data-2002-06.tar -rw-r--r-- 1 6995 iepm 319150080 Jan 25 2005 data-2002-07.tar -rw-r--r-- 1 6995 iepm 320245760 Jan 25 2005 data-2002-08.tar -rw-r--r-- 1 6995 iepm 336117760 Jan 25 2005 data-2002-09.tar -rw-r--r-- 1 6995 iepm 335669760 Jan 25 2005 data-2002-10.tar -rw-r--r-- 1 6995 iepm 303858176 Jan 25 2005 data-2002-11.tar -rw-r--r-- 1 6995 iepm 323323904 Jan 25 2005 data-2002-12.tar 168cottrell@pinger:~$cp /nfs/slac/g/net/pinger/pingerdata/2002/data-2002-01.tar /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/ $cd /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/ $tar -xvf /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/data-2002-01.tar cache01.ansp.br/ping-2002-01-01.txt.gz cache01.ansp.br/ping-2002-01-02.txt.gz ... yumj2.kek.jp/ping-2002-01-30.txt.gz yumj2.kek.jp/ping-2002-01-31.txt.gz 182cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$ls 172.23.52.7/ monitor.seecs.edu.pk/ pinger.cdacmumbai.in/ pingerlhr-pu.pern.edu.pk/ aup.seecs.edu.pk/ moore.ece.rice.edu/ pinger.cemb.edu.pk/ pingerpwr.pern.edu.pk/ ... 177cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$cp yumj2.kek.jp/ping-2002-01-31.txt.gz /tmp/ 178cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$gunzip /tmp/ping-2002-01-31.txt.gz 180cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$tail /tmp/ping-2002-01-31.txt yumj2.kek.jp 130.87.34.37 ultra.edu.uy 164.73.128.70 100 1012520610 10 10 379.815 381.366 384.206 0 1 2 3 4 5 6 7 8 9 380.758 380.938 382.205 380.480 380.878 384.206 379.815 381.087 382.547 380.743 yumj2.kek.jp 130.87.34.37 frcu.eun.eg 193.227.1.1 100 1012520610 10 10 364.870 399.626 446.021 0 1 2 3 4 5 6 7 8 9 398.877 395.008 388.069 396.147 416.314 392.862 398.349 399.744 364.870 446.021
Note that after the above, the raw input data to wrap-analyze-hourly.pl (see below) for 1998..2003 comes from /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/ rather than /nfs/slac/g/net/pinger/pingerdata/hep/data/.
There is a script /afs/slac/package/pinger/pre2004.pl that will take the data in /nfs/slac/g/net/pinger/pingerdata/<1998..2003> copy it, unzip, and untar into /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/.
There is a second script /afs/slac/package/pinger/pre2004-hourly.pl that using /afs/slac/package/pinger/analysis/analyze-all.pl reads the raw data from /afs/slac/g/net/pinger/pingerdata/hep/data.unite/ and analyzes and stores the hourly data for the selected years.
- Unite-monthly
- There is a script /afs/slac/package/pinger/unite-monthly.pl that unites all the data for a month into a single file of the form /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>.txt. It is run from a trscronjob at 4:00am each morning
- Data formats for timeping.pl, pinger2.pl, ping_data.pl, getdata.pl and getdata_all.pl
- Check the data has been gathered:
- /afs/slac/package/pinger/checkdata_gif.pl calls /afs/slac/package/pinger/checkdata.pl to see if there is gathered data, and then writes out http://www-iepm.slac.stanford.edu/monitoring/checkdata/
- Analyze
- There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.
- Note that to be included in the analyzed data the host must be in NODEDETAILS.
The first script to be executed is wrap-analyze-hourly.pl (which is executed by calling the wrapper analyze-all.pl --date 1days from the trscronjob) which takes as input data the output of getdata.pl (raw data) and from this aggregates the data to by day and writes the latest to the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the file name <metric><size><by><yyyy><mm>-<dd>.txt.gz., this is refrred to as the hourly analyzed data. The analyze-hourly.pl script is run daily from the trscrontab on pinger and by default analyzes the data gathered for yesterday.
Example output filename for the minimum_rtt metric:Code Block /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz
By default the above file is created once thus the directory appears as:
Code Block 57cottrell@pinger:~>ls -l /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05* -rw-rw-r-- 1 pinger iepm 492144 May 2 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz -rw-rw-r-- 1 pinger iepm 545968 May 3 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-02.txt.gz -rw-rw-r-- 1 pinger iepm 561661 May 4 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-03.txt.gz -rw-rw-r-- 1 pinger iepm 566550 May 5 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-04.txt.gz -rw-rw-r-- 1 pinger iepm 537127 May 6 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-05.txt.gz -rw-rw-r-- 1 pinger iepm 538830 May 7 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-06.txt.gz -rw-rw-r-- 1 pinger iepm 488360 May 8 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-07.txt.gz -rw-rw-r-- 1 pinger iepm 499020 May 9 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-08.txt.gz -rw-rw-r-- 1 pinger iepm 563840 May 10 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-09.txt.gz -rw-rw-r-- 1 pinger iepm 583454 May 11 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-10.txt.gz -rw-rw-r-- 1 cottrell iepm 577949 May 12 22:08 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-11.txt.gz -rw-rw-r-- 1 cottrell iepm 102 May 12 17:25 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-12.txt.gz
Example output format. Following the 1st line in the file there is 1 line like the following per day/per host pair. Between the initial and final src_name and tgt_name tokens there are 24 tokens one for each hour of the day, missing data is identified by a dot followed by a space (. ), e.g.:
Code Block icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 . . . . . . . . 108.892 . . . . . . . . . . . . . 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu
The first line in the file contains a label for each of the time slots (e.g. hours):Code Block 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
The remaining analyze scripts (wrap-analyze-daily.pl, wrap-analyze-monthly.pl, wrap-analyze-allmonths.pl, and wrap-analyze-allyears.pl) take as input the data from wrap-analyze-hourly.pl, wrap-analyze-daily.pl, and wrap-analyze-allmonths.pl and create files of the form:
Code Block /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>(-<YYYY>?)(-<mm>?)(-<dd>?).txt.gz /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<60|120|365>days.txt.gz /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<allmonths|allyears>.txt.gz
There are ~ 16 metrics:
<option value="MOS">Mean Opinion Score</option> <option value="alpha">Directivity</option> <option value="average_rtt" selected>Average Round Trip Time</option> <option value="conditional_loss_probability">Conditional Loss Probability</option> <option value="duplicate_packets">Duplicate Packets</option> <option value="ipdv">Inter-Packet Delay Variation</option> <option value="iqr">Inter-Quartile Range</option> <option value="maximum_rtt">Maximum Round Trip Time</option> <option value="minimum_packet_loss">Minimum Packet Loss</option> <option value="minimum_rtt">Minimum Round Trip Time</option> <option value="out_of_order_packets">Out of Order Packets</option> <option value="packet_loss">Packet Loss</option> <option value="throughput">TCP Throughput (kbits/s)</option> <option value="unpredictability">Ping Unpredictability</option> <option value="unreachability">Ping Unreachability</option> <option value="zero_packet_loss_frequency">Zero Packet Loss Frequency</option> Information on these can be found at http://www.slac.stanford.edu/comp/net/wan-mon/tutorial.html.
- More information can be found on running getdata.pl and the analyze scripts to recover missing data.
- There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.
- PRM
- /afs/slac/package/pinger/prm/prm.pl
- The script prm.pl, is used to create csv files for the analyzed data which can then be used by PingER motion charts or PingER Executive Plots.
The script requires a configuration file, which contains entries for all the reports prm should create. These entries are of the form:
Code Block <metric name> <monitoring site> <country||continent> <tick> (<filter>?)
- It gets its data from pingtable.pl either by calling it locally or by accessing it via the web.
- The reports created by prm are available online.
- The reports are used by the metric motion plots, the metric maps and the executive plots.
...