Introduction
This page describes, how the data collected by the SLAC PingER site, ends up as information, which is used by pingtable, motion charts, intensity maps and several other applications. Various scripts are used in turn to generate the data. The scripts in their turn are driven by a trscrontab that executes on pinger@pinger.slac.stanford.edu.
Scripts
- TimePing:** /afs/slac/package/pinger/timeping.pl
- Represents the old script to perform ping measurements and store raw data.
- Stores raw data on nfs at:
/nfs/slac/g/net/pinger/pinger_mon_data/ping-<YYYY>-<MM>.txt
- PingER2:
- /afs/slac/package/pinger/pinger2/share/pinger/pinger2.pl
- Represents the new Tool to perform ping measurements and store raw data.
- See http://confluence.slac.stanford.edu/display/IEPM/PingER2+at+SLAC]
- Stores raw data on nfs at:
/nfs/slac/g/net/pinger/pinger2/data/ping-<YYYY>-<MM>.txt e.g. /nfs/slac/g/net/pinger/pinger2/data/ping-2011-02.txt
- Ping_Data
- /afs/slac/g/www/cgi-wrap-bin/net/offsite_mon/ping_data/ping_data.pl
- Represents the web interface(cgi script) used to fetch data from remote monitoring node.
- zips the data before transfer.
- GetData
- /afs/slac/package/pinger/getdata.pl
- queries ping_data http://slac.stanford.edu/cgi-wrap/ping_data.pl to fetch data.
- Stores old files as zipped File at
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2011-03-22.txt.gz
- There is also a wrapper to get data from multiple days. It is at /afs/slac/package/pinger/getdata_all.pl
- See Restoring PingER data for more details.
- Data formats for timeping.pl, pinger2.pl, ping_data.pl, getdata.pl and getdata_all.pl
- Analyze
- There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.
- The first script to be executed is analyze-hourly.pl which takes as input data the output of getdata.pl and from this aggregaytes the data to by day and writes the latest to the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the file name <metric><size><by><yyyy><mm>-<dd>.txt.gz. The analyze-hourly.pl script is run daily from the trscrontab on pinger and by default analyzes the data gathered for yesterday.
Example output filename:By default the above file is created once thus the directory appears as:/nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz
Example output format (there are 24 hourly lines like this per day/per host pair):57cottrell@pinger:~>ls -l /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05* -rw-rw-r-- 1 pinger iepm 492144 May 2 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz -rw-rw-r-- 1 pinger iepm 545968 May 3 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-02.txt.gz -rw-rw-r-- 1 pinger iepm 561661 May 4 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-03.txt.gz -rw-rw-r-- 1 pinger iepm 566550 May 5 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-04.txt.gz -rw-rw-r-- 1 pinger iepm 537127 May 6 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-05.txt.gz -rw-rw-r-- 1 pinger iepm 538830 May 7 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-06.txt.gz -rw-rw-r-- 1 pinger iepm 488360 May 8 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-07.txt.gz -rw-rw-r-- 1 pinger iepm 499020 May 9 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-08.txt.gz -rw-rw-r-- 1 pinger iepm 563840 May 10 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-09.txt.gz -rw-rw-r-- 1 pinger iepm 583454 May 11 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-10.txt.gz -rw-rw-r-- 1 cottrell iepm 577949 May 12 22:08 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-11.txt.gz -rw-rw-r-- 1 cottrell iepm 102 May 12 17:25 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-12.txt.gz
icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 ... 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu
- The remaining analyze scripts (analyze-daily.pl, analyze-monthly.pl, analyze-allmonths.pl, and analyze-allyears.pl) take as input the data from analyze-hourly.pl, analyze-daily.pl, and analyze-monthly.pl and create files of the form:
/nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>(-<YYYY>?)(-<mm>?)(-<dd>?).txt.gz
- More information can be found on running getdata.pl and the analyze scripts to recover missing data.
- PRM
- /afs/slac/package/pinger/prm/prm.pl
- The script prm.pl, is used to create csv files for the analyzed data which can then be used by PingER motion charts or PingER Executive Plots.
- The script requires a configuration file, which contains entries for all the reports prm should create. These entries are of the form:
<metric name> <monitoring site> <country||continent> <tick> (<filter>?)
- It gets its data from pingtable.pl either by calling it locally or by accessing it via the web.
- The reports created by prm are available online.
- The reports are used by the metric motion plots, the metric maps and the executive plots.
Anomalies
We have spotted anomalies between the values reported by:
- table.pl and pingtable.pl
- and pingtable.pl at SLAC and pingtable.pl at SEECS
They are discussed and explained in the Anomalies report.
Backup
On 3/9/2012, we requested unix-admin@slac.stanford.edu to backup /nfs/slac/g/net/pinger/ on a regular basis. This was added to the nightly backup by Andrew May.