Introduction

This page describes, how the data collected by the SLAC PingER site, ends up as information, which is used by pingtable, motion charts, intensity maps and several other applications. Various scripts are used in turn to generate the data. The scripts in their turn are driven by a trscrontab that executes on pinger@pinger.slac.stanford.edu.

Scripts

TimePing:** /afs/slac/package/pinger/timeping.pl
- Represents the old script to perform ping measurements and store raw data.
- Stores raw data on nfs at:
  /nfs/slac/g/net/pinger/pinger_mon_data/ping-<YYYY>-<MM>.txt
PingER2:
- /afs/slac/package/pinger/pinger2/share/pinger/pinger2.pl
- Represents the new Tool to perform ping measurements and store raw data.
- See http://confluence.slac.stanford.edu/display/IEPM/PingER2+at+SLAC]
- Stores raw data on nfs at:
  /nfs/slac/g/net/pinger/pinger2/data/ping-<YYYY>-<MM>.txt e.g. /nfs/slac/g/net/pinger/pinger2/data/ping-2011-02.txt
Ping_Data
- /afs/slac/g/www/cgi-wrap-bin/net/offsite_mon/ping_data/ping_data.pl
- Represents the web interface(cgi script) used to fetch data from remote monitoring node.
- zips the data before transfer.

GetData

/afs/slac/package/pinger/getdata.pl
queries ping_data http://slac.stanford.edu/cgi-wrap/ping_data.pl to fetch data.

Stores old files as zipped File at

/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
e.g.
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2011-03-22.txt.gz
/nfs/slac/g/net/pinger/pingerdata/hep/data/pcgiga.cern.ch/ping-2006-09-28.txt.gz

There is also a wrapper to get data from multiple days. It is at /afs/slac/package/pinger/getdata_all.pl
See Restoring PingER data for more details.

Data formats for timeping.pl, pinger2.pl, ping_data.pl, getdata.pl and getdata_all.pl

Analyze

There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.

The first script to be executed is analyze-hourly.pl which takes as input data the output of getdata.pl and from this aggregaytes the data to by day and writes the latest to the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the file name <metric><size><by><yyyy><mm>-<dd>.txt.gz. The analyze-hourly.pl script is run daily from the trscrontab on pinger and by default analyzes the data gathered for yesterday.
Example output filename:

/nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz

By default the above file is created once thus the directory appears as:

57cottrell@pinger:~>ls -l /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05*
-rw-rw-r--  1 pinger   iepm 492144 May  2 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz
-rw-rw-r--  1 pinger   iepm 545968 May  3 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-02.txt.gz
-rw-rw-r--  1 pinger   iepm 561661 May  4 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-03.txt.gz
-rw-rw-r--  1 pinger   iepm 566550 May  5 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-04.txt.gz
-rw-rw-r--  1 pinger   iepm 537127 May  6 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-05.txt.gz
-rw-rw-r--  1 pinger   iepm 538830 May  7 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-06.txt.gz
-rw-rw-r--  1 pinger   iepm 488360 May  8 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-07.txt.gz
-rw-rw-r--  1 pinger   iepm 499020 May  9 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-08.txt.gz
-rw-rw-r--  1 pinger   iepm 563840 May 10 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-09.txt.gz
-rw-rw-r--  1 pinger   iepm 583454 May 11 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-10.txt.gz
-rw-rw-r--  1 cottrell iepm 577949 May 12 22:08 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-11.txt.gz
-rw-rw-r--  1 cottrell iepm    102 May 12 17:25 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-12.txt.gz

Example output format (there are 24 hourly lines like this per day/per host pair):

icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 ... 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu

The remaining analyze scripts (analyze-daily.pl, analyze-monthly.pl, analyze-allmonths.pl, and analyze-allyears.pl) take as input the data from analyze-hourly.pl, analyze-daily.pl, analyze-daily.pl, and analyze-allmonthly.pl and create files of the form:
/nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>(-<YYYY>?)(-<mm>?)(-<dd>?).txt.gz
More information can be found on running getdata.pl and the analyze scripts to recover missing data.

PRM
- /afs/slac/package/pinger/prm/prm.pl
- The script prm.pl, is used to create csv files for the analyzed data which can then be used by PingER motion charts or PingER Executive Plots.
- The script requires a configuration file, which contains entries for all the reports prm should create. These entries are of the form:
  <metric name> <monitoring site> <country||continent> <tick> (<filter>?)
- It gets its data from pingtable.pl either by calling it locally or by accessing it via the web.
- The reports created by prm are available online.
- The reports are used by the metric motion plots, the metric maps and the executive plots.

Data

/nfs/slac/g/net/pinger/pinger_mon_data/ping-<YYYY><MM>.txt contains data from 2005-2010./nfs/slac/g/net/pinger/pinger2/data/ping<YYYY>-<MM>.txt contains data from 2009-2012./nfs/slac/g/net/pinger/pingerdata/ contains data from 1997-2003/nfs/slac/g/net/pinger/pingerdata/hep/data/ contains data 1997-2007

Anomalies

We have spotted anomalies between the values reported by:

table.pl and pingtable.pl
and pingtable.pl at SLAC and pingtable.pl at SEECS

They are discussed and explained in the Anomalies report.

Backup

On 3/9/2012, we requested unix-admin@slac.stanford.edu to backup /nfs/slac/g/net/pinger/ on a regular basis. This was added to the nightly backup by Andrew May.