Table of Contents

Introduction

This page describes, how the data collected by the SLAC PingER site, ends up as information, which is used by pingtable, motion charts, intensity maps and several other applications. Various scripts are used in turn to generate the data. The scripts in their turn are driven by a trscrontab that executes on pinger@pinger.slac.stanford.edu.

Scripts

PingER scripts are available from  /afs/slac/package/pinger/.

Data

Data flow

  1. The raw data is measured by pinger2.pl  and gathered from the MAs using getdata_all.pl. The data is saved in /nfs/slac/g/net/pinger/pingerdata/hep/data/<MA>/ping-<YYYY>-MM-DD.txt.gz
    An example record is:  pinger.slac.stanford.edu 134.79.240.30 ping.slac.stanford.edu 134.79.18.21 100 1152489601 10 10 0.255 0.341 0.467 0 1 2 3 4 5 6 7 8 9 0.287 0.380 0.467 0.391 0.327 0.387 0.291 0.332 0.255 0.299
    To unzip the file:

    [pinger@pinger ~]$ cp /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2023-09-10.txt.gz ping-2023-09-10.txt.gz

    [pinger@pinger ~]$ gunzip ping-2023-09-10.txt.gz

    [pinger@pinger ~]$ ls -l ping-2023-09-10.txt

    -rw-r--r--. 1 pinger sf 16911992 Sep 12 12:19 ping-2023-09-10.txt

    To look at the last record use:

    [pinger@pinger ~] tail ping-2023-09-10.txt

    To see when logged look at the unix GMT time, eg  the 1694389609 in:

    pinger.slac.stanford.edu 134.79.104.80 www.mmbbank.com.mm 43.242.135.186 1000 1694389609 31 0

    use https://www.unixtimestamp.com/

  2. The hourly data is created from the raw data in /nfs/slac/g/net/pinger/pingerdata/hep/data/<site> (e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2021-03-06.txt.gz) by /afs/slac/package/pinger/analysis/analyze-all.pl and saved in the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with rhe file name <metric>-<size>-<by>-<yyyy>-<mm>-<dd>.txt.gz. E.g.  output filename: /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz. There are 24 hour lines per day/per host pair that appear as e.g.: icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 ... 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu

  3. The daily data is created from the hourly data by /afs/slac/package/pinger/analysis/wrap-analyze-daily.pl. It creates files for 30, 120, 180 and 365 days, that are saved in files of the form:  /nfs/slac/g/net/pinger/pingerreports/hep/data/[metric]/[metric]-[size]-by-[site|node]-YYYY-MM.txt.gz, e.g.      /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2007-05.txt.gz   
    /nfs/slac/g/net/pinger/pingerreports/hep/throughput/throughput-100-by-node-120days.txt.gz
  4. The monthly data is created from the daily data by /afs/slac/package/pinger/analysis/wrap-analyze-monthly.pl and the output goes to the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the filename <metric>-<size>-<by>.txt.gz.  e.g. /nfs/slac/g/net/pinger/pingerreports/hep/packet_loss/packet_loss-100-by-node.txt.gz

  5. The yearly data is created from the monthly data by /afs/slac/package/pinger/analysis/wrap-analyze-allyears.pl and it creates files such as /nfs/slac/g/net/pinger/pingerreports/hep/average_rtt/average_rtt-100-by-node-allyears.txt.gz. The output file lines appear as:

    1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
121.52.146.180 adl-a-ext1.aarnet.net.au . . . . . . . . . . . . . . . . . 314.0620 338.4232 281.2957 . . . . 121.52.146.180 adl-a-ext1.aarnet.net.au
121.52.146.180 airuniversity.seecs.edu.pk . . . . . . . . . . . . . . . . . . 5.8450 . . . . . 121.52.146.180 airuniversity.seecs.edu.pk
...


/nfs/slac/g/net/pinger/pinger_mon_data/ping-<YYYY>-<MM>.txt contains raw data from 2005-2010, just for SLAC.
/nfs/slac/g/net/pinger/pinger2/data/ping-<YYYY>-<MM>.txt contains raw data from 2009-present, just for SLAC
/nfs/slac/g/net/pinger/pingerdata/<data> contains data from 1997-2003 contains raw data
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ contains raw data 1997-200

Data volumes Sep 2014

The total PingER data volume is about 550Gbytes.

We estimate that there are about 60 GBytes of uncompressed hourly data for 100 Byte pings by node, as of September 2014. One would estimate to about quadruple that if one added 1000 byte pings and by site. See Volume of PingER data Sep 2014. 

Archiving

See Archiving PingER data by tar for retrieval by anonymous ftp

Missing data

Anomalies

We have spotted anomalies between the values reported by:

They are discussed and explained in the Anomalies report.

Problems

I get the error message below:

Your "cron" job


/afs/slac/package/pinger/analysis/wrap-analyze-daily.pl
--basedir /nfs/slac/g/net/pinger --usemetric --dataset hep --by by-site --size 1000

produced the following output:

Thu Dec  3 05:00:02 2015 Warning /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-1000-by-site-2015-12-01.txt.gz does not exist

Since the trscronjob analyze-all.pl --date 1days only reads and analyzes the most recent day's raw data if the job fails to run, then next days there will be missing data and you wilr get the above message. To recover the missing daily data run analyze-all.pl --date 2015-12-01 for the missing date (in this example 2015-12-01).

Backup

On 3/9/2012, we requested unix-admin@slac.stanford.edu to backup /nfs/slac/g/net/pinger/ on a regular basis. This was added to the nightly backup by Andrew May.

Order of running jobs in trscrontab


##############Basic PingER files scripts follow ######################
#Update nodes.cf (Perl dump of Guthrie config) from the NODEDETAIL Guthrie database, add by Cottrell 1/31/07 Though node.pl is called from runall.pl, runall.pl is not called until 5am, so it is updated here first to be sure.It takes about 3 seconds to run
pinger 30 0,6,12,18 * * * /afs/slac/package/pinger/guthrie/node.pl -o > /afs/slac/package/pinger/nodes.cf  #Takes about 30 secs
# Create the offsite.nodes list of nodes to ping from Guthrie NODEDETAILS database, add by Cottrell 1/3/2107 takes ~25 minutes
# The following files are created:
# /afs/slac.stanford.edu/www/comp/net/mon/offsite.nodes #txt file of all node, format: name address #comment
# /afs/slac.stanford.edu/g/www/www-iepm/pinger/beacons.txt #txt file of Beacons
# /afs/slac.stanford.edu/g/www/www-iepm/pinger/pinger.xml #xml file for non SLAC MA
# /afs/slac.stanford.edu/package/pinger/pinger2/share/pinger/pinger.xml  #xml file for SLAC MA
lnxcron 00 23 * * * /afs/slac/package/pinger/write_offsitenodes.pl > /afs/slac/g/www/www-iepm/pinger/host-warnings.txt 2>&1 #takes ~15  minutes on Pinger
#--------------Normal complete set for getdata.pl----------------------
pinger;200 32 1 * * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err #Takes ~20 minutes 7/31/2019
#Minimal subset is /afs/slac/package/pinger/getdata_all.pl -h pinger.slac.stanford.edu #Takes < 1 minute for pinger.slac.stanford.edu
#-----------------analyze-hourly.pl runs first for both 100B & 1000B pings, by-host, by site--------------------------
lnxcron;120 55 2 * * * Time-year.2023c
#Takes 25:14 10/20/2011 (55 mins 9/21/2011, 70 minutes 5/11/2018)
#-----------------analyze-daily-------------------------------------------
pinger;300 30 4 * * * /afs/slac/package/pinger/analysis/wrap-analyze-daily.pl --basedir /nfs/slac/g/net/pinger --use/metric --dataset hep # Takes ~ 5 hours for whole month on pinger 11/10/2012
#------------------analyze-monthly.pl for defaults by-node, size=100B takes about 60 minutes--------------
lnxcron 12 10 * * * /usr/local/bin/bsub -W 200 -o /dev/null -q xxl /afs/slac/package/pinger/analysis/wrap-analyze-monthly.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep > /dev/null #Takes about 55mins 9/19/2011, 61 mins 10/9/2019 on pinger or 7 mins on rhel6 11/13/2011.
#------------------sites-per-country---------------------------------------
pinger 00 09 * * * /afs/slac/package/pinger/sites-per-country.pl #Takes anout 10 minutes, run after analyze monthly

There is a script  /u/sf/cottrell/bin/manual-upd8-sites-per-country.pl  that will run in sequence the minimal set of scripts needed to update sites-per-country.html.

Checks:

  1. View pinger.xml at SLAC
  2. View log of measurements from SLAC, or view form
  3. View gathered data for SLAC at /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-<YYYY>-<MM>-<DD>.txt.gz
  4. View analyzed hourly data
  5. View analyzed daily data
  6. View analyzed monthly data
  7. View sites per country

Accessing PinGER data

Directories

See here

/usr/local/share/pinger/ is accessed by: