You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 52 Next »

Introduction

This page describes, how the data collected by the SLAC PingER site, ends up as information, which is used by pingtable, motion charts, intensity maps and several other applications. Various scripts are used in turn to generate the data. The scripts in their turn are driven by a trscrontab that executes on pinger@pinger.slac.stanford.edu.

Scripts

  • Analyze
    • There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.

    • The first script to be executed is wrap-analyze-hourly.pl which takes as input data the output of getdata.pl and from this aggregates the data to by day and writes the latest  to the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the file name <metric><size><by><yyyy><mm>-<dd>.txt.gz. The analyze-hourly.pl script is run daily from the trscrontab on pinger and by default analyzes the data gathered for yesterday. 
      Example output filename for the minimum_rtt metric:

      /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz

      By default the above file is created once thus the directory appears as:

      57cottrell@pinger:~>ls -l /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05*
-rw-rw-r--  1 pinger   iepm 492144 May  2 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz
-rw-rw-r--  1 pinger   iepm 545968 May  3 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-02.txt.gz
-rw-rw-r--  1 pinger   iepm 561661 May  4 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-03.txt.gz
-rw-rw-r--  1 pinger   iepm 566550 May  5 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-04.txt.gz
-rw-rw-r--  1 pinger   iepm 537127 May  6 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-05.txt.gz
-rw-rw-r--  1 pinger   iepm 538830 May  7 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-06.txt.gz
-rw-rw-r--  1 pinger   iepm 488360 May  8 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-07.txt.gz
-rw-rw-r--  1 pinger   iepm 499020 May  9 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-08.txt.gz
-rw-rw-r--  1 pinger   iepm 563840 May 10 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-09.txt.gz
-rw-rw-r--  1 pinger   iepm 583454 May 11 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-10.txt.gz
-rw-rw-r--  1 cottrell iepm 577949 May 12 22:08 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-11.txt.gz
-rw-rw-r--  1 cottrell iepm    102 May 12 17:25 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-12.txt.gz

      Example output format. Following the 1st line in the file there is 1 line like the following per day/per host pair.  Between the initial and final src_name and tgt_name tokens there are 24 tokens one for each hour of the day, missing data is identified by a dot followed by a space (. ), e.g.:

      icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 . . . . . . . . 108.892 . . . . . . . . . . . . . 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu


      The first line in the file contains a label for each of the time slots (e.g. hours):

      0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

    • The remaining analyze scripts (wrap-analyze-daily.pl, wrap-analyze-monthly.pl, wrap-analyze-allmonths.pl, and wrap-analyze-allyears.pl) take as input the data from wrap-analyze-hourly.pl, wrap-analyze-daily.pl, and wrap-analyze-allmonths.pl and create files of the form:  

      /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>(-<YYYY>?)(-<mm>?)(-<dd>?).txt.gz
/nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<60|120|365>days.txt.gz
/nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<allmonths|allyears>.txt.gz

    • There are ~ 16 metrics:

      <option value="MOS">Mean Opinion Score</option>
      <option value="alpha">Directivity</option>
      <option value="average_rtt" selected>Average Round Trip Time</option>
      <option value="conditional_loss_probability">Conditional Loss Probability</option>
      <option value="duplicate_packets">Duplicate Packets</option>
      <option value="ipdv">Inter-Packet Delay Variation</option>
      <option value="iqr">Inter-Quartile Range</option>
      <option value="maximum_rtt">Maximum Round Trip Time</option>
      <option value="minimum_packet_loss">Minimum Packet Loss</option>
      <option value="minimum_rtt">Minimum Round Trip Time</option>
      <option value="out_of_order_packets">Out of Order Packets</option>
      <option value="packet_loss">Packet Loss</option>
      <option value="throughput">TCP Throughput (kbits/s)</option>
      <option value="unpredictability">Ping Unpredictability</option>
      <option value="unreachability">Ping Unreachability</option>
      <option value="zero_packet_loss_frequency">Zero Packet Loss Frequency</option>

      Information on these can be found at http://www.slac.stanford.edu/comp/net/wan-mon/tutorial.html.

       

    • More information can be found on running getdata.pl and the analyze scripts to recover missing data.
  • PRM
    • /afs/slac/package/pinger/prm/prm.pl
    • The script prm.pl, is used to create csv files for the analyzed data which can then be used by PingER motion charts or PingER Executive Plots.

    • The script requires a configuration file, which contains entries for all the reports prm should create. These entries are of the form:

      <metric name>    <monitoring site>    <country||continent>    <tick>    (<filter>?)
    • It gets its data from pingtable.pl either by calling it locally or by accessing it via the web.
    • The reports created by prm are available online.
    • The reports are used by the metric motion plots, the metric maps and the executive plots.

Data

/nfs/slac/g/net/pinger/pinger_mon_data/ping-<YYYY>-<MM>.txt contains data from 2005-2010.
/nfs/slac/g/net/pinger/pinger2/data/ping-<YYYY>-<MM>.txt contains data from 2009-2012.
/nfs/slac/g/net/pinger/pingerdata/ contains data from 1997-2003
/nfs/slac/g/net/pinger/pingerdata/hep/data/ contains data 1997-2007

Data volumes Sep 2014

The total PingER data volume is about 550Gbytes.

We estimate that there are about 60 GBytes of uncompressed hourly data for 100 Byte pings by node, as of September 2014. One would estimate to about  quadruple that if one added 1000 byte pings and by site. See Volume of PingER data Sep 2014. 

Archiving

See Archiving PingER data by tar for retrieval by anonymous ftp

Missing data

Anomalies

We have spotted anomalies between the values reported by:

  • table.pl and pingtable.pl
  • and pingtable.pl at SLAC and pingtable.pl at SEECS

They are discussed and explained in the Anomalies report.

Backup

On 3/9/2012, we requested unix-admin@slac.stanford.edu to backup /nfs/slac/g/net/pinger/ on a regular basis. This was added to the nightly backup by Andrew May.

  • No labels