Page History

...

• Check the data has been gathered:
  • /afs/slac/package/pinger/checkdata_gif.pl calls /afs/slac/package/pinger/checkdata.pl to see if there is gathered data, and then writes out http://www-iepm.slac.stanford.edu/monitoring/checkdata/
• Analyze
  • There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.
    • Note that to be included in the analyzed data the host must be in NODEDETAILS.

  • The first script to be executed is wrap-analyze-hourly.pl (which is executed by calling the wrapper analyze-all.pl --date 1days from the trscronjob) which takes as input data the output of getdata.pl (raw data) and from this aggregates the data to by day and writes the latest  to the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the file name <metric><size><by><yyyy><mm>-<dd>.txt.gz., this is refrred to as the hourly analyzed data. The wrap-analyze-hourly.pl script is run daily from the trscrontab on pinger and by default analyzes the data gathered for yesterday. 
    Example output filename for the minimum_rtt metric:

    Code Block
    /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz

    By default the above file is created once thus the directory appears as:

    Code Block

    57cottrell@pinger:~>ls -l /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05* -rw-rw-r-- 1 pinger iepm 492144 May 2 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz -rw-rw-r-- 1 pinger iepm 545968 May 3 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-02.txt.gz -rw-rw-r-- 1 pinger iepm 561661 May 4 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-03.txt.gz -rw-rw-r-- 1 pinger iepm 566550 May 5 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-04.txt.gz -rw-rw-r-- 1 pinger iepm 537127 May 6 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-05.txt.gz -rw-rw-r-- 1 pinger iepm 538830 May 7 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-06.txt.gz -rw-rw-r-- 1 pinger iepm 488360 May 8 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-07.txt.gz -rw-rw-r-- 1 pinger iepm 499020 May 9 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-08.txt.gz -rw-rw-r-- 1 pinger iepm 563840 May 10 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-09.txt.gz -rw-rw-r-- 1 pinger iepm 583454 May 11 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-10.txt.gz -rw-rw-r-- 1 cottrell iepm 577949 May 12 22:08 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-11.txt.gz -rw-rw-r-- 1 cottrell iepm 102 May 12 17:25 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-12.txt.gz

    Example output format. Following the 1st line in the file there is 1 line like the following per day/per host pair.  Between the initial and final src_name and tgt_name tokens there are 24 tokens one for each hour of the day, missing data is identified by a dot followed by a space (. ), e.g.:

    Code Block
    icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 . . . . . . . . 108.892 . . . . . . . . . . . . . 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu

    
    The first line in the file contains a label for each of the time slots (e.g. hours):

    Code Block
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

  • The remaining analyze scripts (wrap-analyze-daily.pl, wrap-analyze-monthly.pl, wrap-analyze-allmonths.pl, and wrap-analyze-allyears.pl) take as input the data from wrap-analyze-hourly.pl, wrap-analyze-daily.pl, and wrap-analyze-allmonths.pl and create files of the form:  

    Code Block
    /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>(-<YYYY>?)(-<mm>?)(-<dd>?).txt.gz /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<60|120|365>days.txt.gz /nfs/slac/g/net/pinger/pingerreports/hep/<metric>-<size>-by-<site|node>-<allmonths|allyears>.txt.gz

  • There are ~ 16 metrics:

    <option value="MOS">Mean Opinion Score</option>

    <option value="alpha">Directivity</option>

    <option value="average_rtt" selected>Average Round Trip Time</option>

    <option value="conditional_loss_probability">Conditional Loss Probability</option>

    <option value="duplicate_packets">Duplicate Packets</option>

    <option value="ipdv">Inter-Packet Delay Variation</option>

    <option value="iqr">Inter-Quartile Range</option>

    <option value="maximum_rtt">Maximum Round Trip Time</option>

    <option value="minimum_packet_loss">Minimum Packet Loss</option>

    <option value="minimum_rtt">Minimum Round Trip Time</option>

    <option value="out_of_order_packets">Out of Order Packets</option>

    <option value="packet_loss">Packet Loss</option>

    <option value="throughput">TCP Throughput (kbits/s)</option>

    <option value="unpredictability">Ping Unpredictability</option>

    <option value="unreachability">Ping Unreachability</option>

    <option value="zero_packet_loss_frequency">Zero Packet Loss Frequency</option>

    Information on these can be found at http://www.slac.stanford.edu/comp/net/wan-mon/tutorial.html.

     

  • More information can be found on running getdata.pl and the analyze scripts to recover missing data.
• PRM
  • /afs/slac/package/pinger/prm/prm.pl
  • The script prm.pl, is used to create csv files for the analyzed data which can then be used by PingER motion charts or PingER Executive Plots.

  • The script requires a configuration file, which contains entries for all the reports prm should create. These entries are of the form:

    Code Block
    <metric name> <monitoring site> <country||continent> <tick> (<filter>?)

  • It gets its data from pingtable.pl either by calling it locally or by accessing it via the web.
  • The reports created by prm are available online.
  • The reports are used by the metric motion plots, the metric maps and the executive plots.

  •   An example of the output is seen below:

    allyearly,?,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010

    EDU.SLAC.STANFORD.N3-to-Afghanistan,.,.,.,.,.,.,.,.,.,.,12.504,4.481,9.257,5.284

    EDU.SLAC.STANFORD.N3-to-Albania,.,.,.,.,.,.,.,.,.,.,.,4.949,7.133,4.590

     later it is atomically copied to files in the permanent directory

    A period(.) means there was no data.

    There is a directory with lots of these analyses, see https://www-iepm.slac.stanford.edu/pinger/prmout/

Order of running jobs in trscrontab

...