...
- Check the data has been gathered:
- /afs/slac/package/pinger/checkdata_gif.pl calls /afs/slac/package/pinger/checkdata.pl to see if there is gathered data, and then writes out a summary web page to http://www-iepm.slac.stanford.edu/monitoring/checkdata/
- You can manually run /afs/slac/package/pinger/checkdata.pl for a particular date with debugging
[pinger@cent7a ~]$ perl -d ~cottrell/bin/checkdata.pl 2023-06-03 2 #Check whether data for 2023-06-03 is present with debug set to 2.
- Analyze
- There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.
- Note that to be included in the analyzed data the host must be in NODEDETAILS.
The first script to be executed is wrap-analyze-hourly.pl (which is executed by calling the wrapper analyze-all.pl --date 1days from the trscronjob) which takes as input data the output of getdata.pl (raw data) and from this aggregates the data to by day and writes the latest to the /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/ directory with the file name <metric><size><by><yyyy><mm>-<dd>.txt.X., this is referred to as the hourly analyzed data. The wrap-analyze-hourly.pl script is run daily from the trscrontab on pinger and by default analyzes the data gathered for yesterday.
Example of analyze-all.pl
Code Block [pinger@pinger ~]$ ~cottrell/bin/analyze-all.pl --date 2023-08-01 1 #analyze the Sat Aug 12 11:44:42 2023(0s so far) executing /afs/slac/package/pinger/analysis/wrap-analyze-hourly.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep --debug 0 --datadir /nfs/slac/g/net/pinger/pingerdata/hep/data --size 100 --by by-node --date 2023-08-01 --set_metric 4 ... Sat Aug 12 11:44:43 2023: wrap-analyze-hourly.pl unzipped(debug=0, size=100, by=by-node, 1s so far) (85/229)16532 lines from /nfs/slac/g/net/pinger/pingerdata/hep/data/ping.desy.de/ping-2023-08-01.txt.gz Read 16532 lines for $monitoring_site ping.desy.de: found 8218 valid lines, 0 corrupted lines & 140 alpha>1.1 lines ... for each metric wrap-analyze-hourly.pl opening metric=maximum_rtt (16/1) for write to /nfs/slac/g/net/pinger/pingerreports/hep/maximum_rtt/maximum_rtt-100-by-node-2023-08-01.txt Wrote 1955 lines to /nfs/slac/g/net/pinger/pingerreports/hep/maximum_rtt/maximum_rtt-100-by-node-2023-08-01.txt.gz ... new value of --by (--by node, --by site), also will be --size 100, and --size 1000 Sat Aug 12 11:49:33 2023: wrap-analyze-hourly.pl unzipped(debug=0, size=100, by=by-site, 0s so far) (85/229)16532 lines from /nfs/slac/g/net/pinger/pingerdata/hep/data/ping.desy.de/ping-2023-08-01.txt.gz
The file names appear as:Code Block /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz
You can test whether the latest data is there (e.g. say today is 6/4/2023)
Code Block [cottrell@pinger ~]$ date Sun Jun 4 16:32:37 PDT 2023 [cottrell@pinger ~]$ ls -l /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2023-06-*.txt.gz -rw-rw-r--. 1 pinger iepm 153081 Jun 2 02:58 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2023-06-01.txt.gz -rw-rw-r--. 1 pinger iepm 154116 Jun 3 02:58 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2023-06-02.txt.gz i.e. the 2003-06-03 data is missing.
The output appears as:
Code Block cottrell@pinger $ cp /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2006-09-28.txt.gz /tmp/ cottrell@pinger $ zcat /tmp/minimum_rtt-100-by-node-2006-09-28.txt.gz cottrell@pinger $ head /tmp/minimum_rtt-100-by-node-2006-09-28.txt 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 pinger.slac.stanford.edu manila.global.net.pg 547.877 548.021 547.934 548.611 548.680 548.069 549.023 548.124 548.687 547.573 548.639 549.181 547.998 548.302 547.858 547.464 548.470 548.251 547.849 547.708 548.463 547.473 547.651 547.732 pinger.slac.stanford.edu manila.global.net.pg pinger.slac.stanford.edu lobelia.physics.wisc.edu 61.089 60.958 61.037 61.332 60.958 61.262 61.073 61.093 61.040 61.086 60.974 60.979 60.956 61.070 60.942 61.035 61.994 61.022 60.927 60.957 61.091 61.009 61.024 60.974 pinger.slac.stanford.edu lobelia.physics.wisc.edu
By default the above file is created once thus the directory appears as:
Code Block 57cottrell@pinger:~>ls -l /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05* -rw-rw-r-- 1 pinger iepm 492144 May 2 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz -rw-rw-r-- 1 pinger iepm 545968 May 3 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-02.txt.gz -rw-rw-r-- 1 pinger iepm 561661 May 4 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-03.txt.gz -rw-rw-r-- 1 pinger iepm 566550 May 5 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-04.txt.gz -rw-rw-r-- 1 pinger iepm 537127 May 6 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-05.txt.gz -rw-rw-r-- 1 pinger iepm 538830 May 7 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-06.txt.gz -rw-rw-r-- 1 pinger iepm 488360 May 8 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-07.txt.gz -rw-rw-r-- 1 pinger iepm 499020 May 9 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-08.txt.gz -rw-rw-r-- 1 pinger iepm 563840 May 10 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-09.txt.gz -rw-rw-r-- 1 pinger iepm 583454 May 11 02:17 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-10.txt.gz -rw-rw-r-- 1 cottrell iepm 577949 May 12 22:08 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-11.txt.gz -rw-rw-r-- 1 cottrell iepm 102 May 12 17:25 /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-12.txt.gz
Example output format. Following the 1st line in the file there is 1 line like the following per day/per host pair. Between the initial and final src_name and tgt_name tokens there are 24 tokens one for each hour of the day, missing data is identified by a dot followed by a space (. ), e.g.:
Code Block icfamon.dl.ac.uk lns62.lns.cornell.edu 108.871 . . . . . . . . 108.892 . . . . . . . . . . . . . 109.620 icfamon.dl.ac.uk lns62.lns.cornell.edu
The first line in the file contains a label for each of the time slots (e.g. hours):Code Block 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Example output filename for the minimum_rtt metric:
- There is a group of analysis scripts, that pick up zipped data; do their analysis, aggregate data and prepare web reports.
...