...
- PingER2:
- /afs/slac/package/pinger/pinger2/share/pinger/pinger2.pl: is the new/current tool to perform ping measurements and store raw data.
- At SLAC pinger2.pl is called by /afs/slac.stanford.edu/package/pinger/pinger2/share/pinger/pinger_startup.pl
This is to check that all is well (pinger2.pl exists, pinger.xml exists and pinger2.pl is not already/still running).
It executes:/usr/bin/perl /afs/slac/package/pinger/pinger2/share/pinger/pinger2.pl >/afs/slac/package/pinger/pinger2/share/pinger/pingerCronStat.stdout
Look at /afs/slac/package/pinger/pinger2/share/pinger/pingerCronStat.stdout to see how it is doing. - It "use"'s
useXML::Simple;
useData::Dumper;
useFcntl;
useFile::stat;
useNet::Domain qw(hostname hostfqdn hostdomain); Since it is run under perl, it shows up in top as:
Code Block PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15038 pinger 20 0 3769872 3.5g 11604 R 5.8 7.3 231:43.85 perl 112054 ytl 20 0 202832 10032 1372 S 0.1 0.0 8:52.46 sshd
- At SLAC pinger2.pl is called by /afs/slac.stanford.edu/package/pinger/pinger2/share/pinger/pinger_startup.pl
- See http://confluence.slac.stanford.edu/display/IEPM/PingER2+at+SLAC]
At SLAC stores raw data on nfs at:
Code Block /nfs/slac/g/net/pinger/pinger2/data/ping-<YYYY>-<MM>.txt e.g. /nfs/slac/g/net/pinger/pinger2/data/ping-2021-08.txt lines aree of the form: pinger.slac.stanford.edu 134.79.104.80 www.alief.dz 194.163.133.136 100 1627776002 10 10 148.694 149.496 152.268 1 2 3 4 5 6 7 8 9 10 148 152 148 150 148 148 151 148 148 148
This file is opened and written to for each set of pings in pingers.pl via the sub log_it in pinger2.pl. Failure to access the file will result in the following message example in the CronStat.stderr file:
Code Block Wed Sep 8 17:09:03 2021 Can't open PINGDATA /afs/slac/package/pinger/pinger2/share/pinger/ping-2021-09.txt: No such file or directory
- At SLAC:
- Information on Pinger hosts is kept in an Oracle database called NODEDETAILS
- The various target host files are updated from NODEDETAILS by a trscrontab job scheduled at 08:00, 13:00, 18:00, 23:00 hours calling
/afs/slac/package/pinger/write_offsitenodes.pl > /afs/slac/g/www/www-iepm/pinger/host-warnings.txt 2>&1 and taking ~ 17 mins - write_offsitenodes.pl reads input from the PingER Guthrie NODEDETAILS database and writes:
a file of nodes /afs/slac.stanford.edu/www/comp/net/mon/offsite.nodes
a file of beacons: /afs/slac/g/www/www-iepm/pinger/beacons.txt, this is accessed via https://www-iepm.slac.stanford.edu/pinger/beacons.txt
an XML file for worldwide (non-SLAC) pingER monitoring sites: /afs/slac.stanford.edu/g/www/www-iepm/pinger/pinger.xml, this is accessed via the URL http://www-iepm.slac.stanford.edu/pinger/pinger.xml
an XML file for the local SLAC pinger2 monitoring: /afs/slac.stanford.edu/package/pinger/pinger2/share/pinger/pinger.xml (includes all enabled i.e. M, B, NOT-SET)
A backup is also created at /afs/slac.stanford.edu/www/comp/net/mon/offsite.nodes.back
- The ping commands are:
<PingV4Cmd>/bin/ping -n -w %deadline -c %count -i %interval -s %packetsize %destination</PingV4Cmd>
with defaults %deadline=30, %ccount 10, %packetsize 100
<PingV6Cmd>/bin/ping6 -n -w %deadline -c %count -i %interval -s %packetsize %destination</PingV6Cmd> - The log file is at /afs/slac.stanford.edu/package/pinger/pinger2/share/pinger/pingerCronStat.stdout
- pinger2.pl gets the hosts to ping and other configuration information from the file /afs/slac.stanford.edu/package/pinger/pinger2/share/pinger/pinger.xml
- To test
Copy the pinger2.xml file to pinger3.xml. In pinger3.xml strip most of the hosts from <BeaconList>...</BeaconList>, <HostList>...</HostList>
Copy pinger2.pl to pinger3.pl, in pinger3.pl edit the line:
my$xmlfile='pinger.xml'; #$cwd/pinger.xml, configuration file for pinger2.pl
to yield
my $xmlfile='pinger3.xml'; #$cwd/pinger.xml, configuration file for pinger2.pl
In trscrontab -e comment out /afs/slac.stanford.edu/package/pinger/pinger2/share/pinger/pinger_startup.pl and wait for next half-hour interval (by which time the current pinger2.pl should have stopped running).
Then run cd /afs/slac.stanford.edu/package/pinger/pinger2/share/pinger ; perl -d /afs/slac.stanford.edu/package/pinger/pinger2/share/pinger/pinger3.pl;
- /afs/slac/package/pinger/pinger2/share/pinger/pinger2.pl: is the new/current tool to perform ping measurements and store raw data.
- Ping_Data
- /afs/slac/g/www/cgi-bin/net/offsite_mon/ping_data/ping_data.pl:
- Represents the web interface(cgi script) used to fetch data from remote monitoring node.
- zips the data before transfer.
- It is used by getdata.pl
- getdata-all.pl [-h monitoring_hostname] [-d days_ago] [-s start_date] [-e end_date]
- getdata.pl -h pinger.slac.stanford.edu -d 5
- getdata.pl -h all -s 2023-08-27 -h 2023-08-30
- getdatagetdata.pl
- Typically called from /afs/slac/package/pinger/getdata_all.pl
- /afs/slac/package/pinger/getdata.pl
- queries ping_data http://slac.stanford.edu/cgi-bin/ping_data.pl to fetch data (e.g. from /nfs/slac/g/net/pinger/pinger2/data)
Since 2004 it stores old files as zipped File at
Code Block /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2022-11-23.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/pcgiga.cern.ch/ping-2006-09-28.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/2001:da8:270:2018:f816:3eff:fef3:bd3/ping-2018-09-09.txt.gz or [pinger@pinger ~]$ ls -l /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2022-11-23.txt.gz -rw-rw-r--. 1 pinger iepm 2720699 Nov 24 09:05 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2022-11-23.txt.gz
At the Measurement Agents (MAs) it is typically stored at: /usr/local/share/pinger/data
- There is also a wrapper to get data from multiple days. It is at /afs/slac/package/pinger/getdata_all.pl
- See Restoring PingER data for more details.
Data from 1997-2003 can be found in /nfs/slac/g/net/pinger/pingerdata/. The files are zipped and compressed and contain the gathered data for 100 and 1000 byte pings for each day for all monitors.
Code Block 157cottrell@pinger:~$ls -l /nfs/slac/g/net/pinger/pingerdata/ total 11 drwxrwsr-x 2 6995 iepm 512 Dec 2 2011 1997/ drwxrwsr-x 2 6995 iepm 1024 Jan 25 2005 1998/ drwxrwsr-x 2 6995 iepm 1024 Jan 25 2005 1999/ drwxrwsr-x 2 6995 iepm 1024 Aug 3 2007 2000/ drwxrwsr-x 2 6995 iepm 512 Jan 25 2005 2001/ drwxrwsr-x 2 6995 iepm 512 Jan 25 2005 2002/ drwxrwsr-x 38 6995 iepm 1536 Feb 8 2007 2003/ drwxrwsr-x 6 iepm iepm 512 Jul 20 15:19 hep/ drwxrwsr-x 3 pinger iepm 512 Mar 8 2012 new/ drwxrwsr-x 4 6995 iepm 512 Jan 25 2005 oldftp/ 158cottrell@pinger:~$ls -l /nfs/slac/g/net/pinger/pingerdata/2002/ total 3844288 -rw-r--r-- 1 6995 iepm 349112320 Jan 25 2005 data-2002-01.tar -rw-r--r-- 1 6995 iepm 316467200 Jan 25 2005 data-2002-02.tar -rw-r--r-- 1 6995 iepm 332656640 Jan 25 2005 data-2002-03.tar -rw-r--r-- 1 6995 iepm 326103040 Jan 25 2005 data-2002-04.tar -rw-r--r-- 1 6995 iepm 347064320 Jan 25 2005 data-2002-05.tar -rw-r--r-- 1 6995 iepm 324648960 Jan 25 2005 data-2002-06.tar -rw-r--r-- 1 6995 iepm 319150080 Jan 25 2005 data-2002-07.tar -rw-r--r-- 1 6995 iepm 320245760 Jan 25 2005 data-2002-08.tar -rw-r--r-- 1 6995 iepm 336117760 Jan 25 2005 data-2002-09.tar -rw-r--r-- 1 6995 iepm 335669760 Jan 25 2005 data-2002-10.tar -rw-r--r-- 1 6995 iepm 303858176 Jan 25 2005 data-2002-11.tar -rw-r--r-- 1 6995 iepm 323323904 Jan 25 2005 data-2002-12.tar 168cottrell@pinger:~$cp /nfs/slac/g/net/pinger/pingerdata/2002/data-2002-01.tar /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/ $cd /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/ $tar -xvf /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/data-2002-01.tar cache01.ansp.br/ping-2002-01-01.txt.gz cache01.ansp.br/ping-2002-01-02.txt.gz ... yumj2.kek.jp/ping-2002-01-30.txt.gz yumj2.kek.jp/ping-2002-01-31.txt.gz 182cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$ls 172.23.52.7/ monitor.seecs.edu.pk/ pinger.cdacmumbai.in/ pingerlhr-pu.pern.edu.pk/ aup.seecs.edu.pk/ moore.ece.rice.edu/ pinger.cemb.edu.pk/ pingerpwr.pern.edu.pk/ ... 177cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$cp yumj2.kek.jp/ping-2002-01-31.txt.gz /tmp/ 178cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$gunzip /tmp/ping-2002-01-31.txt.gz 180cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata/hep/data.unite$tail /tmp/ping-2002-01-31.txt yumj2.kek.jp 130.87.34.37 ultra.edu.uy 164.73.128.70 100 1012520610 10 10 379.815 381.366 384.206 0 1 2 3 4 5 6 7 8 9 380.758 380.938 382.205 380.480 380.878 384.206 379.815 381.087 382.547 380.743 yumj2.kek.jp 130.87.34.37 frcu.eun.eg 193.227.1.1 100 1012520610 10 10 364.870 399.626 446.021 0 1 2 3 4 5 6 7 8 9 398.877 395.008 388.069 396.147 416.314 392.862 398.349 399.744 364.870 446.021
Note that after the above, the raw input data to wrap-analyze-hourly.pl (see below) for 1998..2003 comes from /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/ rather than /nfs/slac/g/net/pinger/pingerdata/hep/data/.
There is a script /afs/slac/package/pinger/pre2004.pl that will take the data in /nfs/slac/g/net/pinger/pingerdata/<1998..2003> copy it, unzip, and untar into /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/.
There is a second script /afs/slac/package/pinger/pre2004-hourly.pl that using /afs/slac/package/pinger/analysis/analyze-all.pl reads the raw data from /afs/slac/g/net/pinger/pingerdata/hep/data.unite/ and analyzes and stores the hourly data for the selected years.
- Unite-monthly
There is a script /afs/slac/package/pinger/unite-monthly.pl that unites all the raw data for a month into a single file of the form /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>.txt. It is run from a trscronjob at 4:00am each morning.
- The files appear as:
Code Block [pinger@cent7a ~]$ ls -l /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2023-06-03.txt.gz -rw-rw-r--. 1 pinger iepm 3216199 Jun 4 12:47 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2023-06-03.txt.gz
The output data appears as:
Code Block cottrell@pinger $ head /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2020-03.txt pinger.slac.stanford.edu 134.79.104.80 www.alief.dz 91.121.45.149 100 1583020804 10 10 149.672 149.979 150.649 1 2 3 4 5 6 7 8 9 10 149 149 150 150 149 149 150 149 149 149 pinger.slac.stanford.edu 134.79.104.80 www.ons.dz 196.20.66.100 100 1583020805 10 10 193.218 193.371 193.576 1 2 3 4 5 6 7 8 9 10 193 193 193 193 193 193 193 193 193 193 [pinger@pinger ~]$ tail /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2022-11.txt pinger.slac.stanford.edu 134.79.104.80 mail.neda.af 117.55.192.5 1000 1669247047 11 10 310.965 311.160 311.281 1 2 3 4 5 6 8 9 10 11 311 311 311 311 311 311 311 310 311 311 ... Where for example you can use https://www.epochconverter.com/ to find that the epoch time 1669247047 was equivalent to GMT: Wednesday, November 23, 2022 11:44:07 PM The timestamp on the file appears as: [pinger@cent7a ~]$ ls -l /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2023-06.txt -rw-rw-r--. 1 pinger iepm 68740320 Jun 4 04:00 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2023-06.txt where the date is todays date at 4:00am
- Data formats for timeping.pl, pinger2.pl, ping_data.pl, getdata.pl and getdata_all.pl
...