...
- The copying of data to the anonymous FTP server and moved from there to the PingER raw data archive needs to complete before getdata.pl starts at 32 minutes past midnight local time each night at SLAC
- Once the data is copied from the MA to anonymous FTP incoming space then proxy.pl, needs to be scheduled to copy the data from anonymous FTP directory to the directory accessible by the ping_data.pl CGI script.
- This also has to complete before getdata.pl is scheduled,
- proxy.pl takes < 5 seconds to execute.
- proxy.pl is therefore currently scheduled to run at 20 minutes past midnight localtime
- Once the move is completed then getdata.pl can be scheduled to gather and save the selected data from the MAs in the PingER raw data archive at:
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
- This (getdata.pl) takes about 15 minutes.
- getdata.pl is scheduled to run at 32 minutes past midnight localtime.
The analysis of the hourly data by analyze-all.pl needs to start after getdata.pl has completed. Currently wrap-analyze-all.pl starts as a cron job at 11 minutes past 1 am local time at SLAC each morning.
Code Block 11 1 * * * /usr/local/bin/bsub -W 180 -o /dev/null -q long /afs/slac/package/pinger/analysis/wrap-analyze-allmonths.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep 2>&1 #Takes ~ 2hr 15 min on pinger 11/11/2012.
The analyzed data from analyze-all.pl is saved in files of the form below, the contents are described in PingER data flow at SLAC:
Code Block /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/<metric>-<len>-<by>-<year>-<month>-<day>.txt.gz#len=100|1000, by=by-node|by-site. /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz
The analyzed data is saved in files of the form below, the contents are described in PingER data flow at SLAC:Code Block /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/<metric>-<len>-<by>-<year>-<month>-<day>.txt.gz#len=100|1000, by=by-node|by-site. /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz
See https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC for the data flow. The use of the proxy is triggered by the NODEDETAIL Data Server entry (e.g. proxy=1) to tell ping_data.pl to gather the data from the SLAC anonymous ftp server (via the copy)that is basically acting as a proxy.
...