...
- A job at Beijing is required to do the anonymous ftp of the recent data from the Beijing MA to the incoming FTP server at SLAC. It runs at 14:01 Beijing time or 12:01am California standard (winter) time and 1:01am California summer time.
- The data then needs to be copied (by proxy.pl which takes a couple of seconds to run) from the anonymous FTP incoming space (/afs/slac/public/incoming/) to a directory accessible (see above) for reading from the ping_data.pl web CGI script that is called by wget from getdata.pl.
getdata.pl is then scheduled to run at SLAC to copy the selected data from the accessible directory /nfs/slac/g/net/pinger/pingerdata/hep/data/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/
- We want to standardize the time of the various cronjobs. 1:01am SLAC summer localtime is 4:01pm in China, and 12:01 midnight SLAC standard (winter) localtime is also 4:01pm in China. Thus to catch the Chinese data both winter and summer time California, we schedule the cron jobs at SLAC to run at just past 1:00am.
The various jobs have to be synchronized:
- The copying of data to the anonymous FTP server and moved moving from there to the PingER raw data archive needs to complete before getdata.pl starts at 32 minutes past midnight local time each night at SLAC
- Once the The data is copied from the MA to anonymous FTP incoming space at then proxy.pl, needs to be scheduled to copy the data from anonymous FTP directory to the directory accessible by the ping_data.pl CGI script.
- This also has to complete before getdata.pl is scheduled,
- proxy.pl takes < 5 seconds to execute.
- proxy.pl is therefore currently scheduled to run at 20 minutes past midnight localtime12:00am California standard time
- Once the move is completed by proxy.pl then getdata.pl can be scheduled to gather and save the selected data from the MAs in the PingER raw data archive at:
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
- This (getdata.pl) takes about 15 minutes.
- getdata.pl is scheduled to run at 32 minutes past midnight localtime1:00am localtime.
The analysis of the hourly data by analyze-all.pl needs to start after getdata.pl has completed. Currently analyze-all.pl starts as a cron job at 11 55 minutes past 1 past 2 am local time at SLAC each morning. the schedling of the jobs at SLAC is shown below;
Code Block 1120 1 * * * /usrafs/slac/localpackage/bin/bsub -W 180 -o /dev/null -q longpinger/proxy.pl 32 1 * * * /afs/slac/package/pinger/analysis/wrap-analyze-allmonthsgetdata.pl --basedir> /nfsafs/slac/g/netwww/www-iepm/pinger/slaconly/getdata.err 55 --usemetric2 * * * /afs/slac/package/pinger/analysis/analyze-all.pl --datasetdate hep1days 2>&1 #Takes ~ 2hr 15 min on pinger 11/11/2012.#Takes 25:14 10/20/2011 (55 mins 9/21/2011, 70 minutes 5/11/2018)
The analyzed data from analyze-all.pl is saved in files of the form below, the contents are described in PingER data flow at SLAC:
Code Block /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/<metric>-<len>-<by>-<year>-<month>-<day>.txt.gz#len=100|1000, by=by-node|by-site. /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gzThe analyzed data is saved in files of the form below, the contents are described in PingER data flow at SLAC:
Code Block /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/<metric>-<len>-<by>-<year>-<month>-<day>.txt.gz#len=100|1000, by=by-node|by-site. /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz
...