...
- A job at Beijing is required to do the anonymous ftp of the recent data from the Beijing MA to the incoming FTP server at SLAC. It runs at 1415:01 05 Beijing time or 12:01am California standard (winter) 05am California summer time and 1:01am California summer 05am California standard (winter) time.
- The data then needs to be copied (by proxy.pl which takes a couple of seconds to run) from the anonymous FTP incoming space (/afs/slac/public/incoming/) to a directory accessible (see above) for reading from the ping_data.pl web CGI script that is called by wget from getdata.pl.
getdata.pl is then scheduled to run at SLAC to copy the selected data from the accessible directory /nfs/slac/g/net/pinger/pingerdata/hep/data/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/
- We want to standardize the time of the various cronjobs. 1:01am 05am SLAC summer localtime time is 415:01pm 05 in China, and 12:01 05 midnight SLAC standard (winter) localtime is also 415:01pm 05 in China. Thus to catch the Chinese data both winter and summer time California, we schedule the proxy.pl cron jobs at SLAC to run at just past 1just 01:00am20am.
The various jobs have to be synchronized:
- The copying of data to the anonymous FTP server and moving from there to the PingER raw data archive needs to complete before getdata.pl starts at 01:32 minutes past midnight local time each night at SLAC
- The data is copied from the MA to anonymous FTP incoming space at then proxy.pl, needs to be scheduled to copy the data from anonymous FTP directory to the directory accessible by the ping_data.pl CGI script.
- This also has to complete before getdata.pl is scheduled,
- proxy.pl takes < 5 seconds to execute.
- proxy.pl is therefore currently scheduled to run at 20 minutes past 12:00am California standard time01:20am California localtime
- Once the move is completed by proxy.pl then getdata.pl can be scheduled to gather and save the selected data from the MAs in the PingER raw data archive at:
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
- This (getdata.pl) takes about 15 minutes.
- getdata.pl is scheduled to run at 32 minutes past 1:00am localtime01:32am localtime.
The analysis of the hourly data by analyze-all.pl needs to start after getdata.pl has completed. Currently analyze-all.pl starts as a cron job at 55 minutes past 2 am local time at SLAC each morning. the schedling of the jobs at SLAC is shown below;
Code Block 20 1 * * * /afs/slac/package/pinger/proxy.pl 32 1 * * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err 55 2 * * * /afs/slac/package/pinger/analysis/analyze-all.pl --date 1days #Takes 25:14 10/20/2011 (55 mins 9/21/2011, 70 minutes 5/11/2018)The analyzed data from analyze-all.pl is saved in files of the form below, the contents are described in PingER data flow at SLAC:
Code Block /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/<metric>-<len>-<by>-<year>-<month>-<day>.txt.gz#len=100|1000, by=by-node|by-site. /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz
...