...
- Be aware data is only kept for 4 days in anonymous incoming FTP and then is removed. The directories are preserved. One can delete files and directories by using rm and rmdir on /afs/slac/public/incoming from the unix the Unix command line (one cannot delete files and directories within ftp).
- The anonymous FTP space is not accessible for reading from outside SLAC. This prevents the bad actors from hosting illegal files here.
...
- A job at Beijing is required to do the anonymous ftp of the recent data from the Beijing MA to the incoming FTP server at SLAC. It runs at 15:05 Beijing time or 12:05am California summer time and 1:05am California standard (winter) time.
- The data then needs to be copied (by proxy.pl which takes a couple of seconds to run) from the anonymous FTP incoming space (/afs/slac/public/incoming/) to a directory accessible (see above) for reading from the ping_data.pl web CGI script that is called by wget from getdata.pl.
getdata.pl is then scheduled to run at SLAC to copy the selected data from the accessible directory /nfs/slac/g/net/pinger/pingerdata/hep/data/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/
- We want to standardize the time of the various cronjobs. 1:05am 05 am SLAC summer time is 15:05 in China, and 12:05 midnight SLAC standard (winter) localtime is also 15:05 in China. Thus to catch the Chinese data both winter and summer time summertime California, we schedule the proxy.pl cron jobs at SLAC to run at just 01just 01:20am20 am.
The various jobs have to be synchronized:
- The copying of data to the anonymous FTP server and moving from there to the PingER raw data archive needs to complete beforegetdata.pl starts at 01:32 local time each night at SLAC
- The data is copied from the MA to anonymous FTP incoming space at then proxy.pl, needs to be scheduled to copy the data from anonymous FTP directory to the directory accessible by the ping_data.pl CGI script.
- This also has to complete before getdata.pl is scheduled,
- proxy.pl takes < 5 seconds to execute.
- proxy.pl is therefore currently scheduled to run at01at 01:20amCalifornialocaltime
- Once the move is completed by proxy.pl then getdata.pl can be scheduled to gather and save the selected data from the MAs in the PingER raw data archive at:
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
- This (getdata.pl) takes about 15 minutes.
- getdata.pl is scheduled to run at01:32amlocaltimeat 01:32 am local time.
The analysis of the hourly data by analyze-all.pl needs to start after getdata.pl has completed. Currentlyanalyze-all.pl starts as a cron job at 55 minutes past 2 am local time at SLAC each morning. The scheduling of the jobs at SLAC is shown below:
321* * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err552* * * /afs/slac/package/pinger/analysis/analyze-all.pl --date 1days #Takes25:1410/20/2011(55mins9/21/2011,70minutes5/11/2018)
...
The CGI script ping_data.pl provides remote access to PingER information at the MA. Besides providing access to the measured data, this also provides validation and debugging information (e.g. logging information from execution of pinger2.pl, the pinger.xml configuration file, the version of pinger2.pl running, the pinger.pl environment at the MA, etc). However, it is not possible to access the Beijing website (or a PingER Android) from outside. Thus we need a solution to provide access to the various information that ping_data.pl provides for the MA. This includes (where for example:
...
The Androids do not have a web server so using ping_data.pl to remotely look at the information in gthe the MNA is not going to work. Thus we cannit use ping_data.pl in the process to copy the relevant information to the proxy, so some new code would be needed to copy the files to the proxy. To simplify this we note below the locations of various files in the table below.
...