...
Introduction
CurrentlyOriginally (prior to June 2018), the gathering of data from the MAs is was initiated by the SLAC end.
- However, access to via a web server at the Beijing MA run by Guangzhou is was not allowed.
- Since Also since the MAs from the PingER Android project do did not want to have to run a web server, they need a proxy to needed a proxy to copy their data to. This proxy, in turn, has to be accessible from SLAC.
Possible solution
...
Solution
Since we cannot could not initiate the gathering from the SLAC end, a possibility might be was to use the anonymous ftp service at SLAC (see https://www.slac.stanford.edu/comp/unix/ftp.html).
1. In this case, the MA (e.g. a script initiated at the Beijing MA and run at Beijing by say user Ann who does not have an account at SLAC) would daily run runs a script (as a cronjob) to use the incoming anonymous FTP at SLAC to store the day's data at SLAC.
At most MAs (with the exception of SLAC), the data is stored in /usr/local/share/pinger/data, e.g.
Code Block file: /usr/local/share/pinger/data found on dxmon3.cern.ch total 66756 -rw-r--r-- 1 root root 12244303 Jun 7 18:56 ping-2018-06.txt -rw-r--r-- 1 root root 56049692 Jun 1 01:55 ping-2018-05.txt
The data from the MA is copied to a directory associated with the MA and read-write permissions set for group sf, e.g.
Code Block 169cottrell@pinger:~$ls -l /afs/slac/public/incoming/2001:da8:270:2018:f816:3eff:fef3:bd3/ total 23277 -rw-rw-r-- 1 saqibali sf 9360738 Jun 6 20:20 ping-2018-04.txt -rw-rw-r-- 1 saqibali sf 12394002 Jun 6 20:20 ping-2018-05.txt -rw-rw-r-- 1 saqibali sf 2079565 Jun 6 20:01 ping-2018-06.txt
Since the user Ann does not have a SLAC user account this is done via anonymous ftp to ftp://ftp.slac.stanford.edu/incoming. The first time Ann will need to create the directory for her site (e.g. as seen above 2001:da8:270:2018:f816:3eff:fef3:bd3)
...
However, since due to security concerns*, we cannot access anonymous incoming FTP space from a web CGI script such as ping_data.pl so we first we need to copy the data from the FTP anonymous space via a local (non web job) to a space accessible via the web
- This is done by the script proxy.pl which copies the files from /afs/slac.stanford.edu/public/incoming/pinger/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/ to /nfs/slac/g/net/pinger/pingerdata/hep/data/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/
Then the getdata.pl script that runs nightly at SLAC calls ping_data.pl via wget to gather the data for all the active MAs.
If this is done manually the SLAC account will need permissions to create a directory in /nfs/slac/g/net/pinger/pingerdata/hep/data/. This directory has permissions for user pinger and group iepm
Code Block 176cottrell@pinger:~$ls -ld /nfs/slac/g/net/pinger/pingerdata/hep/data drwxrwsr-x 220 pinger iepm 32768 May 16 00:45 /nfs/slac/g/net/pinger/pingerdata/hep/data/
To add the user to the iepm group
Code Block 178cottrell@pinger:~$ypgroup adduser -group iepm -u saqibali Jun 7 10:30:33 2018 User(s) 'saqibali' added to group 'iepm' Jun 7 10:30:33 2018 User(s) must logout/login for this to take effect. Jun 7 10:30:33 2018 Starting NIS post actions. Jun 7 10:30:33 2018 [/usr/ccs/bin/make group] copied group to /afs/slac/service/admin/NIS updated group pushed group 179cottrell@pinger:~$ypgroup exam -group iepm Group 'iepm': GID: 2087 Comment: Last modified at Jun 7 10:30:32 2018 by cottrell Owners: cottrell Members: cottrell, iepm, pinger, saqibali, ytl This is a secondary group.
Automating
We need to automate the steps The steps are automated with synchronized cronjobs.
- A job at Beijing is required to do the anonymous ftp of the recent data from the Beijing MA to the incoming FTP server at SLAC.
- The data then needs to be copied (by proxy.pl which takes a couple of seconds to run) from the anonymous FTP incoming space (/afs/slac/public/incoming/) to a directory accessible (see above) for reading from the ping_data.pl pl web CGI script that is called by wget from getdata.pl.
getdata.pl is then scheduled to run at SLAC to copy the selected data from the accessible directory
/afs/slac/public/incoming/pinger6.beijing.cnCode Block 32 0 * * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err proxy.pl
The various jobs have to be synchronized:
- The copying of data to the anonymous FTP server needs to complete before getdata.pl starts at 32 minutes past midnight local time each night at SLAC
- Once the data is copied to anonymous FTP incoming space then proxy.pl, needs to be scheduled to copy the data from anonymous FTP directory to the directory accessible by the ping_data.pl CGI script.
- This also has to complete before getdata.pl is scheduled
- Once the copy is completed then getdata.pl can be scheduled to gather and save the selected data to
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
- This (getdata.pl) takes about 15 minutes
The analysis of the hourly data by wrap-analyze-all.pl needs to start after getdata.pl has completed. Currently wrap-analyze-all.pl starts as a cron job at 11 minutes past 1am local time at SLAC each morning.
Code Block 11 1 * * * /usr/local/bin/bsub -W 180 -o /dev/null -q long /afs/slac/package/pinger/analysis/wrap-analyze-allmonths.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep 2>&1 #Takes ~ 2hr 15 min on pinger 11/11/2012.
...