Currently, the gathering of data from the MAs is initiated by the SLAC end.
Initially, this is proposed for the Beijing MA.
Since we cannot initiate the gathering from SLAC, a possibility might be to use the anonymous ftp service at SLAC (see https://www.slac.stanford.edu/comp/unix/ftp.html).
1. In this case, the MA (e.g. a script initiated at the Beijing MA and run at Beijing by say user Ann who does not have an account at SLAC) would daily run a script (as a cronjob) to use the incoming anonymous FTP at SLAC to store the day's data at SLAC.
At most MAs (with the exception of SLAC), the data is stored in /usr/local/share/pinger/data, e.g.
file: /usr/local/share/pinger/data found on dxmon3.cern.ch total 66756 -rw-r--r-- 1 root root 12244303 Jun 7 18:56 ping-2018-06.txt -rw-r--r-- 1 root root 56049692 Jun 1 01:55 ping-2018-05.txt |
The data from the MA is copied to a directory associated with the MA and read-write permissions set for group sf, e.g.
169cottrell@pinger:~$ls -l /afs/slac/public/incoming/2001:da8:270:2018:f816:3eff:fef3:bd3/ total 23277 -rw-rw-r-- 1 saqibali sf 9360738 Jun 6 20:20 ping-2018-04.txt -rw-rw-r-- 1 saqibali sf 12394002 Jun 6 20:20 ping-2018-05.txt -rw-rw-r-- 1 saqibali sf 2079565 Jun 6 20:01 ping-2018-06.txt |
Since the user Ann does not have a SLAC user account this is done via anonymous ftp to ftp://ftp.slac.stanford.edu/incoming. The first time Ann will need to create the directory for her site (e.g. 2001:da8:270:2018:f816:3eff:fef3:bd3)
2. Then it is necessary to move the data from the anonymous FTP server to:
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2018-06-04.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/2001:da8:270:2018:f816:3eff:fef3:bd3/ping-2018-06-04.txt.gz #This <host> name is determined by the Node Name in NODEDETAILS |
However, since due to security concerns*, we cannot access anonymous incoming FTP space from a web CGI script such as ping_data.pl so first we need to copy the data from the FTP anonymous space via a local (non web job) to a space accessible via the web
Then the getdata.pl script that runs nightly at SLAC calls ping_data.pl via wget to gather the data for all the active MAs.
If this is done manually the SLAC account will need permissions to create a directory in /nfs/slac/g/net/pinger/pingerdata/hep/data/. This directory has permissions for user pinger and group iepm
176cottrell@pinger:~$ls -ld /nfs/slac/g/net/pinger/pingerdata/hep/data drwxrwsr-x 220 pinger iepm 32768 May 16 00:45 /nfs/slac/g/net/pinger/pingerdata/hep/data/ |
To add the user to the iepm group
178cottrell@pinger:~$ypgroup adduser -group iepm -u saqibali
Jun 7 10:30:33 2018 User(s) 'saqibali' added to group 'iepm'
Jun 7 10:30:33 2018 User(s) must logout/login for this to take effect.
Jun 7 10:30:33 2018 Starting NIS post actions.
Jun 7 10:30:33 2018 [/usr/ccs/bin/make group]
copied group to /afs/slac/service/admin/NIS
updated group
pushed group
179cottrell@pinger:~$ypgroup exam -group iepm
Group 'iepm':
GID: 2087
Comment:
Last modified at Jun 7 10:30:32 2018 by cottrell
Owners: cottrell
Members: cottrell, iepm, pinger, saqibali, ytl
This is a secondary group. |
We need to automate the steps with synchronized cronjobs.
getdata.pl is then scheduled to run at SLAC to copy the selected data from the accessible directory/afs/slac/public/incoming/pinger6.beijing.cn to /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger6.beijing.cn/. Getdata.pl runs at 32 minutes past midnight local time at SLAC each morning.
32 0 * * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err |
The various jobs have to be synchronized:
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
The analysis of the hourly data by wrap-analyze-all.pl needs to start after getdata.pl has completed. Currently wrap-analyze-all.pl starts as a cron job at 11 minutes past 1am local time at SLAC each morning.
11 1 * * * /usr/local/bin/bsub -W 180 -o /dev/null -q long /afs/slac/package/pinger/analysis/wrap-analyze-allmonths.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep 2>&1 #Takes ~ 2hr 15 min on pinger 11/11/2012. |
See https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC for the data flow. The use of the proxy is triggered by the NODEDETAIL Data Server entry (e.g. proxy=1) to tell ping_data.pl to gather the data from the SLAC anonymous ftp server (via the copy)that is basically acting as a proxy.
This whole mechanism is also interesting since it also could be extended and a step to providing support for the Android PingER project at Amity in Delhi, India which also needs a proxy.