to Introduction
Currently, the gathering of data from the MAs is initiated by the SLAC end.
...
At most MAs (with the exception of SLAC), the data is stored in /usr/local/share/pinger/data, e.g.
Code Block file: /usr/local/share/pinger/data found on dxmon3.cern.ch total 66756 -rw-r--r-- 1 root root 12244303 Jun 7 18:56 ping-2018-06.txt -rw-r--r-- 1 root root 56049692 Jun 1 01:55 ping-2018-05.txt
The data from the MA is copied to a directory associated with the MA and read-write permissions set for group sf, e.g.
Code Block 169cottrell@pinger:~$ls -l /afs/slac/public/incoming/2001:da8:270:2018:f816:3eff:fef3:bd3/ total 23277 -rw-rw-r-- 1 saqibali sf 9360738 Jun 6 20:20 ping-2018-04.txt -rw-rw-r-- 1 saqibali sf 12394002 Jun 6 20:20 ping-2018-05.txt -rw-rw-r-- 1 saqibali sf 2079565 Jun 6 20:01 ping-2018-06.txt
Since the user Ann does not have a SLAC user account this is done via anonymous ftp to ftp://ftp.slac.stanford.edu/incoming. The first time Ann will need to create the directory for her site (e.g. pinger.6.beijing.cn2001:da8:270:2018:f816:3eff:fef3:bd3)
2. Then it is necessary to move the data from the anonymous FTP server to:
| Code Block |
|---|
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2018-06-04.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/2001:da8:270:2018:f816:3eff:fef3:bd3/ping-2018-06-04.txt.gz #This name is determined by the Node Name in NODEDETAILS |
However, since , due to security concerns*, we cannot access anonymous incoming FTP space from a web CGI script such as ping_data.pl so first we need to copy the data from the FTP anonymous space via a local (non web job) to a space accessible via the web
- This is done by the script proxy.pl which copies the files from /afs/slac.stanford.edu/public/incoming/pinger/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/ to /nfs/slac/g/net/pinger/pingerdata/hep/data/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/
Then the getdata automated by the ping_data.pl script that runs nightly at SLAC calls ping_data.pl via wget to gather the data for all the active MAs.
If this is done manually the SLAC account will need permissions to create a directory in /nfs/slac/g/net/pinger/pingerdata/hep/data/. This directory has permissions for user pinger and group iepm
Code Block 176cottrell@pinger:~$ls -ld /nfs/slac/g/net/pinger/pingerdata/hep/data drwxrwsr-x 220 pinger iepm 32768 May 16 00:45 /nfs/slac/g/net/pinger/pingerdata/hep/data/
To add the user to the iepm group
Code Block 178cottrell@pinger:~$ypgroup adduser -group iepm -u saqibali Jun 7 10:30:33 2018 User(s) 'saqibali' added to group 'iepm' Jun 7 10:30:33 2018 User(s) must logout/login for this to take effect. Jun 7 10:30:33 2018 Starting NIS post actions. Jun 7 10:30:33 2018 [/usr/ccs/bin/make group] copied group to /afs/slac/service/admin/NIS updated group pushed group 179cottrell@pinger:~$ypgroup exam -group iepm Group 'iepm': GID: 2087 Comment: Last modified at Jun 7 10:30:32 2018 by cottrell Owners: cottrell Members: cottrell, iepm, pinger, saqibali, ytl This is a secondary group.
Automating
Then we We need to automate the steps with synchronized cronjobs.
- A job at Beijing is required to do the anonymous ftp of the recent data from the Beijing MA to the incoming FTP server at SLAC.
- The data then needs to be copied (by proxy.pl) from the anonymous FTP incoming space (/afs/slac/public/incoming/) to a directory accessible for
...
- reading from the ping_data.pl web CGI script that is called by wget from getdata.pl.
getdata.pl is then scheduled to run at SLAC to copy the selected data
...
from the accessible directory
/afs/slac/public/incoming/pinger6.beijing.cnCode Block 32 0 * * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err
Getdata.pl uses the web CGI script ping_data.pl and wget to get the relevant data.
The various jobs have to be synchronized:
- The copying of data to the anonymous FTP server needs to complete before getdata.pl starts at 32 minutes past midnight local time each night at SLAC
- Once the data ic is copied to anonymous FTP incoming space then small cron job then proxy.pl, needs to be scheduled to copy the data from anonymous FTP directory (/afs/slac/pub;ic/incoming/2001:da8:270:2018:f816:3eff:fef3:bd3) to a to the directory accessible by the ping_data.pl CGI script.
- This also has to complete before getdata.pl is scheduled
- Once the copy is completed then getdata.pl can be scheduled to gather and save the selected data to
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
- This takes about 15 minutes
The analysis of the hourly data by wrap-analyze-all.pl needs to start after getdata.pl has completed.
Currently wrap-analyze-all.pl starts as a cron job at 11 minutes past 1am local time at SLAC each morning.
Code Block 11 1 * * * /usr/local/bin/bsub -W 180 -o /dev/null -q long /afs/slac/package/pinger/analysis/wrap-analyze-allmonths.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep 2>&1 #Takes ~ 2hr 15 min on pinger 11/11/2012.
See https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC for the data flow. It appears getdata.pl should only require a small modification to not ping pinger6.beijing.cn. Also ping_data.pl only requires a small mod to add a 'proxy' to the QUERY_STRING and to point to the location of the relevant data. This will be The use of the proxy is triggered by the NODEDETAIL Data Server entry (e.g. ftp://ftp.slac.stanford.edu/incoming/pinger6.beijing.cnproxy=1) to tell ping_data.pl to gather the data from the SLAC anonymous ftp server (via the copy)that is basically acting as a proxy.
...