Introduction
Currently, the gathering of data from the MAs is initiated by the SLAC end.
- However, access to the Beijing MA run by Guangzhou is not allowed.
- Since the MAs from the PingER Android project do not want to have to run a web server, they need a proxy to copy their data to. This proxy, in turn, has to be accessible from SLAC.
Possible solution
Initially, this is proposed for the Beijing MA.
Since we cannot initiate the gathering from SLAC, a possibility might be to use the anonymous ftp service at SLAC (see https://www.slac.stanford.edu/comp/unix/ftp.html).
1. In this case, the MA (e.g. a script initiated at the Beijing MA and run at Beijing by say user Ann who does not have an account at SLAC) would daily run a script (as a cronjob) to use the incoming anonymous FTP at SLAC to store the day's data at SLAC.
At most (with the exception of SLAC) MAs, the data is stored in /usr/local/share/pinger/data, e.g.
file: /usr/local/share/pinger/data found on dxmon3.cern.ch total 66756 -rw-r--r-- 1 root root 12244303 Jun 7 18:56 ping-2018-06.txt -rw-r--r-- 1 root root 56049692 Jun 1 01:55 ping-2018-05.txt
The data from the MA is copied to a directory associated with the MA and read-write permissions set for group sf, e.g.
169cottrell@pinger:~$ls -l /afs/slac/public/incoming/pinger6.beijing.cn total 23277 -rw-rw-r-- 1 saqibali sf 9360738 Jun 6 20:20 ping-2018-04.txt -rw-rw-r-- 1 saqibali sf 12394002 Jun 6 20:20 ping-2018-05.txt -rw-rw-r-- 1 saqibali sf 2079565 Jun 6 20:01 ping-2018-06.txt
Since the user Ann does not have a SLAC user account this is done via anonymous ftp to ftp://ftp.slac.stanford.edu/incoming. The first time Ann will need to create the directory for her site (e.g. pinger.6.beijing.cn)
2. Then it is necessary to move the data from the anonymous FTP server to:
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz e.g. /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2018-06-04.txt.gz /nfs/slac/g/net/pinger/pingerdata/hep/data/2001:da8:270:2018:f816:3eff:fef3:bd3/ping-2018-06-04.txt.gz #This name is determined by the Node Name in NODEDETAILS
This is automated by the getdata.pl script that runs nightly at SLAC.
If this is done manually the SLAC account will need permissions to create a directory in /nfs/slac/g/net/pinger/pingerdata/hep/data/. This directory has permissions for user pinger and group iepm
176cottrell@pinger:~$ls -ld /nfs/slac/g/net/pinger/pingerdata/hep/data drwxrwsr-x 220 pinger iepm 32768 May 16 00:45 /nfs/slac/g/net/pinger/pingerdata/hep/data/
To add the user to the iepm group
178cottrell@pinger:~$ypgroup adduser -group iepm -u saqibali Jun 7 10:30:33 2018 User(s) 'saqibali' added to group 'iepm' Jun 7 10:30:33 2018 User(s) must logout/login for this to take effect. Jun 7 10:30:33 2018 Starting NIS post actions. Jun 7 10:30:33 2018 [/usr/ccs/bin/make group] copied group to /afs/slac/service/admin/NIS updated group pushed group 179cottrell@pinger:~$ypgroup exam -group iepm Group 'iepm': GID: 2087 Comment: Last modified at Jun 7 10:30:32 2018 by cottrell Owners: cottrell Members: cottrell, iepm, pinger, saqibali, ytl This is a secondary group.
Automating
Then we need to automate the steps with synchronized cronjobs. A job at Beijing is required to to do the anonymous ftp of the recent data from the Beijing MA to the incoming ftp server at SLAC. After the anonymous ftp copy is complete then getdata.pl will run at SLAC to copy the data from /afs/slac/public/incoming/pinger6.beijing.cn to /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger6.beijing.cn/. Getdata.pl runs at 32 minutes past midnight local time at SLAC each morning.
32 0 * * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err
Getdata.pl is synchronized with, so it is completed (it takes about 15 minutes to gather the data from the MAs) before the analysis of the hourly data by wrap-analyze-all, that currently runs as a cron job at 11 minutes past 1am local time at SLAC each morning.
11 1 * * * /usr/local/bin/bsub -W 180 -o /dev/null -q long /afs/slac/package/pinger/analysis/wrap-analyze-allmonths.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep 2>&1 #Takes ~ 2hr 15 min on pinger 11/11/2012.
See https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC for the data flow. It appears getdata.pl should only require a small modification to not ping pinger6.beijing.cn. For the rest it will use the NODEDETAIL Data Server entry (e.g. ftp://ftp.slac.stanford.edu/incoming/pinger6.beijing.cn) to gather the data from the SLAC anonymous ftp server that is basically acting as a proxy.
This whole mechanism is also interesting since it also could be extended and a step to providing support for the Android PingER project at Amity in Delhi, India which also needs a proxy.