...
- This is done by the script proxy.pl which moves** the files from /afs/slac.stanford.edu/public/incoming/pinger/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/ to /nfs/slac/g/net/pinger/pingerdata/hep/data/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/
- the node name (in this case 2001:da8:270:2018:f816:3eff:fef3:bd3) is derived from NODEDETAILS.
- we use a move since one cannot replace an existing file via the anonymous FTP server (one gets a message 'Overwrite permission denied'). Also it ensures we do not fill up the FTP data space.
Then the getdata.pl script runs nightly at SLAC and calls ping_data.pl via wget to gather the data for all the active MAs (including the proxies). Getdata.pl saves the gathered data in:
Code Block /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>.txt /nfs/slac/g/net/pinger/pingerdata/hep/data/2001:da8:270:2018:f816:3eff:fef3:bd3/ping-2018-06.txt
If this is done manually the SLAC account will need permissions to create a directory in /nfs/slac/g/net/pinger/pingerdata/hep/data/. This directory has permissions for user pinger and group iepm
Code Block 176cottrell@pinger:~$ls -ld /nfs/slac/g/net/pinger/pingerdata/hep/data drwxrwsr-x 220 pinger iepm 32768 May 16 00:45 /nfs/slac/g/net/pinger/pingerdata/hep/data/
To add the user to the iepm group
Code Block 178cottrell@pinger:~$ypgroup adduser -group iepm -u saqibali Jun 7 10:30:33 2018 User(s) 'saqibali' added to group 'iepm' Jun 7 10:30:33 2018 User(s) must logout/login for this to take effect. Jun 7 10:30:33 2018 Starting NIS post actions. Jun 7 10:30:33 2018 [/usr/ccs/bin/make group] copied group to /afs/slac/service/admin/NIS updated group pushed group 179cottrell@pinger:~$ypgroup exam -group iepm Group 'iepm': GID: 2087 Comment: Last modified at Jun 7 10:30:32 2018 by cottrell Owners: cottrell Members: cottrell, iepm, pinger, saqibali, ytl This is a secondary group.
...
- The copying of data to the anonymous FTP server and moving from there to the PingER raw data archive needs to complete before getdata.pl starts at 01:32 local time each night at SLAC
- The data is copied from the MA to anonymous FTP incoming space at then proxy.pl, needs to be scheduled to copy the data from anonymous FTP directory to the directory accessible by the ping_data.pl CGI script.
- This also has to complete before getdata.pl is scheduled,
- proxy.pl takes < 5 seconds to execute.
- proxy.pl is therefore currently scheduled to run at 01:20am California localtime
- Once the move is completed by proxy.pl then getdata.pl can be scheduled to gather and save the selected data from the MAs in the PingER raw data archive at:
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
- This (getdata.pl) takes about 15 minutes.
- getdata.pl is scheduled to run at 01:32am localtime.
The analysis of the hourly data by analyze-all.pl needs to start after getdata.pl has completed. Currently analyze-all.pl starts as a cron job at 55 minutes past 2 am local time at SLAC each morning. the schedling Theschedulingg of the jobs at SLAC is shown below;
Code Block 20 1 * * * /afs/slac/package/pinger/proxy.pl 32 1 * * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err 55 2 * * * /afs/slac/package/pinger/analysis/analyze-all.pl --date 1days #Takes 25:14 10/20/2011 (55 mins 9/21/2011, 70 minutes 5/11/2018)The analyzed data from analyze-all.pl is saved in files of the form below, the contents are described in PingER data flow at SLAC:
Code Block /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/<metric>-<len>-<by>-<year>-<month>-<day>.txt.gz#len=100|1000, by=by-node|by-site. /nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz
...
pa+flow+at+
...
SLAC for the data flow. The use of the proxy is triggered by the NODEDETAIL Data Server entry (e.g.
...
proxy=1) to tell ping_data.pl to gather the data from the SLAC anonymous ftp server (via the copy)that is basically acting as a proxy.
This whole mechanism is also interesting since it could be extended and a step to providing support for the Android PingER project at Amity in Delhi, India which also needs a proxy.
...
'* If an anonymous ftp server allows anonymous upload, and also world read access, it quickly becomes a haven for stolen software distribution
'** It moves all the files (but not ./ and ../) from for example /afs/slac/public/incoming/pinger/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/. Hence after a successful move the files will no longer be in /afs/slac/public/incoming/pinger/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/ but should be in /nfs/slac/g/net/pinger/pingerdata/hep/data/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/