Versions Compared

Old Version 21

changes.mady.by.user Les Cottrell

Saved on

compared with

New Version 22

changes.mady.by.user Les Cottrell

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • However, access to a web server at the Beijing MA run by Guangzhou was not allowed. 
  • Also since the MAs from the PingER Android project did not want to have to run a web server, they needed a proxy to copy their data to. This proxy, in turn, has to be accessible from SLAC.Solution

Since we could not initiate the gathering from the SLAC end, a possibility was we chose to use the anonymous ftp service at SLAC (see https://www.slac.stanford.edu/comp/unix/ftp.html).

  • Be aware data is only kept for 4 days in anonymous incoming FTP and then is removed.  The directories are preserved.  One can delete files and directories by using rm and rmdir on /afs/slac/public/incoming from the  command the  unix command line (one cannot delete files and directories within ftp).

1. In this case, the MA (e.g. a script initiated at the Beijing MA and run at Beijing by say user Ann who does not have an account at SLAC)  daily runs a script (as a cronjob) to use the incoming anonymous FTP at SLAC to store the day's data at SLAC. An example of an FTP session to copy a file is seen at the Proxy FTP script document.

...

  • This is done by the script proxy.pl which moves the files from  /afs/slac.stanford.edu/public/incoming/pinger/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/ to /nfs/slac/g/net/pinger/pingerdata/hep/data/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/
    • the node name (in this case 2001:da8:270:2018:f816:3eff:fef3:bd3) is dervied derived from NODEDETAILS.
    • we use a move since one cannot replace an existing file via the anonymous FTP server (one gets a message 'Overwrite permission denied').

  • Then the getdata.pl script runs nightly at SLAC and calls ping_data.pl via wget to gather the data for all the active MAs (including the proxies). Getdata.pl saves the gathered data in:

    Code Block
    /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>.txt
/nfs/slac/g/net/pinger/pingerdata/hep/data/2001:da8:270:2018:f816:3eff:fef3:bd3/ping-2018-06.txt

    If this is done manually the SLAC account will need permissions to create a directory in /nfs/slac/g/net/pinger/pingerdata/hep/data/. This directory has permissions for user pinger and group iepm

    Code Block
    176cottrell@pinger:~$ls -ld /nfs/slac/g/net/pinger/pingerdata/hep/data
drwxrwsr-x 220 pinger iepm 32768 May 16 00:45 /nfs/slac/g/net/pinger/pingerdata/hep/data/

  • To add the user to the iepm group

    Code Block
    178cottrell@pinger:~$ypgroup adduser -group iepm -u saqibali
Jun  7 10:30:33 2018 User(s) 'saqibali' added to group 'iepm'
Jun  7 10:30:33 2018 User(s) must logout/login for this to take effect.
Jun  7 10:30:33 2018 Starting NIS post actions.
Jun  7 10:30:33 2018 [/usr/ccs/bin/make group]
copied group to /afs/slac/service/admin/NIS
updated group
pushed group
179cottrell@pinger:~$ypgroup exam -group iepm
Group 'iepm':
        GID:     2087
        Comment:
        Last modified at Jun  7 10:30:32 2018 by cottrell
        Owners:  cottrell
        Members: cottrell, iepm, pinger, saqibali, ytl
        This is a secondary group.

...

  • The copying of data to the anonymous FTP server and moved from there to the PingER raw data archive needs to complete before getdata.pl starts at 32 minutes past midnight local time each night at SLAC
  • Once the data is copied from the MA to anonymous FTP incoming space then proxy.pl, needs to be scheduled to copy the data from anonymous FTP directory to the directory accessible by the ping_data.pl CGI script.
    • This also has to complete before getdata.pl is scheduled, 
    • proxy.pl takes < 5 seconds to execute. 
    • proxy.pl is therefore currently scheduled to run at 20 minutes past midnight localtime
  • Once the move is completed then getdata.pl can be scheduled to gather and save the selected data from the MAs in the PingER raw data archive at:
    /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
    • This (getdata.pl) takes about 15 minutes.
    • getdata.pl is scheduled to run at 32 minutes past midnight localtime.

  • The analysis of the hourly data by analyze-all.pl needs to start after getdata.pl has completed. Currently wrap- analyze-all.pl starts as a cron job at 11 minutes past 1 am local time at SLAC each morning.

    Code Block
    11 1 * * * /usr/local/bin/bsub -W 180 -o /dev/null -q long /afs/slac/package/pinger/analysis/wrap-analyze-allmonths.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep  2>&1 #Takes ~ 2hr 15 min on pinger 11/11/2012.

    The analyzed data from analyze-all.pl is saved in files of the form below, the contents are described in PingER data flow at SLAC:

    Code Block
    /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/<metric>-<len>-<by>-<year>-<month>-<day>.txt.gz#len=100|1000, by=by-node|by-site.
/nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz

    The analyzed data is saved in files of the form below, the contents are described in PingER data flow at SLAC:

    Code Block
    /nfs/slac/g/net/pinger/pingerreports/hep/<metric>/<metric>-<len>-<by>-<year>-<month>-<day>.txt.gz#len=100|1000, by=by-node|by-site.
/nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz

...