Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. At most MAs (with the exception of SLAC), the data is stored in /usr/local/share/pinger/data, e.g.

    Code Block
    file: /usr/local/share/pinger/data found on dxmon3.cern.ch 
    total 66756
    -rw-r--r-- 1 root root 12244303 Jun  7 18:56 ping-2018-06.txt
    -rw-r--r-- 1 root root 56049692 Jun  1 01:55 ping-2018-05.txt
  2. The data from the MA is  copied to a directory associated with the MA  and read-write permissions set for group sf, e.g.

    Code Block
    169cottrell@pinger:~$ls -l /afs/slac/public/incoming/pinger6.beijing.cn2001:da8:270:2018:f816:3eff:fef3:bd3/
    total 23277
    -rw-rw-r-- 1 saqibali sf  9360738 Jun  6 20:20 ping-2018-04.txt
    -rw-rw-r-- 1 saqibali sf 12394002 Jun  6 20:20 ping-2018-05.txt
    -rw-rw-r-- 1 saqibali sf  2079565 Jun  6 20:01 ping-2018-06.txt

    Since the user Ann does not have a SLAC user account this is done via anonymous ftp to ftp://ftp.slac.stanford.edu/incoming.  The first time Ann will need to create the directory for her site (e.g. pinger.6.beijing.cn)

...

Code Block
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
e.g.
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2018-06-04.txt.gz
/nfs/slac/g/net/pinger/pingerdata/hep/data/2001:da8:270:2018:f816:3eff:fef3:bd3/ping-2018-06-04.txt.gz #This name is determined by the Node Name in NODEDETAILS
  • However, since, due to security concerns*, we cannot access anonymous incoming FTP space from a web CGI script such as ping_data.pl so first we need to copy the data from the FTP anonymous space via a local (non web job) to a space accessible via the web

  • This is automated by the getdatathe ping_data.pl script that runs nightly at SLAC.

  • If this is done manually the SLAC account will need permissions to create a directory in /nfs/slac/g/net/pinger/pingerdata/hep/data/. This directory has permissions for user pinger and group iepm

    Code Block
    176cottrell@pinger:~$ls -ld /nfs/slac/g/net/pinger/pingerdata/hep/data
    drwxrwsr-x 220 pinger iepm 32768 May 16 00:45 /nfs/slac/g/net/pinger/pingerdata/hep/data/
  • To add the user to the iepm group

    Code Block
    178cottrell@pinger:~$ypgroup adduser -group iepm -u saqibali
    Jun  7 10:30:33 2018 User(s) 'saqibali' added to group 'iepm'
    Jun  7 10:30:33 2018 User(s) must logout/login for this to take effect.
    Jun  7 10:30:33 2018 Starting NIS post actions.
    Jun  7 10:30:33 2018 [/usr/ccs/bin/make group]
    copied group to /afs/slac/service/admin/NIS
    updated group
    pushed group
    179cottrell@pinger:~$ypgroup exam -group iepm
    Group 'iepm':
            GID:     2087
            Comment:
            Last modified at Jun  7 10:30:32 2018 by cottrell
            Owners:  cottrell
            Members: cottrell, iepm, pinger, saqibali, ytl
            This is a secondary group.

...

Then we need to automate the steps with synchronized cronjobs. A job at Beijing is required to do the anonymous ftp of the recent data from the Beijing MA to the incoming FTP server at SLAC. The data then needs to be copied from the anonymous FTP incoming space (/afs/slac/public/incoming/to a directory accessible for read from the ping_data.pl  web CGI script that is called by wget from getdata.pl.

getdata.pl is then scheduled to After the anonymous ftp copy is complete then getdata.pl will run at SLAC to copy the selected data data from /afs/slac/public/incoming/pinger6.beijing.cn to /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger6.beijing.cn/. Getdata.pl runs at 32 minutes past midnight local time at SLAC each morning.

Code Block
32 0 * * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err 

Getdata.pl uses the web CGI script ping_data.pl and wget to get the relevant data.

The various jobs have to be synchronized:

  • The copying of data to the anonymous FTP server needs to complete before getdatabefore getdata.pl starts at 32 minutes past midnight local time each night at SLAC
  • Once the data ic copied to anonymous FTP incoming space then  small cron job needs to be scheduled to copy the data from anonymous FTP directory (/afs/slac/pub;ic/incoming/2001:da8:270:2018:f816:3eff:fef3:bd3) to a directory accessible by the ping_data.pl CGI script.
    • This also has to complete before getdata.
    Getdata.pl
    • pl is scheduled
  • Once the copy is completed then getdata.pl can be scheduled save the selected data to 
    /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
    • This takes about 15 minutes
  • The and the analysis of the hourly data by wrap-analyze-all.pl needs to start after getdata.pl has completed. 

Currently wrap-analyze-all.pl starts as a cron job at 11 minutes past 1am local time at SLAC each morning. 

Code Block
11 1 * * * /usr/local/bin/bsub -W 180 -o /dev/null -q long /afs/slac/package/pinger/analysis/wrap-analyze-allmonths.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep  2>&1 #Takes ~ 2hr 15 min on pinger 11/11/2012.


See https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC for the data flow. It appears getdata.pl should only require a small modification to not ping pinger6.beijing.cn. For the rest it will use the . Also ping_data.pl only requires a small mod to add a 'proxy' to the QUERY_STRING and to point to the location of the relevant data. This will be triggered by the  NODEDETAIL Data Server entry (e.g. ftp://ftp.slac.stanford.edu/incoming/pinger6.beijing.cn) to tell ping_data.pl to gather the data from the SLAC anonymous ftp server (via the copy)that is basically acting as a proxy. 

This whole mechanism is also interesting since it also could be extended and a step to providing support for the Android PingER project at Amity in Delhi, India which also needs a proxy.

...

  • If an anonymous ftp server allows anonymous upload, and also world read access, it quickly becomes a haven for stolen software distribution.