Versions Compared

Old Version 16

changes.mady.by.user Les Cottrell

Saved on

compared with

New Version 17

changes.mady.by.user Les Cottrell

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. At most MAs (with the exception of SLAC), the data is stored in /usr/local/share/pinger/data, e.g.

    Code Block
    file: /usr/local/share/pinger/data found on dxmon3.cern.ch 
total 66756
-rw-r--r-- 1 root root 12244303 Jun  7 18:56 ping-2018-06.txt
-rw-r--r-- 1 root root 56049692 Jun  1 01:55 ping-2018-05.txt

  2. The data from the MA is  is copied to a directory associated with the MA  and read-write permissions set for group sf. We chose to use the FTP anonymous incoming directory path /afs/slac/public/incoming/pinger/proxy/ and this was set up. The PingER MA then copies the data to the above directory with the PingER MA host name subdirectory, e.g.

    Code Block
    169cottrell@pinger:~$ls -l /afs/slac/public/incoming/pinger/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/
total 23277
-rw-rw-r-- 1 saqibali sf  9360738 Jun  6 20:20 ping-2018-04.txt
-rw-rw-r-- 1 saqibali sf 12394002 Jun  6 20:20 ping-2018-05.txt
-rw-rw-r-- 1 saqibali sf  2079565 Jun  6 20:01 ping-2018-06.txt

    Since the user Ann does not have a SLAC user account this is done via anonymous ftp to ftp://ftp.slac.stanford.edu/incoming.  The first time Ann will need to create the directory for her site (e.g. as seen above 2001:da8:270:2018:f816:3eff:fef3:bd3).

2. Then it is necessary to move (we use a move the data since one cannot replace an existing file via the anonymous FTP server -one gets a message 'Overwrite permission denied) the file from the anonymous incoming FTP server to:space to the regular PingER raw data archive, e.g.

Code Block
/nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
e.g.
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2018-06-04.txt.gz
/nfs/slac/g/net/pinger/pingerdata/hep/data/2001:da8:270:2018:f816:3eff:fef3:bd3/ping-2018-06-04.txt.gz #This <host> name is determined by the Node Name in NODEDETAILS

  • However, since due to security concerns*, we cannot access anonymous incoming FTP space from a web CGI script such as ping_data.pl we first copy move the data from the FTP anonymous space via a local (non web job) to a space accessible via the web

  • Then the getdata.pl script that runs nightly at SLAC calls ping_data.pl via wget to gather the data for all the active MAs.

  • If this is done manually the SLAC account will need permissions to create a directory in /nfs/slac/g/net/pinger/pingerdata/hep/data/. This directory has permissions for user pinger and group iepm

    Code Block
    176cottrell@pinger:~$ls -ld /nfs/slac/g/net/pinger/pingerdata/hep/data
drwxrwsr-x 220 pinger iepm 32768 May 16 00:45 /nfs/slac/g/net/pinger/pingerdata/hep/data/

  • To add the user to the iepm group

    Code Block
    178cottrell@pinger:~$ypgroup adduser -group iepm -u saqibali
Jun  7 10:30:33 2018 User(s) 'saqibali' added to group 'iepm'
Jun  7 10:30:33 2018 User(s) must logout/login for this to take effect.
Jun  7 10:30:33 2018 Starting NIS post actions.
Jun  7 10:30:33 2018 [/usr/ccs/bin/make group]
copied group to /afs/slac/service/admin/NIS
updated group
pushed group
179cottrell@pinger:~$ypgroup exam -group iepm
Group 'iepm':
        GID:     2087
        Comment:
        Last modified at Jun  7 10:30:32 2018 by cottrell
        Owners:  cottrell
        Members: cottrell, iepm, pinger, saqibali, ytl
        This is a secondary group.

...

  • The copying of data to the anonymous FTP server and moved from there to the PingER raw data archive needs to complete before getdata.pl starts at 32 minutes past midnight local time each night at SLAC
  • Once the data is copied to anonymous FTP incoming space then proxy.pl, needs to be scheduled to copy the data from anonymous FTP directory to the directory accessible by the ping_data.pl CGI script.
    • This also has to complete before getdata.pl is scheduled, 
    • proxy.pl takes < 5 seconds to execute. 
    • proxy.pl is scheduled to run at 20 minutes past midnight localtime
  • Once the copy move is completed then getdata.pl can be scheduled to gather and save the selected data to data in the PingER raw data archive at:
    /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
    • This (getdata.pl) takes about 15 minutes
    • getdata.pl is scheduled to run at 32 minutes past midnight localtime.

  • The analysis of the hourly data by wrap-analyze-all.pl needs to start after getdata.pl has completed. Currently wrap-analyze-all.pl starts as a cron job at 11 minutes past 1am local time at SLAC each morning.

    Code Block
    11 1 * * * /usr/local/bin/bsub -W 180 -o /dev/null -q long /afs/slac/package/pinger/analysis/wrap-analyze-allmonths.pl --basedir /nfs/slac/g/net/pinger --usemetric --dataset hep  2>&1 #Takes ~ 2hr 15 min on pinger 11/11/2012.

...