Versions Compared

Old Version 30

changes.mady.by.user Les Cottrell

Saved on

compared with

New Version 31

changes.mady.by.user Les Cottrell

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • The copying of data to the anonymous FTP server and moving from there to the PingER raw data archive needs to complete before getdatabeforegetdata.pl starts at 01:32 local time each night at SLAC
  • The data is copied from the MA to anonymous FTP incoming space at  then proxyspaceat  thenproxy.pl, needs to be scheduled to copy the data from anonymous FTP directory to the directory accessible by the ping_data.pl CGI script.
    • This also has to complete before getdata.pl is scheduled, 
    • proxy.pl takes < 5 seconds to execute. 
    • proxy.pl is therefore currently scheduled to run at 01:20am California localtimeat01:20amCalifornialocaltime
  • Once the move is completed by proxy.pl then getdata.pl can be scheduled to gather and save the selected data from the MAs in the PingER raw data archive atarchiveat:
    /nfs/slac/g/net/pinger/pingerdata/hep/data/<host>/ping-<YYYY>-<MM>-<DD>.txt.gz
    • This(getdata.pl) takes about 15 minutes.
    • getdata.pl is scheduled to run at 01at01:32am localtime32amlocaltime.

  • The analysis of the hourly data by analyze-all.pl needs to start after getdata.pl has completed. Currently analyzeCurrentlyanalyze-all.pl starts as a cron job at 55 minutes past 2 am local time at SLAC each morning. Theschedulingg of Theschedulinggof the jobs at SLAC is shown below;:

    20 1 * * * /afs/slac/package/pinger/proxy.pl
    32 1 * * * /afs/slac/package/pinger/getdata.pl > /afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err
    55 2 * * * /afs/slac/package/pinger/analysis/analyze-all.pl --date 1days #Takes 25:1410/20/2011(55mins 9/21/2011, 70minutes 5/11/2018)

 

The analyzed data from analyze-all.pl is saved in files of the form below, the contents are described in PingER data flow at SLAC:

 

/nfs/slac/g/net/pinger/pingerreports/hep/<metric>/<metric>-<len>-<by>-<year>-<month>-<day>.txt.gz#len=100|1000, by=by-node|by-site.
/nfs/slac/g/net/pinger/pingerreports/hep/minimum_rtt/minimum_rtt-100-by-node-2011-05-01.txt.gz

 

  • See https://confluence.slac.stanford.edu/display/IEPM/PingER+data+flow+at+SLAC for the data flow. The use of the proxy is triggered by the  NODEDETAIL Data Server entry (e.g. proxy=1) to tell ping_data.pl to gather the data from the SLAC anonymous ftp server (via the copy)that is basically acting as a

    Code Block20 1 * * * /afs/slac/pa+flow+at+SLAC for the data flow. The use of the proxy is triggered by the  NODEDETAIL Data Server entry (e.g. proxy=1) to tell ping_data.pl to gather the data from the SLAC anonymous ftp server (via the copy)that is basically acting as a

    proxy. 

This whole mechanism is also interesting since it could be extended and a step to providing support for the Android PingER project at Amity in Delhi, India which also needs a proxy.

...

'* If an anonymous ftp server allows anonymous upload, and also world read access, it quickly becomes a haven for stolen software distribution

'**  It moves all the files (

...

except ./ and ../) from for example /afs/slac/public/incoming/pinger/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/. Hence after a successful move the files will no longer be in /afs/slac/public/incoming/pinger/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/ but should be in /nfs/slac/g/net/pinger/pingerdata/hep/data/proxy/2001:da8:270:2018:f816:3eff:fef3:bd3/