• Created by Unknown User (jlrulff), last modified on Aug 29, 2016

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

FTP Space

The FTP space is at: ftp://ftp.slac.stanford.edu/users/cottrell/. The equivalent local file space at  SLAC is /afs/slac/public/users/slac

Downloading recent data from the FTP:

The idea is that you retrieve all the existing Archived PingER hourly data from Anonymous FTP one time and then incrementally append recent data on say a daily basis. To append recent data proceed as follows:

There is a file called download.pl in the FTP. This file can be used to download recent data from the FTP.

  1. Download the file from the FTP, i.e. copy it from ftp://ftp.slac.stanford.edu/users/cottrell/download.pl
  2. Copy download.pl  to the location you want to download the files (e.g. /home/Documents/PingER)
  3. Through your command line interface type: perl [path to download.pl]/download.pl -d [number of days] (make sure to have perl installed)
    1. -d specifies how many days to go back in time. For example -d 3 will download files related to the previous three days.

PS: If needed files for the a period larger than 60 days, they must be downloaded manually from the FTP: ftp://ftp.slac.stanford.edu/users/cottrell/, see Retrieving Archived PingER hourly data from Anonymous FTP

Downloading all the data from the FTP:

 

Updating the FTP

This section is only for the FTP maintainer at SLAC. 

The script responsible to update the public FTP site at SLAC is in the following location:

/afs/slac/package/pinger/ftp-update.pl

To run this script you must use the following command:

$ /opt/TWWfsw/bin/perl /afs/slac/package/pinger/ftp-update.pl

You can SSH to the pinger machine with the following command:

ssh pinger@pinger.slac.stanford.edu

Output from running the command ftp_update.pl:

It represents how many files were missing during the update process.

Cron Job to run ftp-upate.pl automatically each day

The script runs every morning at 4AM. To modify the cron job you should edit the trscrontab:

$ trscrontab -e

and edit the following line:

lnxcron;480 00 4 * * * /opt/TWWfsw/bin/perl /afs/slac/package/pinger/ftp-update.pl

Log Files:

Log files with missing files can be found at:

$ /afs/slac/public/users/cottrell/pinger_log_files AKA ftp://ftp.slac.stanford.edu/users/cottrell/pinger_log_files/

Each .txt file contains the names of the files missing in the FTP.

The log files are generated automatically by ftp-update.pl everytime a file is missing in the pingerreports folder.

The files also can be generated by the following script:

$ /opt/TWWfsw/bin/perl /afs/slac/package/pinger/ftp-update-log.pl

 Algorithm

 Foreach metric
	Copy correspondent tar file to pinger_temp_files folder
	Untar the file inside pinger_temp_files folder 
	Check the most recent file in the recently untared folder
	Copy from pingerreports folder to the pinger_temp_files folder all the correspondent files until the current previous day
	Tar the updated folder
	Copy back to the FTP main folder
 

ftp main folder folder:

$ /afs/slac/public/users/cottrell AKA ftp://ftp.slac.stanford.edu/users/cottrell/

pinger_temp_files folder:

$ /afs/slac/public/users/cottrell/pinger_temp_files AKA ftp://ftp.slac.stanford.edu/users/cottrell/pinger_temp_files/

pingerreports folder:

$ /nfs/slac/g/net/pinger/pingerreports/hep

 

 

  • No labels