FTP Space

The FTP space is at: ftp://ftp.slac.stanford.edu/users/cottrell/. The equivalent local file space at  SLAC is /afs/slac/public/users/slac

Downloading recent data from the FTP:

The idea is that you retrieve all the existing Archived PingER hourly data from Anonymous FTP one time and then incrementally append recent data on say a daily basis. To append recent data proceed as follows:

There is a file called download.pl in the FTP. This file can be used to download recent data from the FTP.

  1. Download the file from the FTP, i.e. copy it from ftp://ftp.slac.stanford.edu/users/cottrell/download.pl
  2. Copy download.pl  to the location you want to download the files (e.g. /home/Documents/PingER)
  3. Through your command line interface type: perl [path to download.pl]/download.pl -d [number of days] (make sure to have perl installed)
    1. -d specifies how many days to go back in time. For example -d 3 will download files related to the previous three days.

PS: If needed files for the a period larger than 60 days, they must be downloaded manually from the FTP: ftp://ftp.slac.stanford.edu/users/cottrell/, see Retrieving Archived PingER hourly data from Anonymous FTP

Downloading all the data from the FTP:

  1. Download the file from the FTP, i.e. copy it from ftp://ftp.slac.stanford.edu/users/cottrell/download_all.pl
  2. Copy download_all.pl  to the location you want to download the files (e.g. /home/Documents/PingER)
  3. Through your command line interface type: perl download_all.pl (make sure to have perl installed)

Updating the FTP

This section is only for the FTP maintainer at SLAC. 

The script responsible to update the public FTP site at SLAC is in the following location:

/afs/slac/package/pinger/ftp-update.pl

To run this script you must use the following command:

$ /opt/TWWfsw/bin/perl /afs/slac/package/pinger/ftp-update.pl

You can SSH to the pinger machine with the following command:

ssh pinger@pinger.slac.stanford.edu

Output from running the command ftp_update.pl:

It represents how many files were missing during the update process.

Cron Job to run ftp-upate.pl automatically each day

The script runs every morning at 4AM. To modify the cron job you should edit the trscrontab:

$ trscrontab -e

and edit the following line:

lnxcron;480 00 4 * * * /opt/TWWfsw/bin/perl /afs/slac/package/pinger/ftp-update.pl

Log Files:

Log files with missing files can be found at:

$ /afs/slac/public/users/cottrell/pinger_log_files AKA ftp://ftp.slac.stanford.edu/users/cottrell/pinger_log_files/

Each .txt file contains the names of the files missing in the FTP.

The log files are generated automatically by ftp-update.pl everytime a file is missing in the pingerreports folder.

The files also can be generated by the following script:

$ /opt/TWWfsw/bin/perl /afs/slac/package/pinger/ftp-update-log.pl

 Algorithm

 Foreach metric
	Copy correspondent tar file to pinger_temp_files folder
	Untar the file inside pinger_temp_files folder 
	Check the most recent file in the recently untared folder
	Copy from pingerreports folder to the pinger_temp_files folder all the correspondent files until the current previous day
	Tar the updated folder
	Copy back to the FTP main folder
 

ftp main folder folder:

$ /afs/slac/public/users/cottrell AKA ftp://ftp.slac.stanford.edu/users/cottrell/

pinger_temp_files folder:

$ /afs/slac/public/users/cottrell/pinger_temp_files AKA ftp://ftp.slac.stanford.edu/users/cottrell/pinger_temp_files/

pingerreports folder:

$ /nfs/slac/g/net/pinger/pingerreports/hep

 

 

  • No labels