FTP space
The FTP space is at: ftp://ftp.slac.stanford.edu/users/cottrell/. The equivalent local file space at SLAC is /afs/slac/public/users/slac
Downloading recent data from the FTP:
The idea is that you download all the existing FTP PingER hourly data one time and then incrementally append recent data on say a daily basis. To append recent data proceed as follows:
There is a file called download.pl in the FTP. This file can be used to download recent data from the FTP.
- Download the file from the FTP, i.e. copy it from ftp://ftp.slac.stanford.edu/users/cottrell/download.pl
- Copy download.pl to the location you want to download the files (e.g. /home/Documents/PingER)
- Through your command line interface type: perl [path to download.pl]/download.pl -d [number of days] (make sure to have perl installed)
- -d specifies how many days to go back in time. For example -d 3 will download files related to the previous three days.
- -d specifies how many days to go back in time. For example -d 3 will download files related to the previous three days.
PS: If needed files for the a period larger than 60 days, they must be downloaded manually from the FTP: ftp://ftp.slac.stanford.edu/users/cottrell/
Updating the FTP
The script responsible to update the public FTP is in the following location:
/afs/slac/package/pinger/ftp-update.pl
To run this script you must use the following command:
$ /opt/TWWfsw/bin/perl /afs/slac/package/pinger/ftp-update.pl
You can SSH to the pinger machine with the following command:
ssh pinger@pinger.slac.stanford.edu
Output from running the command ftp_update.pl:
It represents how many files were missing during the update process.
Cron Job to run ftp-upate.pl automatically each day
The script runs every morning at 4AM. To modify the cron job you should edit the trscrontab:
$ trscrontab -e
and edit the following line:
lnxcron;480 00 4 * * * /opt/TWWfsw/bin/perl /afs/slac/package/pinger/ftp-update.pl
Log Files:
Log files with missing files can be found at:
$ /afs/slac/public/users/cottrell/pinger_log_files AKA ftp://ftp.slac.stanford.edu/users/cottrell/pinger_log_files/
Each .txt file contains the names of the files missing in the FTP.
The log files are generated automatically by ftp-update.pl everytime a file is missing in the pingerreports folder.
The files also can be generated by the following script:
$ /opt/TWWfsw/bin/perl /afs/slac/package/pinger/ftp-update-log.pl
Algorithm
Foreach metric Copy correspondent tar file to pinger_temp_files folder Untar the file inside pinger_temp_files folder Check the most recent file in the recently untared folder Copy from pingerreports folder to the pinger_temp_files folder all the correspondent files until the current previous day Tar the updated folder Copy back to the FTP main folder
ftp main folder folder:
$ /afs/slac/public/users/cottrell AKA ftp://ftp.slac.stanford.edu/users/cottrell/
pinger_temp_files folder:
$ /afs/slac/public/users/cottrell/pinger_temp_files AKA ftp://ftp.slac.stanford.edu/users/cottrell/pinger_temp_files/
pingerreports folder:
$ /nfs/slac/g/net/pinger/pingerreports/hep