...
The task is to take the MA data for each MA for each day of a month and aggregate it into a single file per MA per month. Initially don't bother with compression. Eventually this will run as a daily cronjob,thus it must be robust and if run from a cronjob only produce any stdout if there is a problem.
Hints
Write in perl. Start from ~cottrell/bin/template.pl or ~cottrell/sumdir-regexp.pl. Use the perl opendir function to get the directory listing. Use /bin/gunzip or bin/zcat to uncompress (unzip) the files. Use stat if you need file size etc. Look at File::Copy for copying files. You will need to use the append feature of the perl open with >>.
The permissions for the directory are:
| Code Block |
|---|
109cottrell@pinger:~$ls -ld /nfs/slac/g/net/pinger/pingerdata/hep/data/
drwxrwsr-x 210 iepm iepm 6144 Jun 10 10:43 /nfs/slac/g/net/pinger/pingerdata/hep/data/ |
Script
The script to unify this data is /afs/slac/package/pinger/unite-monthly.pl.
Storage space for Unified data
Renata created /pingerdata.archive on netfs03 with about 600GB of space. The directory is /nfs/slac/g/net/pinger/pingerdata/hep/data.unite/.
The space is available via the automounter. It is owned by userid pinger under group sf.