...
We might be able to get away without compression. It will simplify the analysis. We will need to experiment to see how much data is created. Each years year's worth of data compressed for pinger.slac.stanford.edu takes ~ 800MBytes and I am guessing about 4Gbytes uncompressed (guessing compression ratio ~ 5, this needs verifying). For pinger.cern.ch (a more typical MA) we see ~ 50MBytes/year compressed or 200MBytes uncompressed. There are <~ 100 MAs or say 20Gbytes. So in total we may need ~25Gbytes for the uncompressed data.
Task
The task is to take the MA data for each MA for each day of a month and aggregate it into a single file per MA per month. Initially don't bother with compression.
...
Write in perl. Start from ~cottrell/bin/template.pl or ~cottrell/sumdir-regexp.pl. Use the perl opendir function to get the directory listing. Use /bin/gunzip to uncompress (unzip) the files. Use stat if you need file size etc. Look at File::Copy for copying files.