status: Under Construction
last update: 29 June 2011
This page is the beginning of a description of a future reprocessing based on Pass 7. It will evolve into the official task documentation with time.
A 'full' Pass 7 reprocessing is currently envisioned to begin ~August 2011. This task will read DIGI files and emit RECON, MERIT, a selection of other ROOT files, and an array of FITS files. It will be a CPU-intensive and storage-intensive enterprise requiring months of elapsed time and somewhere around 3/4 Pbytes of storage. At the time of this task beginning, there will be about 16,000 science runs in Fermi (3 years accumulation).
In order to avoid occupying a new 750 TB of disk space, the plan is to remove old RECON files once they have been reprocessed. This is a shell game that involves some amount of buffer space and then waiting until the new RECON file has been created and (to some extent) validated before removal. Further, the old RECON file must first be backed up to HPSS. A straw-man proposal for this scheme follows.
Assume 50 TB of xroot buffer space.
Reprocess ~10% of the mission data. Once complete, prepare list of files to be removed (RECON, possibly plus others
like CAL and SVAC if they are also regenerated by the reprocessing task) and hand off to Wilko. He will ensure the files to be deleted have been backed up and then delete the disk version.