Page History
...
Red text indicates a change from the previous policy.
Space | SizeQuota | Backup | Lifetime | Comment |
|---|---|---|---|---|
xtc | UnlimitedNone | Tape archive | 4 months | Raw data |
usrdaq | UnlimitedNone | Tape archive | 4 months | Raw data from users' DAQ systems |
hdf5 | UnlimitedNone | Tape archive | 4 months | Data translated to HDF5 |
scratch | UnlimitedNone | None | 4 months | Temporary data (lifetime not guaranteed) |
|
|
|
| |
|
|
|
| |
results | 2TB | Tape backup | 2 years | Analysis results |
User home | 20GB | Disk + tape | Indefinite | User code |
Tape archive | UnlimitedNone | Two copies | 10 years | Raw data |
...
- Eliminate the short and medium storage classes.
Switch from an experiment wide policy, where all files from one experiment stay on disk for a fixed duration of time after the end of the experiment, to a run wide policy where the expiration status of the files is determined by the access pattern of the files of a specific run. This will be enforced by the constant monitoring of the file systems to determine which files were accessed and when. We will be also tracking if the files were actually analyzed or just "touched" to dodge the policy.- Igor: no need in this statement as this has always been like this. Perhaps we should just say/remind that: "...for raw and translated data all files associated with a run and stored within a particular folder would be a subject for the Policy..."
- Change the lifetime for raw and scratch data disks to 4 months. (For raw data the cleanup will affect all files, i.e. all streams and chunks, which make up one run, rather than individual files.)
- Igor: 4 months for raw data is probably fine. Should we really make the same commitment for the scratch/ folders?
- Igor: Should we also warn users that if we'll be running out of space we'll be clearing scratch/ before the proposed expiration term based on the file creation time?
- Eliminate the /ftc folder.
- Igor: What should we tell users about existing ftc/ folders?
- Rename the usr folder as usrdaq.
- Rename the res folder as results.
- Increase the quota of the results folder to 2TB.
- Igor: What should we say about files restored from tape? Should we assume a different expiration term for them?
- Igor: What should we say about NERSC?
Notes
- The backup of the results and home folders is fundamentally different from the archiving of the xtc/hdf5/usrdaq folders. In the latter case the folders are frozen after the end of the experiments and their contents are stored on tape once. In the case of the res and home, files which are deleted from disk are eventually, i.e. after a long enough time, also deleted from tape.
- Please do not store under the scratch folder data that you cannot recreate because this directory is not backed up and the oldest files on scratch may be deleted at any time to make space for data from new experiments.
...
Overview
Content Tools