Data Retention Policy

Old Policy

Space	Size	Backup	Lifetime	Comment
xtc	Unlimited	Tape archive	1 year	Raw data
hdf5	Unlimited	Tape archive	1 year	Data translated to HDF5
scratch	Unlimited	None	1 year	Analysis results and temporary data
User home	Unlimited	Disk + tape	Indefinite	User code
Tape archive	Unlimited	Two copies	10 years	Raw data

Proposed Policy

Space	Size	Backup	Lifetime	Comment
xtc	Unlimited	Tape archive	3 months	Raw data
ftc	10TB	None	2 years	Filtered, translated, compressed
res	1TB	Disk	2 years	Analysis results
tmp	Unlimited	None	3 months	Temporary data
User home	20GB	Disk + tape	Indefinite	User code
Tape archive	Unlimited	Two copies	10 years	Raw data

Rationale

The main goal of the new proposed policy is twofold:

Allow users to have easy access to the data (ie on disk) for 2 years instead of 1

Encourage users to filter their data to get rid of useless runs or useless events within a run

Notes

LCLS management may authorize the increase of some of the parameters in the table above for particular experiments (in particular xtc lifetime and ftc maximum size).

The new policy will be accompanied by new web tools which will allow users to:

Select runs to be saved to ftc (data would also go through lossless compression step at this stage)

Apply processing framework to select useful events within a run (only when effective; this is where we need help from users who should decide which events are good)

Select runs to be translated to HDF5 (already exists, but now HDF5 files would count against the ftc quota so the existing page would need to be integrated in the new view)

Select runs to be restored from tape to disk (useful when the users decide that a run may have not been useless after all)

File management operations available to users

Managing Files in Web Portal of an Experiment

Child pages