Data Retention Policy

Old Policy

Space	Size	Backup	Lifetime	Comment
xtc	Unlimited	Tape archive	1 year	Raw data
hdf5	Unlimited	Tape archive	1 year	Data translated to HDF5
scratch	Unlimited	None	1 year	Analysis results and temporary data
User home	Unlimited	Disk + tape	Indefinite	User code
Tape archive	Unlimited	Two copies	10 years	Raw data

Proposed Policy

Space	Size	Backup	Lifetime	Storage class	Comment
xtc	Unlimited	Tape archive	3 months	Short-term	Raw data
hdf5	Unlimited	Tape archive	3 months	Short-term	Data translated to HDF5
ftc	10TB	None	2 years	Medium-term	Filtered, translated, compressed
res	1TB	Disk	2 years	Medium-term	Analysis results
tmp	Unlimited	None	3 months	Short-term	Temporary data
User home	20GB	Disk + tape	Indefinite		User code
Tape archive	Unlimited	Two copies	10 years	Long-term	Raw data

Rationale

The main goal of the new proposed policy is twofold:

Allow users to have easy access to the data (ie on disk) for 2 years instead of 1

Encourage users to filter their data to get rid of useless runs or useless events within a run

Notes

LCLS management may authorize the increase of some of the parameters in the table above for particular experiments (in particular xtc lifetime and ftc maximum size).

The new policy will be accompanied by web tools which will allow users to:

Select runs to be saved to medium-term storage

Apply processing framework to select useful events within a run (only when effective; here we'll need help from users who should decide which events are good)

Select runs to be restored from tape to disk (useful when the users decide that a run may have not been useless after all)

Child pages

Data Retention Policy

Old Policy

Proposed Policy

Rationale

Notes