Data Retention Policy

Storage Classes

Tools

Web tools have been created to facilitate users in:

• Restoring files from tape to disk

• Move data across different storage classes

• Applying processing framework to select useful events within a run (only when effective)

Frequently Asked Questions

Will all raw files be deleted after 6 months?

• No, you can extend to 2 years the lifetime on disk of your raw data by selecting the most frequently accessed runs. This selection can be done with the file manager in the experiment web portal.

The total size of our experiment is below the quota, may we select all runs for the 2-years storage?

• You may, but try to be a good citizen by selecting only the most frequently used files and by restoring from tape the runs you rarely use.

What happens to the data under the scratch folder?

• All files under scratch which are older than 6 months will be deleted. Use the ftc or res directories to extend the lifetime of your intermediate data.

Can we restore raw files from tape to disk by ourselves?

• Yes, you can restore raw files (xtc and hdf5) from tape using the file manager tab in the experiment web portal.

Can we restore the contents of the ftc, res and scratch folders from tape?

• No, the contents of the ftc and scratch directories are not backed up or archived. The contents of the res folder, though, can be restored on demand from a disk backup.

How long does it take to restore files from tape to disk?

• The rule of thumb is 1TB per hour. Actual results will vary from 60MB/s (if only one file is restored) to more than 600MB/s (if there are at least ten files in the queue evenly spread across ten available tape drives). Most experiments could be entirely restored on disk in half a day. The largest LCLS experiment would take five days to be restored.

What is the lifetime on disk of files restored from tape?

• Files restored from tape will stay on disk for one month.

How long do the raw data files stay on tape?

• Ten years.

What should we store in the ftc experiment directory?

• The ftc directory is meant to provide a long term disk storage (2-years) for the raw data after basic processing like event filtering, translation to HDF5 or compression. Users can store what they want under ftc as long as they can recreate its contents from the raw data since this directory is not backed up or archived. This space has a default quota of 10TB.

What should we store in the res experiment directory?

• The res directory is meant to provide a long term disk storage (2-years) for the results of the user analysis. Users should store important results and shared code under this directory. This directory is backed up and has a default quota of 1TB.

What should we store in the scratch experiment directory?

• The scratch directory is meant to provide a large, short term disk storage (6-months) for intermediate results. Users can write to this directory during data analysis development and later on move the most important results to res. Please do not store under the scratch folder data that you cannot recreate because this directory is not backed up. This folder doesn't have a quota.

Can we ask for an increase of the 2-years quota for a particular experiment?

• It's fair to ask.

Can we ask for an extension of the deadline for the run selection?

• Yes, but only for experiments taken after April 2011 and try to keep the requested date within 3 months from the proposed deadline.