Page History
Old Policy
Space | Size | Backup | Lifetime | Comment |
|---|---|---|---|---|
xtc | Unlimited | Tape archive | 1 year | Raw data |
hdf5 | Unlimited | Tape archive | 1 year | Data translated to HDF5 |
scratch | Unlimited | None | 1 year | Analysis results and temporary data |
User home | Unlimited | Disk + tape | Indefinite | User code |
Tape archive | Unlimited | Two copies | 10 years | Raw data |
...
Storage Classes
Space | Size | Backup | Lifetime | Storage class | Comment |
|---|---|---|---|---|---|
xtc | Unlimited | Tape archive | 6 months | Short-term | Raw data |
hdf5 | Unlimited | Tape archive | 6 months | Short-term | Data translated to HDF5 |
scratch | Unlimited | None | 6 months | Short-term | Temporary data |
xtc/hdf5 | 10TB | n/a | 2 years | Medium-term | Selected XTC and HDF5 runs |
ftc | 10TB | None | 2 years | Medium-term | Filtered, translated, compressed |
res | 1TB | Disk | 2 years | Medium-term | Analysis results |
User home | 20GB | Disk + tape | Indefinite |
| User code |
Tape archive | Unlimited | Two copies | 10 years | Long-term | Raw data |
...
The goal of the new proposed policy based on three different storage classes is twofold:
- Allow users to have easy access (ie on disk) to the most frequently used data for a longer period of time
- Make better use of the LCLS storage resources
Tools
Tools
Web tools have been created The new policy will be accompanied by web tools to facilitate users in:
- Restoring files from tape to disk
...
Frequently Asked Questions
Why are you doing this to us?
We do believe this is a better policy, above all for the users. We noticed that the previous 1-year policy was not enough. At the same time keeping everything on disk for 2-years is not the best use of LCLS resources. Hence LCLS decided to extend the lifetime on disk to 2-years for the most used data files. This was done by introducing quotas and by letting the users select which files should stay on disk.
We do understand the new policy requires a more active participation by the users so we'll provide tools to help managing the data. For example, we made it much easier to restore data from tape and move data across storage classes. We'll soon provide tools to compress and filter the data.
Also, the scratch directories were sometimes (ab)used unfairly. So we decided to create three scratch areas (ftc, res and scratch) with different characteristics.
So will all raw files be deleted by the proposed deadlinesWill all raw files be deleted after 6 months?
No, you can extend by 2 years the lifetime on disk of your raw data by selecting the most frequently accessed runs. This selection can be done with the file manager in the experiment web portal.
By which date do we need to select runs with the file manager to increase the lifetime on disk of our data?
See the proposed dates here https://confluence.slac.stanford.edu/display/PCDS/Phase+1+Dates
The total size of our experiment is below the quota, may we select all runs for the 2-years storage?
You may, but try to be a good citizen by selecting only the most frequently used files and by restoring from tape the runs you rarely use.
What will happen happens to the data under the scratch folder?
Starting November 1st 2012, all All files under scratch which are older than 6 months will be deleted. Use the ftc or res directories to extend the lifetime of your intermediate data.
...
The ftc directory is meant to provide a long term disk storage (2-years) for the raw data after basic processing like event filtering, translation to HDF5 or compression. Users can store what they want under ftc as long as they can recreate its contents from the raw data since this directory is not backed up or archived. This space has a default quota of 10TB. Note that ftc directories will be made available on October 1st, 2012.
What should we store in the res experiment directory?
...
Overview
Content Tools