Page History

Policy by Folder

Red text indicates a change from the previous policy.

...

Rationale for Proposed Policy

In the past couple of years we have observed a few non ideal some aspects of the LCLS data retention policy which are not ideal:

1. All experiments are treated equally even if a few institutions copy the data home and don't need to have the data on disk at SLAC: ideally we would reserve that disk space for other experiments that do rely on it.
2. Some folders and the different storage classes (short, medium and long term) were not always understood properly understood or used properly  (eg e.g. ftc was often treated as scratch).
3. It's been extremely hard to maintain the promise on the lifetime of preserving all the data on disk under the conditions of limited spacefor their supposed lifetime: this has proved particularly tricky for scratch where the users can easily write tens of terabytes in a few hours.

The policy proposed below relies on two enabling technologies:

1. We can constantly monitor the usage of the file system without affecting its performance.
2. We can actually enforce the quota on a specific experimental directory at the file system level (as opposed to just alert users who use too much space).

The addition of these capabilities allow us to enforce a policy that adapts to the actual usage of the system.

Proposed Policy

Raw data storage (xtc/ and hdf5/ folders)

...

1. Deleting data too early, i.e. when files are still being actively accesses, can cause large, and concurrent, restore operations from tape which affect negatively the performance of the system.

Also, we have studied the data usage over time and we have observed that:

• The rate at which data are accessed starts decreasing around 130 days after the experiment ends, both for raw and generated data.

Proposed Policy

Based on the observations above we propose the following modifications:

...

• Change the lifetime for raw and scratch data on disk to 4 months.
• After the initial 4 months period, the expiration status of a run is determined by the access pattern of its files. 
  • • This will be enforced by the constant monitoring of the file systems to determine which files were

  ...

  • • accessed and when. We will be also tracking if the files were actually analyzed or just "touched" to dodge the policy.

  ...

  • •  
    • Analyzing the data extends the lifetime of the accessed data by 1 month from the access date.
    • Data restored from tape will stay on disk for 1 month, i.e. it's treated as a file access. (Users will be able to restore files by themselves through the web portal interface.)
  • Eliminate the short and medium storage classes.
  • Eliminate the /ftc folder.
    • Data currently under ftc will be moved to scratch and the scratch policy will apply.
  • Rename the usr folder as usrdaq.
  • Rename the res folder as results.
  • Increase the quota of the results folder to 4TB.

  Notes

  • Please do not store under the scratch folder data that you cannot recreate because this directory is not backed up and the oldest files on scratch may be deleted at any time to make space for data from new experiments.
  • The tape archive (xtc, hdf5, usrdaq) and the tape backup (results, home) are fundamentally different:
    • In the tape archive the folders are frozen after the end of the experiments and their contents are stored on tape once. 
    • In the tape backup, the system takes snapshots of the folders as appear at a given time. this implies that files which are deleted from disk are eventually, i.e. after a long enough time, also deleted from tape. 
  • For raw data the cleanup operations will affect all files, i.e. all streams and chunks, which make up one run, rather than individual files.
  • After 2 years from the end of an experiment we'll remove the experiment from disk. At that point we'll take a snapshot of the results and calib folders and archive them to tape so that we can, upon request, restore an entire experiment back to disk.
  • After 10 years we plan to remove the tapes with the archived raw data from the silos and store them in a safe environment.
  • The new policy will apply to all experiments, i.e. it will be retroactive, and its deployment date will coincide with the start of Run 14 (August 10^th 2016).

  User Home

  • Please do not store large files under your home, this space is meant for code/scripts, documents, etc, not science data.
  • Users can check the used and available space under their home with a command like:

  df -h ~<username>

  • Users can automatically access snapshots of the backup of their home here:

  ~<username>/.zfs/snapshot/

   

  The actual expiration threshold will be calculated dynamically based on the amount of free space available on the corresponding file system at the time of the cleanup procedure. Files with the more recent access times will stay on disk. The clean up process will be removing older files until it achieves the goal of preserving 20% of free space on the file system. The cleanup will affect whole runs (not individual files) which were determined as "expired". An automatic notification message will be sent to the PI (or all members?) of the affected experiment after each cleanup.

  User data folders (res/, scratch/ and /ftc)

  The amount of storage available for scratch/ will be increased by redistributing the freed storage resources of the ftc/ file systems. Like for the raw data folders (xtc and hdf5) the retention of files on scratch will switch from the fixed term expiration model to the last access time based model. The cleanup algorithm will be eliminating files with the older access times until the desired goal of having 20% of free space is met.

  To compensate for a loss of the guaranteed longer term storage for experiments due to  the elimination of ftc/ and change in the file expiration terms of scratch/  we will increase the amount of quota on the res/ file systems. Note that res/ is the only user data folder which is backed up to tape. We will also implement the enforcement of the per-experiment storage usage quota for res/ (something which we couldn't do before). The new mechanism will prevent members of an experiment from writing new files onto their res/ folder after the experiment's quota limit is reached.

  The existing 24 months expiration policy will be respected after the change. Though, at the same time we're changing the expiration enforcement technology for the file system by switching to Lustre HSM/HPSS. The technology  will allow expired files to stay in the file system _namespace_ (be visible to users with the 'ls' command) while the actual file content will be eliminated from the disk. The files which are not actually on disk will be automatically restored from tape at a first attempt to open them. This process is supposed to be transparent to a user application except an extra delay before the file content will be available to a user process. When a process hits this type of files it will just hang at the file open stage before the file is brought back from tape to disk.

  Note that the last measure may also allow an experiment to exceed the actual amount of data associated with res/. The per-experiment quota will only apply to files which are on disk only. The older files which were copied to tape and whose content is not present on disk will be not counted toward the experiment's hard quota. We may also allow users to control the file migration process by giving them a command line and/or a Web-based tool for inspecting a status of files on disk and requesting transitions (eliminating files from disk or restoring from tape). This should prevent inefficient use of the batch system by analysis jobs hitting those not-actually-on-disk files.

  ...