XRootD Staging

Stage/Migrate/Purge for XRootD

The current Xrootd cluster is only loosely integrated with HPSS. Most of the L1 data files are migrated to HPSS in order to have a backup but there is no automatic procedure to copy files back to disk. Files are not purged from disk if the space is filling up.

The following discusses the tight integration of Xrootd with HPSS that allows purging old files from disk and transparently copy them back to disk if they clients try to access them. The three task that facilitate the integration are:

• Staging: Copy a file from HPSS to disk if a client opens a file that is not on disk. Staging is transparent to a client.
• Purging: Remove files from disk. Only files that have previously been copied to HPSS are eligible for purging. Policies which files to purge can be applied.
• Migration: Copy a newly created file to HPSS.

In this scenario the disk space in the xrootd cluster could be smaller than the data produced by Fermi. As mentioned above the access to files that are not on disk is transparent to a client. The only difference accessing a non disk resident file is that opening the file will take longer as the file has to be staged. This overhead can be avoided by staging files prior to running the clients.

The recon (followed by cal and svac files) data files account for a large fraction of the total data (see HPSS Usage) and are the prime candidates for purging and staging. All other data types should be kept on disk and not purged.

In order to enable the above scenario the following issues need to be answered:

• Dual copy of files in HPSS (otherwise there will be only on copied if a file is purged)
• Client interaction with HPSS (open, stat, rm, ...).
• Policy to decide which files should be purged.
• Tools to prestage files. Efficiency of prestaging.
• Performance of HPSS. How many TB can be transfer from HPSS to disk per day.
• Costs.

Dual copy

Only a single copy of a file exists if it is purged from disk. There is a small chance that retrieving a file from a tape will fail due to tape media error or a broken tape. This risk can be mitigated creating dual copies of files in HPSS. However this will increase the costs as twice as many tapes are needed. If the lost data can be reproduced dual copies might not be necessary (there should be a more or less automatic procedure to reproduce lost data files).

Staging setup

Staging is transparent to a client. The only difference to reading a disk resident file is that the client has to wait while the file is copied from HPSS to disk (which typically is a couple of minutes and depends on the file size). Staging (also migration/purging) is not need for all data and the filename space would be split into:

Files below the /glast/Data/Flight are marked as migratable and some of these files are migrated to HPSS (not all. A list generated from the data catalog selects which file should be copied to HPSS).
As this space is the only one for which files are migrated to HPSS it is the only one that should be stageable.
Files below /glast/Scratch/ are temporary files and neither staging nor migration is needed.
All other files (/glast/) are treated as migratable but no active migration is running (but could be turned on any time).

Client interaction with HPSS

When HPSS is used with xrootd there are two places files can be found (disk, HPSS) and therefore the behavior of the meta-data commands (rm, stat, dirlist, ...) could be issued against both. The behavior is configurable through the xrootd configuration file and the tools xrootd uses to communicate with HPSS.
The choices for the most common operations are:

• With the check configuration xrootd would check in HPSS if a file exists and would not allow the client to write if it does. In this case, if the client wants to overwrite a file, it must be able to remove the copy in HPSS.
• Without check the client will be able to write a file even if it exists in HPSS. Eventually the file will be migrated and overwrite the version in HPSS.
• There will be a small overhead if commands are issued against HPSS but they are typically negligible.
• Using the HPSS commands will couple the client to HPSS. If for example HPSS is not available and the meta-commands might hang and so the client.
• Currently the meta-data commands are only issued against the files on disk and this is the preferred setup.

Purging

Purging removes files that have been migrated to HPSS from disk if disk space is needed. A policy is needed to decide which file should be purged. The default policy is to purge the oldest (with respect to access time) files which might not be suitable for FGST. Simple to implement would be some regular expression or reading a list from a file.

Tools to stage files.

A client that encounters a file that is not on disk will automatically stage the file as long as it is in HPSS.
The xprep tool is used to prestage files. It is issued against the redirector and accepts a list with files to stage. The redirector will distribute the stage request over the data servers.
Using the HPSS hsi tool allows to find the tape name and position for a file. Sorting a stage request by tape and tape position might be advantageous to increase the staging throughput. However, the faster method to stage files will be to first stage all files from a tape to the HPSS disk cache and then have xrootd copy the files to the data server from the HPSS cache.

Testing staging performance

A critical aspect of staging how long it takes to stage a file and if HPSS will be able to keep up with the throughput required to stage recon files for a reprocessing task.

Action item

• Should meta-data commands (rm, stat, ...) access HPSS?
• How to decide which files should be purged?
• Dual copy of files in HPSS ?