Winter 2010-11 Disk Storage Strategy

Oracle has pulled a Scrooge on us, voiding our November order for 320 TB of Thors - before they were discontinued. We didn't make it in time. Now we need to explore options and plans for our disk storage. Also to get us through this particular crunch - we have some 55 TB free at the time of writing, and owe 128 TB to KIPAC and BABAR!

Possible elements of the plan:

• clean up existing space
• explore replacing disks in existing fileservers with 2 TB ones
• find a new vendor(s)
• change storage model

Clean up

We have 1.05 PB of space in xrootd now. Some 640 TB of that is taken up by L1, and 90% of that space is occupied by recon, svac and cal tuples.

[added by Tom]
Data current as of 12/8/2010.
Current xroot holdings:

Current usage:

Consumption rate:

Therefore, with current holdings and usage, there is sufficient space for 71 days of Level 1 data (running out ~17 Feb 2011)

Usage distribution:

Detail of Reprocess directories:

Detail of top-level Monte Carlo directory:

Detail of Monte Carlo 'ServiceChallenge' directories is quite lengthy and appears in full here.

Breakdown of space usage by Level 1 data products (from dataCatalog):

Replacing Existing Disks

This option was received with a frown from CD. Of course, Oracle has thought of this and charges a lot for the replacement drives, which have special brackets. And then there is the manpower to replace and the shell game of moving the data around. We'd have to press a lot harder to get traction on this option.

New Vendors

 Two obvious candidates are in use at the Lab now: DDN used by LCLS, and Dell el cheapo by ATLAS.

DDN

LCLS is not super thrilled by what they got, though we are told that other labs have them and are happy. CD is getting price quotes for thor-sized systems.

Dell

These are 5200 rpm & $160/TB, but low density. At the same density, 7200 rpm disks (and maybe better connectivity?) are $275/TB. This is less than half the thor costs.

Change Storage Model

 The idea is to use HPSS as a storage layer to transparently retrieve xrootd files on demand. Wilko thinks the system can push 30-50 TB/day from tape. This is comparable to the rate needed for a merit reprocessing and so is thought not to be a major impediment. In this model, we would remove the big files from disk after making 2 tape copies. They would be retrieved back into a disk buffer when needed. So we would have a relatively fixed disk buffer, and a growing tape presence.

Here are some thoughts how to implement the new storage model.
The xrootd has the be configured to automatically stage a missing file from tape to disk. This is well known and in production for BaBar. However, instead of having a client somewhat randomly staging files it will be more efficient to sort the files by tape and then stage them in that order before the client needs the files.

If the xrootd cluster is retrieving files from HPSS a policy is needed to purge files from disks. In the beginning I assume that the purging will be run manually by providing the files that should be purged (either a list of files or
a regular expression filenames will be matched against). Purging will be done before and during a processing that requires a large amount of data to be staged.

Monitoring should be setup to have a record of how much data is being staged and of the available disk space in xrootd.
A warning should be sent if the disk space will be close to exhaustion. One has to look at the total xrootd disk space as some servers will be filled and some will have space.

Currently files are backed up to single tape copies. In order to create dual tape copies a new HPSS file family has to be created. Existing files have to be re-migrated to the new file family. One should also think about if one could have a file family for just the recon files. Currently they are (might be) mixed on the same tape with all the other L1 file types which might make the retrieving less efficient.

Below are the steps that were outlined above. I put some rough time estimates for the different steps.

1. Testing large scale staging. (2 weeks)
2. Deploy a new xrootd version (the current version supports only an old depreciated interface to HPSS) (1 week)
3. configure the xroot to stage files from HPSS to disk if they are missing (1 day)
4. Setup monitoring of the staging (partly comes with xrootd) (2 days)
5. Setup (I guess Nagios) to send out warnings if the xrootd disk space will be filled up
6. Develop code to do allow purging files using file lists or regEx (1 week)
7. Setup new dual copy file family (2 days)
8. Re-migrate files to have dual copies (1 day work, 2-3 weeks for migration)
9. Tools to sort files by tape and prestage them in order to increase the HPSS throughput.