Oracle has pulled a Scrooge on us, voiding our November order for 320 TB of Thors - before they were discontinued. We didn't make it in time. Now we need to explore options and plans for our disk storage. Also to get us through this particular crunch - we have some 55 TB free at the time of writing, and owe 128 TB to KIPAC and BABAR!
Possible elements of the plan:
- clean up existing space
- explore replacing disks in existing fileservers with 2 TB ones
- find a new vendor(s)
- change storage model
Clean up
We have 1.05 PB of space in xrootd now. Some 640 TB of that is taken up by L1, and 90% of that space is occupied by recon, svac and cal tuples.
[added by Tom]
Data current as of 12/8/2010.
Current xroot holdings:
Total Space |
1043 TB |
35 servers (wain006 removed) |
xrootd overhead |
21 TB |
(disk 'buffer' space) |
Available Space |
1022 TB |
|
Current usage:
Space used |
966.778 TB |
95% |
Space free |
55.580 TB |
5% |
Consumption rate:
Level 1 |
783 GB/day |
(averaged over period since 4 Aug 2008) |
Therefore, with current holdings and usage, there is sufficient space for 71 days of Level 1 data (running out ~17 Feb 2011)
Usage distribution:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="b4afc2b6-5fdc-40e3-9ce0-3cb0f9742c59"><ac:plain-text-body><![CDATA[ |
path |
size [TB] |
#files |
Notes |
]]></ac:plain-text-body></ac:structured-macro> |
|---|---|---|---|---|---|
/glast/Data |
785 |
807914 |
|
||
/glast/Data/Flight |
767 |
792024 |
|
||
/glast/Data/Flight/Level1/ |
713 |
535635 |
670 TB registered in dataCat |
||
/glast/mc |
82 |
11343707 |
|
||
/glast/mc/ServiceChallenge/ |
56 |
6649775 |
|
||
/glast/Scratch |
51 |
358511 |
~50 TB recovery possible |
||
/glast/Data/Flight/Reprocess/ |
50 |
226757 |
~20 TB recovery possible |
||
/glast/level0 |
13 |
2020329 |
|
||
/glast/bt |
4 |
760384 |
|
||
/glast/test |
2 |
2852 |
|
||
/glast/scratch |
1 |
51412 |
|
||
/glast/mvmd5 |
0 |
738 |
|
||
/glast/admin |
0 |
108 |
|
||
/glast/ASP |
0 |
19072 |
|
Detail of Reprocess directories:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="31e51dac-4943-4826-8b3c-4e1178973d78"><ac:plain-text-body><![CDATA[ |
path |
size [GB] |
#files |
Notes |
]]></ac:plain-text-body></ac:structured-macro> |
|---|---|---|---|---|---|
/glast/Data/Flight/Reprocess/P120 |
22734 |
65557 |
removal candidate |
||
/glast/Data/Flight/Reprocess/P110 |
14541 |
49165 |
removal candidate |
||
/glast/Data/Flight/Reprocess/P106-LEO |
8690 |
1025 |
|
||
/glast/Data/Flight/Reprocess/P90 |
3319 |
2004 |
removal candidate |
||
/glast/Data/Flight/Reprocess/CREDataReprocessing |
631 |
22331 |
|
||
/glast/Data/Flight/Reprocess/P115-LEO |
533 |
1440 |
|
||
/glast/Data/Flight/Reprocess/P110-LEO |
524 |
1393 |
removal candidate |
||
/glast/Data/Flight/Reprocess/P120-LEO |
503 |
627 |
removal candidate |
||
/glast/Data/Flight/Reprocess/P105 |
264 |
33465 |
|
||
/glast/Data/Flight/Reprocess/P107 |
212 |
4523 |
|
||
/glast/Data/Flight/Reprocess/P116 |
109 |
25122 |
|
||
/glast/Data/Flight/Reprocess/Pass7-repro |
22 |
16 |
removal candidate |
||
/glast/Data/Flight/Reprocess/P100 |
8 |
20089 |
removal candidate |
Detail of top-level Monte Carlo directory:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="4cb49d5f-c8df-441c-99f7-bfee157d862a"><ac:plain-text-body><![CDATA[ |
path |
size [GB] |
#files |
Notes |
]]></ac:plain-text-body></ac:structured-macro> |
|---|---|---|---|---|---|
/glast/mc/ServiceChallenge |
57458 |
6649775 |
|
||
/glast/mc/OpsSim |
24691 |
4494890 |
|
||
/glast/mc/DC2 |
1818 |
136847 |
|
||
/glast/mc/OctoberTest |
41 |
29895 |
|
||
/glast/mc/XrootTest |
33 |
18723 |
removal candidate |
||
/glast/mc/Test |
14 |
13403 |
removal candidate |
Detail of Monte Carlo 'ServiceChallenge' directories (with \> 100GB consumed):
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="60bd0dec-7745-4083-82f1-59d7c0bbc211"><ac:plain-text-body><![CDATA[ |
path |
size [GB] |
#files |
Notes |
]]></ac:plain-text-body></ac:structured-macro> |
|---|---|---|---|---|---|
/glast/mc/ServiceChallenge/allHI-Fe-GR-v15r53p3-Lyon |
6040 |
199881 |
|||
/glast/mc/ServiceChallenge/backgnd-proton-GR-v15r6p1 |
4318 |
46676 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r0-2 |
3054 |
561645 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v13r9p3-Lyon |
2676 |
90000 |
|||
/glast/mc/ServiceChallenge/Interleave55d-GR-v11r17 |
2230 |
47520 |
|||
/glast/mc/ServiceChallenge/Interleave55d-GR-v11r12 |
2187 |
39600 |
|||
/glast/mc/ServiceChallenge/backgndSC-GR-v11r2 |
2158 |
108000 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r0 |
1555 |
258517 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v15r0 |
1447 |
60020 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v13r9p12 |
1337 |
50001 |
|||
/glast/mc/ServiceChallenge/Interleave55d-GR-v11r12-r-P5 |
1267 |
7920 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r40-Limbo2 |
1039 |
50505 |
|||
/glast/mc/ServiceChallenge/allGamma18GeV-GR-v13r9p3-Lyon |
990 |
90000 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r40-Limbo |
964 |
46980 |
|||
/glast/mc/ServiceChallenge/GRBgrid-GR-v13r9p22 |
959 |
12000 |
|||
/glast/mc/ServiceChallenge/GRBgrid-GR-v11r17 |
869 |
12000 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r9p16 |
753 |
130006 |
|||
/glast/mc/ServiceChallenge/GRBgrid-GR-v13r9p4-Lyon |
737 |
12000 |
|||
/glast/mc/ServiceChallenge/backgnd-electron-GR-v15r6p1 |
582 |
5600 |
|||
/glast/mc/ServiceChallenge/backgnd-positron-GR-v15r6p1 |
572 |
5600 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r0-HE |
435 |
48282 |
|||
/glast/mc/ServiceChallenge/skyPointed-GR-v11r17 |
346 |
21600 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r0-Day |
345 |
18729 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-HEAD1_1171_2_1-Day |
345 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v14r11-Day |
344 |
18728 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r5-Day |
343 |
17221 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-HEAD1_1171_4_2-Day |
340 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r35-Day |
339 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r33-Day |
339 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r3-Day |
339 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r29-Day |
339 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r21-Day |
339 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r13-Day |
339 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-HEAD1_1171_4_4-Day |
339 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-HEAD-1085-4-2-Day |
339 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-HEAD1-1148-Day |
338 |
18677 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r51p1-Lyon |
337 |
17292 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v14r8-Day |
323 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r5p8-Day |
323 |
15840 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r5p6-Day |
323 |
15840 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v14r2-Day |
316 |
18723 |
|||
/glast/mc/ServiceChallenge/opssim2-P6repro-GR-v13r9p19 |
313 |
3446 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v14r7-Day |
310 |
18723 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v14r6-Day |
310 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v14r4-Day |
310 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r9p12-Day |
310 |
17285 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r11p1-Day |
308 |
18720 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r7-Day |
307 |
15840 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r9-Day |
306 |
15840 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r8-Day |
306 |
15840 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-HEAD1_1165 |
302 |
15842 |
|||
/glast/mc/ServiceChallenge/GRBgrid-GR-v13r9p20 |
300 |
3606 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r1p1-Day |
295 |
14400 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r11p1-Day-22b |
286 |
15840 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r11p1-Day-22 |
286 |
15840 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r11p1-Day-21 |
286 |
15840 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r39p1-HE-350 |
284 |
780029 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r5p8-Day-Lyon |
283 |
12947 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v12r4-Day |
276 |
14400 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v11r17-Earth10 |
270 |
727248 |
|||
/glast/mc/ServiceChallenge/bkg-GR-v13r9p3-Day-OBF2 |
268 |
12960 |
|||
/glast/mc/ServiceChallenge/bkg-GR-v13r9p3-Day-OBF1 |
268 |
12960 |
|||
/glast/mc/ServiceChallenge/backgnd-neutron-GR-v15r6p1 |
265 |
2800 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v15r29-Lyon |
245 |
5263 |
|||
/glast/mc/ServiceChallenge/backgndSC-GR-v11r5-Day-Bertini |
238 |
7200 |
|||
/glast/mc/ServiceChallenge/backgndSC-GR-v11-Day-Bertini |
238 |
7200 |
|||
/glast/mc/ServiceChallenge/backgndSC-GR-v11r11-Day |
233 |
7200 |
|||
/glast/mc/ServiceChallenge/backgndSC-GR-v11-Day-oldGeom |
232 |
7200 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v15r0-LowE |
221 |
60003 |
|||
/glast/mc/ServiceChallenge/backgnd-alpha-GR-v15r6p1 |
213 |
1869 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v13r9p12-LowE |
196 |
50001 |
|||
/glast/mc/ServiceChallenge/opssim2-P6repro-GR-v15r6p1 |
177 |
1733 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r39p1-HE-100 |
164 |
120010 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v13r5p5-Day |
152 |
15840 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v11r17-Day-CRtest |
144 |
6302 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v15r7-LPM |
142 |
6000 |
|||
/glast/mc/ServiceChallenge/backgnd-GR-v15r38-Nadir |
133 |
37171 |
|||
/glast/mc/ServiceChallenge/backgnd-gamma-GR-v15r6p1 |
128 |
1869 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v11r14 |
121 |
1000 |
|||
/glast/mc/ServiceChallenge/AG-GR-v17r35p1-IRFS72-allE-L |
117 |
8011 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v15r50p9-OVERLAY |
113 |
4800 |
|||
/glast/mc/ServiceChallenge/allGamma-GR-v15r50p9-LOWNOISE |
110 |
4800 |
Breakdown of space usage by Level 1 data products (from dataCatalog):
Filetype: |
TotSize/#Runs = |
Avg file size |
(%) |
|---|---|---|---|
RECON: |
399257.600/13532 = |
29.505 GiB |
(58.2%) |
CAL: |
100147.200/13536 = |
7.399 GiB |
(14.6%) |
SVAC: |
71270.400/13535 = |
5.266 GiB |
(10.4%) |
DIGI: |
61542.400/13537 = |
4.546 GiB |
( 9.0%) |
FASTMONTUPLE: |
28262.400/13537 = |
2.088 GiB |
( 4.1%) |
MERIT: |
22528.000/13537 = |
1.664 GiB |
( 3.3%) |
GCR: |
676.800/13537 = |
51.196 MiB |
( 0.1%) |
FASTMONTREND: |
432.800/13537 = |
32.739 MiB |
( 0.1%) |
DIGITREND: |
337.100/13537 = |
25.500 MiB |
( 0.0%) |
FILTEREDMERIT: |
324.900/ 7306 = |
45.538 MiB |
( 0.0%) |
MAGIC7HP: |
244.700/13315 = |
18.819 MiB |
( 0.0%) |
LS1: |
183.700/13537 = |
13.896 MiB |
( 0.0%) |
CALHIST: |
181.400/13536 = |
13.723 MiB |
( 0.0%) |
RECONTREND: |
172.700/13537 = |
13.064 MiB |
( 0.0%) |
TKRANALYSIS: |
172.400/13536 = |
13.042 MiB |
( 0.0%) |
LS3: |
156.800/13537 = |
11.861 MiB |
( 0.0%) |
RECONHIST: |
124.400/13536 = |
9.411 MiB |
( 0.0%) |
MAGIC7L1: |
55.000/13315 = |
4.230 MiB |
( 0.0%) |
FASTMONHIST: |
53.400/13535 = |
4.040 MiB |
( 0.0%) |
LS1BADGTI: |
48.200/ 7306 = |
6.756 MiB |
( 0.0%) |
CALTREND: |
45.500/13537 = |
3.442 MiB |
( 0.0%) |
FT1: |
39.100/13537 = |
2.958 MiB |
( 0.0%) |
DIGIHIST: |
32.600/13537 = |
2.466 MiB |
( 0.0%) |
etc... |
|
|
|
Replacing Existing Disks
New Vendors
Two obvious candidates are in use at the Lab now: DDN by LCLS, and Dell el cheapo by ATLAS.
DDN
Dell
These are $160/TB, but low density.
Change Storage Model
The idea is to use HPSS as a storage layer to transparently retrieve xrootd files on demand. Wilko thinks the system can push 30-50 TB/day from tape. This is comparable to the rate needed for a merit reprocessing and so is thought not to be a major impediment. In this model, we would remove the big files from disk after making 2 tape copies. They would be retrieved back into a disk buffer when needed. So we would have a relatively fixed disk buffer, and a growing tape presence.