A place to track projections for hardware going into Building 50 - for power, space and cooling planning purposes.
Hardware Proposal: PPA has requested $600k of annual hardware funding from DoE to cover small experiment and peak computing needs. This is primarily for cycles; storage needs seem modest.
FY13-14 Projection/update
Raw inputs here
Fermi
$600k Options include
- need ~3000 cores for 6 months for reprocessing
- replacing 2 Oracle servers
- potentially another PB disk
- unknown amount of tape (5 TB?)
ATLAS
- curent allocation about 3400 cores. Agreed with ATLAS to increase about 1000 cores. Always in use.
EXO
- 200 core DC + bursts to 1000 needed
CDMS
- much like EXO
BABAR
- 2600 core allocation - no plan to add more in general queues. LTDA good for now?
HEP Theory
- peak needs about 250k CPU-hrs in 1-2 week episodes: 1000-2000 cores
General Users
- Allocation of about 2350 cores
Core numerology
Note: 1500 cores are hanging by a thread in the to-be-retired Black Boxes.
Rule of thumb used: 10 allocation units is ~1 core - taken from recent bqueues -l.
USER/GROUP SHARES PRIORITY STARTED RESERVED CPU_TIME RUN_TIME exoprodgrp 1000 333.333 0 0 0.0 0 lcdprodgrp 836 278.667 0 0 0.0 0 cdmsdata 1000 166.667 1 0 0.0 654048 rpgrp 418 139.333 0 0 0.0 0 glastdata 25174 40.806 29 0 2709927.5 297435 glastgrp 8107 20.155 67 0 1020309.2 24120 hpsprodgrp 600 16.304 4 0 112119.7 72947 lcd 418 7.420 7 0 166287.5 133324 rdgrp 342 1.275 0 0 1364227.6 0 babarAll 26332 1.132 989 0 104347464.0 16872557 glastusers 2000 1.087 361 0 3881826.5 16058350 AllUsers 23545 0.780 2092 0 123027648.0 18098542 theorygrp 1000 0.628 5 0 8096284.0 140116 atlasgrp 34307 0.606 3172 1 242260000.0 140770233
FY13 Projection
Fermi
$600k Options include
- replacing 2 Oracle servers
- 35 nodes to go into the general queues
- potentially another 400 TB disk
- unknown amount of tape (5 TB?)
BABAR
- ballpark $100k in various areas - replacing old file servers, tape etc. Optimization not done yet.
ATLAS
LSST
EXO
- $32k for disk
- $20k for standalone linux servers
- $5k tapes
CDMS
- $16k for disk, but would prefer buying in to a shared storage solution
HEP Theory
LCD
Mid FY12 Update
Fermi
$500k left to spend in CY12. Options include
- replacing 25 glastlnx standalone servers (with perhaps 8 beefier ones)
- contribute $35k for 2 5 TB tape drives
- 35 nodes to go into the general queues
- potentially another 400 TB disk
- unknown amount of tape (1 vs 5 TB?)
BABAR
- $35k for 2 5 TB tape drives
- 20 nodes added to LTDA
- line card for the LTDA switch, and new Etherite for the serial consoles
ATLAS
- 2 cabinets of mostly storage
LSST
- 4-5 compute nodes
EXO
Hoping for the following. Not sure how much of the disk money is really left due to other EXO expenditures.
- $32k for disk
- $20k for standalone linux servers
- $5k tapes
KIPAC
- replace interactive login servers ki-ls01-06
CDMS
- nada
HEP Theory
- 24 TB disk hoped for
LCD
- 1 Dell R610 for grid access to their fileserver
Totals (roughly):
Storage |
$ |
Standalone |
$ |
"Batch" nodes |
$ |
Tape |
|
Total $ |
---|---|---|---|---|---|---|---|---|
1500 TB |
500k |
18 |
90k |
60 |
300k |
4 |
70k |
960k |
FY2012
BABAR
complete LTDA purchase - CD already knows the details. Retire old fileservers, using returned thumpers from Fermi.
Fermi
expect ~400 TB fileservers and 10 standalone linux servers. Presumably to be used for retiring old servers.
CDMS
perhaps 1-2 fileservers
ATLAS
The ATLAS Tier 2 projection is that we will spend about $220k on hardware (more if recharge costs are low, less if they are high). In addition, it is a reasonable guess that there will be up to $50k of other ATLAS-related purchases.
According to my rules of thumb, this will translate into at most 90 rack units and 27 kW dissipated.
We should also expect to retire our three oldest Thumpers, and offer to retire our 79 Sun X2200 M2s that are in one of the Black boxes.
KIPAC
- our FY11 CEP for storage should complete soonish with about 1/2 rack (4x4U + servers), power density is modest (~250W/U)
- FY12 is unclear, if we get the requested amounts then we would likely get ~48core/2U machines filling 2/3 rack at higher power
density (700W/U)
DES:
- If DES uses BaBar leftovers, we need to keep those running and buy storage. Estimated upper limit is 1/2 rack at modest power
density (~250W/U)
So in summary we need something like 2 racks, one at high power and one and medium/low. It is possible that something old gets retired
but nothing significant this year.
EXO
We have budgeted approximately $32k/year for file servers, $20k/year for miscellaneous servers (or whatever) and $5k/year for tape for the next few years. Given that we are now taking data at a fairly steady rate we will need to review our requirements and may need to make some adjustments.
Theory
Marvin Weinstein
I have an LDRD proposal that is being considered for the development of DQC. If it is funded I plan to purchase 3 machines with a minimum of 48 processors and a good GPU and 64 GB of ram and at least 2 TB of hard drive. Since these machines will have 4 or more real CPUs, they will have to run Windows Server. I assume building 50 is your domain, I certainly plan to try and house these beasties with you.