...
- 6 full chassis @97.227k => 583.4k (quote 661664648)
- 5 blades w/IB added to existing empty slots @4@5.817k 018k => 2425.1k (quote 661664256)
- 4 IB switches with cables @10.5k => 48k 50.2k (includes active fiber IB cables for lustre switch) (quote 662008993)
- 2 60x2TB disk trays with controllers @34k @30.3k => 62k60.6k (quote 661663331)
Total is $717$719.5k 3k for 1648 1616 cores and storage expansion.
Gross bullet cluster core count would then be 2960+1616=4576 (all IB)
costs are:
- ATLAS: 60 blades @4.9375k => 296.25k (960c) (Based on quote 659024769 for a non-IB full chassis)
- Theory: 97k 97.227k + 5*45.817k 018k => 121.1k 122.32k (336c)
- PPA: 300.1k 73k (320c)
Note the PPA cost/core is bad because it includes the storage expansion and IB infrastructure.
Benefit here is that we have a uniform cluster.
...
Option 2: Expand to 15 full IB chassis and 4 non-IB chassis
...
- 4 full non-IB chassis @79k => 316k
- 3 full IB chassis @97k => 291k
- 7 blades w/IB @4.817k => 33.72k
- 2 60x2TB disk trays with controllers @34k => 68k
Total is $709k for 1904 cores and storage.
Gross bullet IB cores is then 3840.
Gross bullet non-IB cores is 1024.
costs are:
- ATLAS: 4 @79 => 316k (1024c)
- Theory: 97k + 7x4.817k => 130.72k (368c)
- PPA: 262k (512c)
Notes
- Revised on 8/21 to account for IB price change since the original nodes were purchased. Full IB chassis changed from 91->97 when IB changed from QDR to FDR (increase in performance).
- Need to verify Theory (Hoeche) budget (is 131k too large?)
- Need to verify Atlas budget (option 2 is over 300k)
- Revised 9/3 with latest quotes and to reflect choice of option one and the actual budget estimates.
Add-on to either option:
We could get new GPU servers (kipac's are old!) which are equivalent to bullet blades with ~5000 gpu-cores for ~10k each. So we could top off to 300k if we got 3 of these. We (PPA) do need to replace or existing GPU "system" that is hosted by kipac. A good case for adding some of these is presented here by Debbie Bard.
...