...
This proposal is to expand the bullet cluster with combined funds from PPA, ATLAS, and Theory. This would double our existing parallel file system size (173->346TB) and add either 1649 or 1904 cores depending on which option we choose. The first option is to provision infiniband (IB) in all nodes and add IB switches to allow additional future expansion of the IB network. Because of the IB network topology allowing future expansion implies a jump in the number of core switches from 4 to 8. The second option would split the cluster into IB and non-IB parts with the ATLAS nodes being non-IB. Note the pricing below is based on several different quotes that would have to be been refreshed. Hence the pricing is approximate and hopefully not low-balledto be verified but very close to actual. The details are:
Option 1: Expand to 18 fully populated chassis with all-IB and future expansion capability (revised for increased IB cost (+6k/chassis))
- 6 full chassis @97.227k => 583.4k (quote 661664648)
- 5 7 blades w/IB added to existing empty slots @5.018k => 2535.1k 2k (quote 661664256)
- 4 IB switches with cables 50.2k (includes active fiber IB cables for lustre switch) (quote 662008993)
- 2 60x2TB disk trays with controllers @30.3k => 60.6k (quote 661663331)
Total is $719$729.3k for 1616 cores and storage expansion.
Gross bullet cluster core count would then be 2960+16161648=4576 4608 (all IB)
costs are:
- ATLAS: 60 blades @4.9375k 96730k => 296.25k 73k (960c) (Based on quote 659024769 for a non-IB full chassis)
- Theory: 97.227k + 5*5.018k => 122 122.32k (336c)
- PPA: 300.73k 310.83k - 2.7k => 308.93 (320c)
...
- (352c)
Notes:
- PPA cost/core is
...
- not meaningful because it includes the storage expansion and subsidizing the IB infrastructure
...
- for the Atlas blades
Benefit here is that we have a uniform cluster.
...