Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • hsd-hsd outer product (symmetric, same hsd, save a factor of 2)
  • hsd-fzp outer product (fzp is piranha: 2048)

Note: Timing values shown below are from single precision variables

Code Block
languagebash
(ps-4.5.26) monarin@drp-srcf-eb003 (master *) tmolw8819 👁)$ mpirun -n 3 python run_outer_products.py 0 1 2048 59400
nevent: 1 vls: (2048,) inner: (59400,) outer: (59400,) dt:159.693s86s. load:0.037s04s calc:159.656s outer vls/wf:0.29s, wf/wf:6.68s add vls/wf:0.29s wf/wf:8.39s tt:15.693s82s tt:9.86s. rate:0.06Hz10Hz
nevent: 2 vls: (2048,) inner: (59400,) outer: (59400,) dt:106.523s37s. load:0.004s00s calc:10.519s outer vls/wf:0.29s, wf/wf:7.69s add vls/wf:0.09s wf/wf:2.45s tt:26.216s6.36s tt:16.22s. rate:0.08Hz12Hz
nevent: 3 vls: (2048,) inner: (59400,) outer: (59400,) dt:106.526s37s. load:0.003s00s calc:106.523s outer vls/wf:0.29s, wf/wf:7.70s add vls/wf:0.09s wf/wf:2.44s tt:36.742s36s tt:22.59s. rate:0.08Hz13Hz
nevent: 4 vls: (2048,) inner: (59400,) outer: (59400,) dt:106.445s38s. load:0.003s00s calc:10.442s outer vls/wf:0.34s, wf/wf:7.57s add vls/wf:0.09s wf/wf:2.44s tt:47.187s6.38s tt:28.97s. rate:0.08Hz14Hz
nevent: 5 vls: (2048,) inner: (59400,) outer: (59400,) dt:106.627s37s. load:0.003s00s calc:10.624s outer vls/wf:0.33s, wf/wf:7.74s add vls/wf:0.09s wf/wf:2.46s tt:57.814s6.37s tt:35.34s. rate:0.09Hz14Hz
nevent: 6 vls: (2048,) inner: (59400,) outer: (59400,) dt:106.420s34s. load:0.003s00s calc:10.417s outer vls/wf:0.33s, wf/wf:7.56s add vls/wf:0.09s wf/wf:2.44s6.33s tt:6841.234s68s. rate:0.09Hz14Hz
nevent: 7 vls: (2048,) inner: (59400,) outer: (59400,) dt:106.580s37s. load:0.003s00s calc:106.577s outer vls/wf:0.35s, wf/wf:7.69s add vls/wf:0.09s wf/wf:2.45s tt:78.814s36s tt:48.04s. rate:0.09Hz15Hz
nevent: 8 vls: (2048,) inner: (59400,) outer: (59400,) dt:106.469s35s. load:0.005s00s calc:10.463s outer vls/wf:0.33s, wf/wf:7.59s add vls/wf:0.09s wf/wf:2.46s tt:89.283s6.34s tt:54.39s. rate:0.09Hz15Hz
nevent: 8 tt:9256.53516s44s. rate:0.011566894771298394kHz00706kHz
(ps-4.5.26) monarin@drp-srcf-eb003 (master *) tmolw8819 👁)$ mpirun -n 3 python run_outer_products.py 0 1 2048 8000
nevent: 1 vls: (2048,) inner: (8000,) outer: (8000,) dt:0.495s30s. load:0.035s03s calc:0.461s outer vls/wf:0.04s, wf/wf:0.18s add vls/wf:0.05s wf/wf:0.19s26s tt:0.495s30s. rate:23.02Hz39Hz
nevent: 2 vls: (2048,) inner: (8000,) outer: (8000,) dt:0.323s17s. load:0.004s00s calc:0.319s outer vls/wf:0.06s, wf/wf:0.21s add vls/wf:0.01s wf/wf:0.05s 16s tt:0.818s46s. rate:24.44Hz31Hz
nevent: 3 vls: (2048,) inner: (8000,) outer: (8000,) dt:0.323s17s. load:0.003s00s calc:0.321s outer vls/wf:0.06s, wf/wf:0.21s add vls/wf:0.01s wf/wf:0.05s tt:1.142s. rate:2.63Hz16s tt:0.63s. rate:4.76Hz
nevent: 4 vls: (2048,) inner: (8000,) outer: (8000,) dt:0.319s17s. load:0.003s00s calc:0.316s outer vls/wf16s tt:0.06s, wf/wf:0.20s add vls/wf:0.01s wf/wf:0.05s tt:1.461s. rate:2.74Hz80s. rate:5.01Hz
nevent: 5 vls: (2048,) inner: (8000,) outer: (8000,) dt:0.324s17s. load:0.003s00s calc:0.321s outer vls/wf16s tt:0.06s, wf/wf:0.21s add vls/wf:0.01s wf/wf:0.05s tt:1.784s. rate:2.80Hz97s. rate:5.18Hz
nevent: 6 vls: (2048,) inner: (8000,) outer: (8000,) dt:0.319s17s. load:0.003s00s calc:0.316s outer vls/wf:0.06s, wf/wf:0.20s add vls/wf:0.01s wf/wf:0.05s tt:2.103s16s tt:1.13s. rate:25.85Hz30Hz
nevent: 7 vls: (2048,) inner: (8000,) outer: (8000,) dt:0.323s17s. load:0.003s00s calc:0.320s outer vls/wf:0.06s, wf/wf:0.21s add vls/wf:0.01s wf/wf:0.05s tt:2.426s16s tt:1.30s. rate:25.89Hz39Hz
nevent: 8 vls: (2048,) inner: (8000,) outer: (8000,) dt:0.322s17s. load:0.005s00s calc:0.317s outer vls/wf:0.05s, wf/wf:0.20s add vls/wf:0.01s wf/wf:0.05s tt:2.748s16s tt:1.47s. rate:25.91Hz45Hz
nevent: 8 tt:94.95117s29s. rate:0.00124389665434137kHz00054kHz

Reducing wf length from 59400 to 8000,

...

the 2 outer products ran 40x faster.

mona should test:

  • the "sparse" outer product from fex
  • full 8000x8000 (plus 8000x2048 for fzp):
    • we can do 8000 samples from 2 hsd channels per drp node at 100kHz (3.2GB/s) from a data-volume perspective but need to check from a CPU perspective.

Memory Usage

the full-matrix memory usage with mpi: looks like 8GB for 60 cores for 8000 samples

...

Code Block
languagebash
>>> b,vls,hsd=(8,2048,8000)
>>> ((vls*b*2)+(hsd*b*2)+(vls*hsd*b)+(hsd*hsd*b))/1e9
0.643232768
>>> b,vls,hsd=(4,2048,8000)
>>> ((vls*b*2)+(hsd*b*2)+(vls*hsd*b)+(hsd*hsd*b))/1e9
0.321616384
>>> b,vls,hsd=(2,2048,8000)
>>> ((vls*b*2)+(hsd*b*2)+(vls*hsd*b)+(hsd*hsd*b))/1e9
0.160808192

Observed memory usage

Single precision

1 core: 1.3GB

10 cores: 7.9GB (0.8GB per core)

50 cores: 17.3GB (0.35GB per core)

Double precision

50 cores: 50.8GB (1GB per core)

in future with 1MHz hopefully have a better detector for fzp that also runs at 1MHz

...