Cython multithreading (via openmp) is used in Smd0 for parallel read. This however is currently a bottleneck since one prange loop takes ~1ms to complete as demonstrated in this test.
This test reads 30,000 events from 16 smalldata files.
| Time Spent per Batch (ms)/ BATCH_SIZE | 10000 | 1000 | 100 | 1 |
|---|---|---|---|---|
| Average Time w/o multithreading | 6.20 | 0.63 | 0.07 | 0.0009 |
| Average Time w prange | 2.31 | 1.09 | 0.99 | 0.94 |
| Max Time w/o multithreading | 15.49 | 13.79 | 13.45 | 13.34 |
| Max Time w prange | 4.28 | 2.69 | 3.11 | 9.18 |
| Min Time w/o multithreading | 1.54 | 0.16 | 0.01 | 0.0 ... |
| Min Time w prange | 1.31 | 0.92 | 0.78 | 0.67 |
| Std. Time w/o multithreading | 6.56 | 2.44 | 0.77 | 0.08 |
| Std. Time w prange | 1.39 | 0.30 | 0.14 | 0.13 |
Gdb shows that threads were reused in the case of multithreading thread.
Overview
Content Tools