• 6GB/s input per daq node via PGP
• 2GB/s written to disk per process (up to 10GB/s per node)
• 20GB/s aggregate over all detectors written to disk (per expt)

Filesystem Performance

(courtesy of Wilko Kroeger)

Single process FileWriter performance

The following plot was obtained from Grafana when testing FileWriter using its BufferedFileWriterMT class, modified to open the file with O_DIRECT.  Each peak corresponds to a record size, starting with 8 MB on the left and increasing by a factor of 2 for each step toward the right, ending with 8 GB records.  From this plot we see that the write performance tops out at ~4.5 GB/s with 512 MB records.

A consequence of using O_DIRECT on some file systems is that the buffer alignment and size must be quantized on boundaries given by the file system's block size or the write() system call causes 'write error: Invalid argument' (EINVAL).  The Weka file system used in SRCF does not appear to have this constraint, but the Lustre file system used in NEH (block size 4096) and Lab3's /u1 (block size 512), do.