Versions Compared

Old Version 4

changes.mady.by.user Christopher O'Grady

Saved on

compared with

New Version Current

changes.mady.by.user Ric Claus

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

(courtesy of Wilko Kroeger)

Image ModifiedSingle process FileWriter performance

The following plot was obtained from Grafana when testing FileWriter using its BufferedFileWriterMT class, modified to open the file with O_DIRECT.  Each peak corresponds to a record size, starting with 8 MB on the left and increasing by a factor of 2 for each step toward the right, ending with 8 GB records.  From this plot we see that the write performance tops out at ~4.5 GB/s with 512 MB records.

Image Added

A consequence of using O_DIRECT on some file systems is that the buffer alignment and size must be quantized on boundaries given by the file system's block size or the write() system call causes 'write error: Invalid argument' (EINVAL).  The Weka file system used in SRCF does not appear to have this constraint, but the Lustre file system used in NEH (block size 4096) and Lab3's /u1 (block size 512), do.