...
- Lustre server version 2.5.1 on RHEL6.5
- 1 MDS and 8 OSS servers using Dell R610 systems
- Four LSI Engenio (Dell MD3260) arrays with dual-redundant controllers
- 16 OSTs using RAID6 LUNs with 2TB NL SAS drives
- ~344TB of usable capacity
- 10Gb ethernet links to SLAC network
- QDR Infiniband links to "bullet" PPA compute cluster
Benchmarking tool is IOR. IOR is a parallel I/O benchmarking utility that runs MPI-coordinated file transfers using . The test clients are "bullet" cluster nodes with Lustre mounted over QDR Infiniband. Maximum throughput is > 6GB/sec using multiple nodes or ~3GB/sec with a single node.
1 process on 1 node
Max Write: 419.58 MiB/sec (439.96 MB/sec)
Max Read: 399.31 MiB/sec (418.71 MB/sec)
Command line used: src/C/IOR -F -a POSIX -r -w -b 60g -t 1m -o /lustre/ki/pfs/yemi/IORFile -O lustreStripeCount=0 -v -i 2
api = POSIX
test filename = /lustre/ki/pfs/yemi/IORFile
access = file-per-process
pattern = segmented (1 segment)
ordering in a file = sequential offsets
ordering inter file= no tasks offsets
clients = 1 (1 per node)
repetitions = 2
xfersize = 1 MiB
blocksize = 60 GiB
aggregate filesize = 60 GiB
Lustre stripe size = Use default
stripe count = Use default
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize
--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
write 419.58 412.07 415.82 3.75 419.58 412.07 415.82 3.75 147.76775 1 1 2 1 0 1 0 0 1 64424509440 1048576 64424509440 -1 POSIX EXCEL
read 399.31 391.06 395.18 4.13 399.31 391.06 395.18 4.13 155.48960 1 1 2 1 0 1 0 0 1 64424509440 1048576 64424509440 -1 POSIX EXCEL
...