Benchmarking MD3460 array with Dynamic Disk Pools

The Dell MD3460 controller array can support up to 120 drives with an MD3060e expansion tray attached. Single drive capacities have been steadily increasing, but transfer speeds have remained fairly constant. As a result, RAID6 rebuild times can exceed 24hrs when using 4TB drives. This is the main reason for choosing Dynamic Disk Pools over RAID6 - lower rebuild times are possible since the RAID parity data is distributed across all drives. For the latest Fermi storage configuration, we are benchmarking eight 15-disk DDP volumes on each 120-drive MD3460+MD3060e storage config. Both arrays are dual-attached to a failover pair of servers (fermi-gpfs01,2). In a failover scenario, one machine could serve all sixteen DDP volumes.

The sgpdd-survey benchmark simulates parallel I/O using raw unformatted LUNs. The tests spawn and increasing number of threads (thr) that write and read to an increasing number of concurrent regions (crg). As the tests progress, I/O becomes more random. This set of tests transfer data in 1MB records (rsz).

This first run includes just a single 15-disk LUN. Max aggregate write speed is ~1GB/sec and max read is also ~1GB/sec:

Mon Oct 6 10:26:06 PDT 2014 sgpdd-survey on /dev/raw/raw1 from fermi-gpfs01
total_size 8388608K rsz 1024 crg 1 thr 1 write 438.64 MB/s 1 x 438.94 = 438.94 MB/s read 893.60 MB/s 1 x 894.41 = 894.41 MB/s 
total_size 8388608K rsz 1024 crg 1 thr 2 write 482.91 MB/s 1 x 483.15 = 483.15 MB/s read 882.93 MB/s 1 x 883.72 = 883.72 MB/s 
total_size 8388608K rsz 1024 crg 1 thr 4 write 469.27 MB/s 1 x 469.49 = 469.49 MB/s read 871.93 MB/s 1 x 872.71 = 872.71 MB/s 
total_size 8388608K rsz 1024 crg 1 thr 8 write 476.82 MB/s 1 x 477.05 = 477.05 MB/s read 876.47 MB/s 1 x 877.22 = 877.22 MB/s 
total_size 8388608K rsz 1024 crg 1 thr 16 write 473.50 MB/s 1 x 474.09 = 474.09 MB/s read 869.43 MB/s 1 x 870.23 = 870.23 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 2 write 813.07 MB/s 2 x 406.87 = 813.73 MB/s read 489.03 MB/s 2 x 244.65 = 489.29 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 4 write 855.20 MB/s 2 x 427.98 = 855.96 MB/s read 482.71 MB/s 2 x 241.48 = 482.96 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 8 write 848.02 MB/s 2 x 424.39 = 848.77 MB/s read 493.07 MB/s 2 x 246.66 = 493.32 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 16 write 835.96 MB/s 2 x 418.38 = 836.75 MB/s read 480.13 MB/s 2 x 240.18 = 480.37 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 32 write 830.18 MB/s 2 x 415.48 = 830.96 MB/s read 487.15 MB/s 2 x 243.70 = 487.40 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 4 write 1042.60 MB/s 4 x 260.93 = 1043.70 MB/s read 669.98 MB/s 4 x 167.62 = 670.47 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 8 write 972.11 MB/s 4 x 243.27 = 973.09 MB/s read 668.05 MB/s 4 x 167.13 = 668.53 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 16 write 1019.04 MB/s 4 x 255.04 = 1020.16 MB/s read 640.78 MB/s 4 x 160.30 = 641.21 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 32 write 1034.45 MB/s 4 x 258.92 = 1035.69 MB/s read 656.30 MB/s 4 x 164.19 = 656.78 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 64 write 1016.87 MB/s 4 x 254.52 = 1018.07 MB/s read 644.36 MB/s 4 x 161.21 = 644.84 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 8 write 823.00 MB/s 8 x 102.97 = 823.75 MB/s read 866.24 MB/s 8 x 108.39 = 867.08 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 16 write 788.51 MB/s 8 x 98.66 = 789.26 MB/s read 862.38 MB/s 8 x 107.90 = 863.19 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 32 write 781.63 MB/s 8 x 97.79 = 782.32 MB/s read 863.46 MB/s 8 x 108.03 = 864.26 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 64 write 790.44 MB/s 8 x 98.90 = 791.17 MB/s read 868.12 MB/s 8 x 108.63 = 869.06 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 128 write 787.10 MB/s 8 x 98.49 = 787.89 MB/s read 847.31 MB/s 8 x 106.02 = 848.16 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 16 write 609.99 MB/s 16 x 38.16 = 610.50 MB/s read 1028.24 MB/s 16 x 64.33 = 1029.36 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 32 write 585.41 MB/s 16 x 36.61 = 585.78 MB/s read 1018.23 MB/s 16 x 63.72 = 1019.59 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 64 write 589.83 MB/s 16 x 36.89 = 590.21 MB/s read 1009.79 MB/s 16 x 63.18 = 1010.89 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 128 write 585.71 MB/s 16 x 36.64 = 586.24 MB/s read 1024.85 MB/s 16 x 64.13 = 1026.00 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 256 write 596.42 MB/s 16 x 37.31 = 596.92 MB/s read 1008.59 MB/s 16 x 63.11 = 1009.83 MB/s 
total_size 8388608K rsz 1024 crg 32 thr 32 write 667.16 MB/s 32 x 20.88 = 668.03 MB/s read 946.89 MB/s 32 x 29.62 = 947.88 MB/s 
total_size 8388608K rsz 1024 crg 32 thr 64 write 695.57 MB/s 32 x 21.76 = 696.41 MB/s read 960.61 MB/s 32 x 30.06 = 961.91 MB/s 
total_size 8388608K rsz 1024 crg 32 thr 128 write 694.31 MB/s 32 x 21.72 = 695.19 MB/s read 956.88 MB/s 32 x 29.94 = 957.95 MB/s 
total_size 8388608K rsz 1024 crg 32 thr 256 write 682.67 MB/s 32 x 21.36 = 683.59 MB/s read 951.72 MB/s 32 x 29.79 = 953.37 MB/s 
total_size 8388608K rsz 1024 crg 64 thr 64 write 659.29 MB/s 64 x 10.32 = 660.40 MB/s read 757.79 MB/s 64 x 11.85 = 758.67 MB/s 
total_size 8388608K rsz 1024 crg 64 thr 128 write 656.74 MB/s 64 x 10.28 = 657.96 MB/s read 757.13 MB/s 64 x 11.84 = 758.06 MB/s 
total_size 8388608K rsz 1024 crg 64 thr 256 write 644.79 MB/s 64 x 10.10 = 646.36 MB/s read 756.03 MB/s 64 x 11.84 = 757.45 MB/s 
total_size 8388608K rsz 1024 crg 128 thr 128 write 602.88 MB/s 128 x 4.73 = 605.47 MB/s read 638.55 MB/s 128 x 5.01 = 640.87 MB/s 
total_size 8388608K rsz 1024 crg 128 thr 256 write 588.55 MB/s 128 x 4.61 = 589.60 MB/s read 623.18 MB/s 128 x 4.88 = 625.00 MB/s 
total_size 8388608K rsz 1024 crg 256 thr 256 write 470.98 MB/s 256 x 1.84 = 471.19 MB/s read 564.40 MB/s 256 x 2.22 = 568.85 MB/s