The Dell MD3460 controller array can support up to 120 drives with an MD3060e expansion tray attached. Single drive capacities have been steadily increasing, but transfer speeds have remained fairly constant. As a result, RAID6 rebuild times can exceed 24hrs when using 4TB drives. This is the main reason for choosing Dynamic Disk Pools over RAID6 - lower rebuild times are possible since the RAID parity data is distributed across all drives. For the latest Fermi storage configuration, we are benchmarking eight 15-disk DDP volumes on each 120-drive MD3460+MD3060e storage config. Both arrays are dual-attached to a failover pair of servers (fermi-gpfs01,2). In a failover scenario, one machine could serve all sixteen DDP volumes.
The sgpdd-survey benchmark simulates parallel I/O using raw unformatted LUNs. The tests spawn and increasing number of threads (thr) that write and read to an increasing number of concurrent regions (crg). As the tests progress, I/O becomes more random. This set of tests transfer data in 1MB records (rsz).
One 15-disk LUNs
Max write ~1GB/sec , Max read ~1GB/sec
Mon Oct 6 10:26:06 PDT 2014 sgpdd-survey on /dev/raw/raw1 from fermi-gpfs01
total_size 8388608K rsz 1024 crg 1 thr 1 write 438.64 MB/s 1 x 438.94 = 438.94 MB/s read 893.60 MB/s 1 x 894.41 = 894.41 MB/s
total_size 8388608K rsz 1024 crg 1 thr 2 write 482.91 MB/s 1 x 483.15 = 483.15 MB/s read 882.93 MB/s 1 x 883.72 = 883.72 MB/s
total_size 8388608K rsz 1024 crg 1 thr 4 write 469.27 MB/s 1 x 469.49 = 469.49 MB/s read 871.93 MB/s 1 x 872.71 = 872.71 MB/s
total_size 8388608K rsz 1024 crg 1 thr 8 write 476.82 MB/s 1 x 477.05 = 477.05 MB/s read 876.47 MB/s 1 x 877.22 = 877.22 MB/s
total_size 8388608K rsz 1024 crg 1 thr 16 write 473.50 MB/s 1 x 474.09 = 474.09 MB/s read 869.43 MB/s 1 x 870.23 = 870.23 MB/s
total_size 8388608K rsz 1024 crg 2 thr 2 write 813.07 MB/s 2 x 406.87 = 813.73 MB/s read 489.03 MB/s 2 x 244.65 = 489.29 MB/s
total_size 8388608K rsz 1024 crg 2 thr 4 write 855.20 MB/s 2 x 427.98 = 855.96 MB/s read 482.71 MB/s 2 x 241.48 = 482.96 MB/s
total_size 8388608K rsz 1024 crg 2 thr 8 write 848.02 MB/s 2 x 424.39 = 848.77 MB/s read 493.07 MB/s 2 x 246.66 = 493.32 MB/s
total_size 8388608K rsz 1024 crg 2 thr 16 write 835.96 MB/s 2 x 418.38 = 836.75 MB/s read 480.13 MB/s 2 x 240.18 = 480.37 MB/s
total_size 8388608K rsz 1024 crg 2 thr 32 write 830.18 MB/s 2 x 415.48 = 830.96 MB/s read 487.15 MB/s 2 x 243.70 = 487.40 MB/s
total_size 8388608K rsz 1024 crg 4 thr 4 write 1042.60 MB/s 4 x 260.93 = 1043.70 MB/s read 669.98 MB/s 4 x 167.62 = 670.47 MB/s
total_size 8388608K rsz 1024 crg 4 thr 8 write 972.11 MB/s 4 x 243.27 = 973.09 MB/s read 668.05 MB/s 4 x 167.13 = 668.53 MB/s
total_size 8388608K rsz 1024 crg 4 thr 16 write 1019.04 MB/s 4 x 255.04 = 1020.16 MB/s read 640.78 MB/s 4 x 160.30 = 641.21 MB/s
total_size 8388608K rsz 1024 crg 4 thr 32 write 1034.45 MB/s 4 x 258.92 = 1035.69 MB/s read 656.30 MB/s 4 x 164.19 = 656.78 MB/s
total_size 8388608K rsz 1024 crg 4 thr 64 write 1016.87 MB/s 4 x 254.52 = 1018.07 MB/s read 644.36 MB/s 4 x 161.21 = 644.84 MB/s
total_size 8388608K rsz 1024 crg 8 thr 8 write 823.00 MB/s 8 x 102.97 = 823.75 MB/s read 866.24 MB/s 8 x 108.39 = 867.08 MB/s
total_size 8388608K rsz 1024 crg 8 thr 16 write 788.51 MB/s 8 x 98.66 = 789.26 MB/s read 862.38 MB/s 8 x 107.90 = 863.19 MB/s
total_size 8388608K rsz 1024 crg 8 thr 32 write 781.63 MB/s 8 x 97.79 = 782.32 MB/s read 863.46 MB/s 8 x 108.03 = 864.26 MB/s
total_size 8388608K rsz 1024 crg 8 thr 64 write 790.44 MB/s 8 x 98.90 = 791.17 MB/s read 868.12 MB/s 8 x 108.63 = 869.06 MB/s
total_size 8388608K rsz 1024 crg 8 thr 128 write 787.10 MB/s 8 x 98.49 = 787.89 MB/s read 847.31 MB/s 8 x 106.02 = 848.16 MB/s
total_size 8388608K rsz 1024 crg 16 thr 16 write 609.99 MB/s 16 x 38.16 = 610.50 MB/s read 1028.24 MB/s 16 x 64.33 = 1029.36 MB/s
total_size 8388608K rsz 1024 crg 16 thr 32 write 585.41 MB/s 16 x 36.61 = 585.78 MB/s read 1018.23 MB/s 16 x 63.72 = 1019.59 MB/s
total_size 8388608K rsz 1024 crg 16 thr 64 write 589.83 MB/s 16 x 36.89 = 590.21 MB/s read 1009.79 MB/s 16 x 63.18 = 1010.89 MB/s
total_size 8388608K rsz 1024 crg 16 thr 128 write 585.71 MB/s 16 x 36.64 = 586.24 MB/s read 1024.85 MB/s 16 x 64.13 = 1026.00 MB/s
total_size 8388608K rsz 1024 crg 16 thr 256 write 596.42 MB/s 16 x 37.31 = 596.92 MB/s read 1008.59 MB/s 16 x 63.11 = 1009.83 MB/s
total_size 8388608K rsz 1024 crg 32 thr 32 write 667.16 MB/s 32 x 20.88 = 668.03 MB/s read 946.89 MB/s 32 x 29.62 = 947.88 MB/s
total_size 8388608K rsz 1024 crg 32 thr 64 write 695.57 MB/s 32 x 21.76 = 696.41 MB/s read 960.61 MB/s 32 x 30.06 = 961.91 MB/s
total_size 8388608K rsz 1024 crg 32 thr 128 write 694.31 MB/s 32 x 21.72 = 695.19 MB/s read 956.88 MB/s 32 x 29.94 = 957.95 MB/s
total_size 8388608K rsz 1024 crg 32 thr 256 write 682.67 MB/s 32 x 21.36 = 683.59 MB/s read 951.72 MB/s 32 x 29.79 = 953.37 MB/s
total_size 8388608K rsz 1024 crg 64 thr 64 write 659.29 MB/s 64 x 10.32 = 660.40 MB/s read 757.79 MB/s 64 x 11.85 = 758.67 MB/s
total_size 8388608K rsz 1024 crg 64 thr 128 write 656.74 MB/s 64 x 10.28 = 657.96 MB/s read 757.13 MB/s 64 x 11.84 = 758.06 MB/s
total_size 8388608K rsz 1024 crg 64 thr 256 write 644.79 MB/s 64 x 10.10 = 646.36 MB/s read 756.03 MB/s 64 x 11.84 = 757.45 MB/s
total_size 8388608K rsz 1024 crg 128 thr 128 write 602.88 MB/s 128 x 4.73 = 605.47 MB/s read 638.55 MB/s 128 x 5.01 = 640.87 MB/s
total_size 8388608K rsz 1024 crg 128 thr 256 write 588.55 MB/s 128 x 4.61 = 589.60 MB/s read 623.18 MB/s 128 x 4.88 = 625.00 MB/s
total_size 8388608K rsz 1024 crg 256 thr 256 write 470.98 MB/s 256 x 1.84 = 471.19 MB/s read 564.40 MB/s 256 x 2.22 = 568.85 MB/s
Four 15-disk LUNs
Max write ~2GB/sec , Max read ~3.7GB/sec
total_size 33554432K rsz 1024 crg 4 thr 4 write 1328.92 MB/s 4 x 332.39 = 1329.57 MB/s read 2911.00 MB/s 4 x 728.76 = 2915.04 MB/s
total_size 33554432K rsz 1024 crg 4 thr 8 write 1454.79 MB/s 4 x 364.14 = 1456.57 MB/s read 2895.55 MB/s 4 x 724.64 = 2898.56 MB/s
total_size 33554432K rsz 1024 crg 4 thr 16 write 1456.28 MB/s 4 x 364.27 = 1457.10 MB/s read 2458.23 MB/s 4 x 615.12 = 2460.48 MB/s
total_size 33554432K rsz 1024 crg 4 thr 32 write 1445.62 MB/s 4 x 361.72 = 1446.88 MB/s read 2994.64 MB/s 4 x 749.94 = 2999.76 MB/s
total_size 33554432K rsz 1024 crg 4 thr 64 write 1434.84 MB/s 4 x 358.97 = 1435.89 MB/s read 2740.28 MB/s 4 x 685.58 = 2742.31 MB/s
total_size 33554432K rsz 1024 crg 8 thr 8 write 2001.29 MB/s 8 x 250.36 = 2002.87 MB/s read 1639.01 MB/s 8 x 204.96 = 1639.71 MB/s
total_size 33554432K rsz 1024 crg 8 thr 16 write 1980.60 MB/s 8 x 247.88 = 1983.03 MB/s read 1440.08 MB/s 8 x 180.07 = 1440.58 MB/s
total_size 33554432K rsz 1024 crg 8 thr 32 write 1947.07 MB/s 8 x 243.69 = 1949.54 MB/s read 1661.00 MB/s 8 x 207.72 = 1661.76 MB/s
total_size 33554432K rsz 1024 crg 8 thr 64 write 2016.98 MB/s 8 x 252.47 = 2019.73 MB/s read 1715.79 MB/s 8 x 214.57 = 1716.54 MB/s
total_size 33554432K rsz 1024 crg 8 thr 128 write 1961.61 MB/s 8 x 245.53 = 1964.26 MB/s read 1682.21 MB/s 8 x 210.47 = 1683.73 MB/s
total_size 33554432K rsz 1024 crg 16 thr 16 write 2097.46 MB/s 16 x 131.28 = 2100.52 MB/s read 2379.03 MB/s 16 x 148.78 = 2380.52 MB/s
total_size 33554432K rsz 1024 crg 16 thr 32 write 2078.31 MB/s 16 x 130.00 = 2080.08 MB/s read 2389.99 MB/s 16 x 149.53 = 2392.43 MB/s
total_size 33554432K rsz 1024 crg 16 thr 64 write 2075.24 MB/s 16 x 129.81 = 2077.03 MB/s read 2193.77 MB/s 16 x 137.20 = 2195.13 MB/s
total_size 33554432K rsz 1024 crg 16 thr 128 write 2088.21 MB/s 16 x 130.62 = 2090.00 MB/s read 2206.04 MB/s 16 x 137.96 = 2207.34 MB/s
total_size 33554432K rsz 1024 crg 16 thr 256 write 2085.91 MB/s 16 x 130.49 = 2087.86 MB/s read 2393.09 MB/s 16 x 149.68 = 2394.87 MB/s
total_size 33554432K rsz 1024 crg 32 thr 32 write 2008.13 MB/s 32 x 62.81 = 2009.89 MB/s read 3197.92 MB/s 32 x 100.09 = 3202.82 MB/s
total_size 33554432K rsz 1024 crg 32 thr 64 write 1979.69 MB/s 32 x 61.92 = 1981.51 MB/s read 3190.03 MB/s 32 x 99.79 = 3193.36 MB/s
total_size 33554432K rsz 1024 crg 32 thr 128 write 1997.90 MB/s 32 x 62.49 = 1999.82 MB/s read 3237.11 MB/s 32 x 101.26 = 3240.36 MB/s
total_size 33554432K rsz 1024 crg 32 thr 256 write 2005.33 MB/s 32 x 62.71 = 2006.84 MB/s read 3213.86 MB/s 32 x 100.64 = 3220.52 MB/s
total_size 33554432K rsz 1024 crg 32 thr 512 write 1981.90 MB/s 32 x 62.00 = 1983.95 MB/s read 3277.99 MB/s 32 x 102.53 = 3280.94 MB/s
total_size 33554432K rsz 1024 crg 64 thr 64 write 1986.19 MB/s 64 x 31.06 = 1987.92 MB/s read 3835.45 MB/s 64 x 60.01 = 3840.33 MB/s
total_size 33554432K rsz 1024 crg 64 thr 128 write 1985.35 MB/s 64 x 31.06 = 1987.92 MB/s read 3804.40 MB/s 64 x 59.53 = 3809.81 MB/s
total_size 33554432K rsz 1024 crg 64 thr 256 write 2009.90 MB/s 64 x 31.47 = 2014.16 MB/s read 3827.19 MB/s 64 x 59.99 = 3839.11 MB/s
total_size 33554432K rsz 1024 crg 64 thr 512 write 1986.98 MB/s 64 x 31.09 = 1989.75 MB/s read 3876.90 MB/s 64 x 60.76 = 3888.55 MB/s
total_size 33554432K rsz 1024 crg 64 thr 1024 write 1978.71 MB/s 64 x 30.96 = 1981.20 MB/s read 3807.05 MB/s 64 x 59.57 = 3812.26 MB/s
total_size 33554432K rsz 1024 crg 128 thr 128 write 1999.93 MB/s 128 x 15.65 = 2003.17 MB/s read 3608.74 MB/s 128 x 28.24 = 3614.50 MB/s
total_size 33554432K rsz 1024 crg 128 thr 256 write 2016.86 MB/s 128 x 15.80 = 2022.71 MB/s read 3600.97 MB/s 128 x 28.17 = 3605.96 MB/s
total_size 33554432K rsz 1024 crg 128 thr 512 write 2012.85 MB/s 128 x 15.75 = 2016.60 MB/s read 3541.17 MB/s 128 x 27.71 = 3547.36 MB/s
total_size 33554432K rsz 1024 crg 128 thr 1024 write 2012.09 MB/s 128 x 15.75 = 2016.60 MB/s read 3506.41 MB/s 128 x 27.54 = 3525.39 MB/s
total_size 33554432K rsz 1024 crg 256 thr 256 write 1868.84 MB/s 256 x 7.61 = 1948.24 MB/s read 2809.95 MB/s 256 x 11.05 = 2829.59 MB/s
total_size 33554432K rsz 1024 crg 256 thr 512 write 1944.84 MB/s 256 x 7.65 = 1958.01 MB/s read 2729.52 MB/s 256 x 10.76 = 2753.91 MB/s
total_size 33554432K rsz 1024 crg 256 thr 1024 write 1960.35 MB/s 256 x 7.69 = 1967.77 MB/s read 2814.37 MB/s 256 x 11.07 = 2834.47 MB/s
total_size 33554432K rsz 1024 crg 512 thr 512 write 1671.90 MB/s 512 x 3.29 = 1684.57 MB/s read 2378.45 MB/s 512 x 4.67 = 2392.58 MB/s
total_size 33554432K rsz 1024 crg 512 thr 1024 write 1637.35 MB/s 512 x 3.22 = 1650.39 MB/s read 2373.27 MB/s 512 x 4.68 = 2397.46 MB/s
total_size 33554432K rsz 1024 crg 1024 thr 1024 write 1231.95 MB/s 1024 x 1.20 = 1230.47 MB/s read 2094.72 MB/s 1024 x 2.06 = 2109.38 MB/s
dfdf