Page History

The Dell MD3460 controller array can support up to 120 drives with an MD3060e expansion tray attached. Single drive capacities have been steadily increasing, but transfer speeds have remained fairly constant. As a result, RAID6 rebuild times can exceed 24hrs when using 4TB drives. This is the main reason for choosing Dynamic Disk Pools over RAID6 - (DDP) provide lower rebuild times are possible since the RAID parity data is distributed compared to RAID6 because the hotspare space is spread across all drives in the pool. For the latest Fermi storage configuration, we are benchmarking eight 15-disk DDP volumes on each 120-drive MD3460+MD3060e storage config. Both arrays are dual-attached to a failover pair of servers (fermi-gpfs01,2). In a failover scenario, one machine could serve all sixteen DDP volumes.

The sgpdd-survey benchmark simulates performs parallel I/O O using raw unformatted LUNs. The tests spawn and an increasing number of threads (thr) that write and read to an increasing number of concurrent regions (crg). As the tests progress, I/O becomes more random. This set of tests transfer data in The transfer operations use 1MB records/chunks (rsz).

One 15-disk LUNs

Max aggregate write speed is ~1GB/sec and max read is also ~1GB/sec:

This is a local test that does not require any networking or formatted filesystem. The purpose is to understand the performance limits of the storage hardware before we create filesystems (GPFS, Lustre, etc.). The benchmark results show the MD3460 is capable of sustaining throughput well above 2GB/sec. In order to avoid networking bottlenecks, we may use 4x10Gb ethernet link aggregation instead of the usual 2x10Gb. We also have a limited number of 40Gb ports on the SLAC network.

 One 15-disk LUN

Max write ~1GB/sec , Max read ~1GB/sec

Mon Oct 6 10:26:06 PDT 2014 sgpdd-survey on /dev/raw/raw1 from fermi-gpfs01
total_size 8388608K rsz 1024 crg 1 thr 1 write 438.64 MB/s 1 x 438.94 = 438.94 MB/s read 893.60 MB/s 1 x 894.41 = 894.4164 MB/s 1 x 438.94 = 438.94 MB/s read 893.60 MB/s 1 x 894.41 = 894.41 MB/s 
total_size 8388608K rsz 1024 crg 1 thr 2 write 482.91 MB/s 1 x 483.15 = 483.15 MB/s read 882.93 MB/s 1 x 883.72 = 883.72 MB/s 
total_size 8388608K rsz 1024 crg 1 thr 4 write 469.27 MB/s 1 x 469.49 = 469.49 MB/s read 871.93 MB/s 1 x 872.71 = 872.71 MB/s 
total_size 8388608K rsz 1024 crg 1 thr 8 write 476.82 MB/s 1 x 477.05 = 477.05 MB/s read 876.47 MB/s 1 x 877.22 = 877.22 MB/s 
total_size 8388608K rsz 1024 crg 1 thr 16 write 473.50 MB/s 1 x 474.09 = 474.09 MB/s read 869.43 MB/s 1 x 870.23 = 870.23 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 2 write 813.07 MB/s 2 x 406.87 = 813.73 MB/s read 489.03 MB/s 2 x 244.65 = 489.29 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 4 write 855.20 MB/s 2 x 427.98 = 855.96 MB/s read 482.71 MB/s 2 x 241.48 = 482.96 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 8 write 848.02 MB/s 2 x 424.39 = 848.77 MB/s read 493.07 MB/s 2 x 246.66 = 493.32 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 16 write 835.96 MB/s 2 x 418.38 = 836.75 MB/s read 480.13 MB/s 2 x 240.18 = 480.37 MB/s 
total_size 8388608K rsz 1024 crg 2 thr 32 write 830.18 MB/s 2 x 415.48 = 830.96 MB/s read 487.15 MB/s 2 x 243.70 = 487.40 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 4 write 1042.60 MB/s 4 x 260.93 = 1043.70 MB/s read 669.98 MB/s 4 x 167.62 = 670.47 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 8 write 972.11 MB/s 4 x 243.27 = 973.09 MB/s read 668.05 MB/s 4 x 167.13 = 668.53 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 16 write 1019.04 MB/s 4 x 255.04 = 1020.16 MB/s read 640.78 MB/s 4 x 160.30 = 641.21 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 32 write 1034.45 MB/s 4 x 258.92 = 1035.69 MB/s read 656.30 MB/s 4 x 164.19 = 656.78 MB/s 
total_size 8388608K rsz 1024 crg 4 thr 64 write 1016.87 MB/s 4 x 254.52 = 1018.07 MB/s read 644.36 MB/s 4 x 161.21 = 644.84 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 8 write 823.00 MB/s 8 x 102.97 = 823.75 MB/s read 866.24 MB/s 8 x 108.39 = 867.08 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 16 write 788.51 MB/s 8 x 98.66 = 789.26 MB/s read 862.38 MB/s 8 x 107.90 = 863.19 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 32 write 781.63 MB/s 8 x 97.79 = 782.32 MB/s read 863.46 MB/s 8 x 108.03 = 864.26 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 64 write 790.44 MB/s 8 x 98.90 = 791.17 MB/s read 868.12 MB/s 8 x 108.63 = 869.06 MB/s 
total_size 8388608K rsz 1024 crg 8 thr 128 write 787.10 MB/s 8 x 98.49 = 787.89 MB/s read 847.31 MB/s 8 x 106.02 = 848.16 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 16 write 609.99 MB/s 16 x 38.16 = 610.50 MB/s read 1028.24 MB/s 16 x 64.33 = 1029.36 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 32 write 585.41 MB/s 16 x 36.61 = 585.78 MB/s read 1018.23 MB/s 16 x 63.72 = 1019.59 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 64 write 589.83 MB/s 16 x 36.89 = 590.21 MB/s read 1009.79 MB/s 16 x 63.18 = 1010.89 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 128 write 585.71 MB/s 16 x 36.64 = 586.24 MB/s read 1024.85 MB/s 16 x 64.13 = 1026.00 MB/s 
total_size 8388608K rsz 1024 crg 16 thr 256 write 596.42 MB/s 16 x 37.31 = 596.92 MB/s read 1008.59 MB/s 16 x 63.11 = 1009.83 MB/s 
total_size 8388608K rsz 1024 crg 32 thr 32 write 667.16 MB/s 32 x 20.88 = 668.03 MB/s read 946.89 MB/s 32 x 29.62 = 947.88 MB/s 
total_size 8388608K rsz 1024 crg 32 thr 64 write 695.57 MB/s 32 x 21.76 = 696.41 MB/s read 960.61 MB/s 32 x 30.06 = 961.91 MB/s 
total_size 8388608K rsz 1024 crg 32 thr 128 write 694.31 MB/s 32 x 21.72 = 695.19 MB/s read 956.88 MB/s 32 x 29.94 = 957.95 MB/s 
total_size 8388608K rsz 1024 crg 32 thr 256 write 682.67 MB/s 32 x 21.36 = 683.59 MB/s read 951.72 MB/s 32 x 29.79 = 953.37 MB/s 
total_size 8388608K rsz 1024 crg 64 thr 64 write 659.29 MB/s 64 x 10.32 = 660.40 MB/s read 757.79 MB/s 64 x 11.85 = 758.67 MB/s 
total_size 8388608K rsz 1024 crg 64 thr 128 write 656.74 MB/s 64 x 10.28 = 657.96 MB/s read 757.13 MB/s 64 x 11.84 = 758.06 MB/s 
total_size 8388608K rsz 1024 crg 64 thr 256 write 644.79 MB/s 64 x 10.10 = 646.36 MB/s read 756.03 MB/s 64 x 11.84 = 757.45 MB/s 
total_size 8388608K rsz 1024 crg 128 thr 128 write 602.88 MB/s 128 x 4.73 = 605.47 MB/s read 638.55 MB/s 128 x 5.01 = 640.87 MB/s 
total_size 8388608K rsz 1024 crg 128 thr 256 write 588.55 MB/s 128 x 4.61 = 589.60 MB/s read 623.18 MB/s 128 x 4.88 = 625.00 MB/s 
total_size 8388608K rsz 1024 crg 256 thr 256 write 470.98 MB/s 256 x 1.84 = 471.19 MB/s read 564.40 MB/s 256 x 2.22 = 568.85 MB/s

Four 15-disk LUNs

Max write ~2GB/sec , Max read ~3.8GB/sec

Mon Oct 6 11:06:36 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 from fermi-gpfs01
total_size 33554432K rsz 1024 crg 4 thr 4 write 1328.92 MB/s 4 x 332.39 = 1329.57 MB/s read 2911.00 MB/s 4 x 728.76 = 2915.04 MB/s 
total_size 33554432K rsz 1024 crg 4 thr 8 write 1454.79 MB/s 4 x 364.14 = 1456.57 MB/s read 2895.55 MB/s 4 x 724.64 = 2898.56 MB/s 
total_size 33554432K rsz 1024 crg 4 thr 16 write 1456.28 MB/s 4 x 364.27 = 1457.10 MB/s read 2458.23 MB/s 4 x 615.12 = 2460.48 MB/s 
total_size 33554432K rsz 1024 crg 4 thr 32 write 1445.62 MB/s 4 x 361.72 = 1446.88 MB/s read 2994.64 MB/s 4 x 749.94 = 2999.76 MB/s 
total_size 33554432K rsz 1024 crg 4 thr 64 write 1434.84 MB/s 4 x 358.97 = 1435.89 MB/s read 2740.28 MB/s 4 x 685.58 = 2742.31 MB/s 
total_size 33554432K rsz 1024 crg 8 thr 8 write 2001.29 MB/s 8 x 250.36 = 2002.87 MB/s read 1639.01 MB/s 8 x 204.96 = 1639.71 MB/s 
total_size 33554432K rsz 1024 crg 8 thr 16 write 1980.60 MB/s 8 x 247.88 = 1983.03 MB/s read 1440.08 MB/s 8 x 180.07 = 1440.58 MB/s 
total_size 33554432K rsz 1024 crg 8 thr 32 write 1947.07 MB/s 8 x 243.69 = 1949.54 MB/s read 1661.00 MB/s 8 x 207.72 = 1661.76 MB/s 
total_size 33554432K rsz 1024 crg 8 thr 64 write 2016.98 MB/s 8 x 252.47 = 2019.73 MB/s read 1715.79 MB/s 8 x 214.57 = 1716.54 MB/s 
total_size 33554432K rsz 1024 crg 8 thr 128 write 1961.61 MB/s 8 x 245.53 = 1964.26 MB/s read 1682.21 MB/s 8 x 210.47 = 1683.73 MB/s 
total_size 33554432K rsz 1024 crg 16 thr 16 write 2097.46 MB/s 16 x 131.28 = 2100.52 MB/s read 2379.03 MB/s 16 x 148.78 = 2380.52 MB/s 
total_size 33554432K rsz 1024 crg 16 thr 32 write 2078.31 MB/s 16 x 130.00 = 2080.08 MB/s read 2389.99 MB/s 16 x 149.53 = 2392.43 MB/s 
total_size 33554432K rsz 1024 crg 16 thr 64 write 2075.24 MB/s 16 x 129.81 = 2077.03 MB/s read 2193.77 MB/s 16 x 137.20 = 2195.13 MB/s 
total_size 33554432K rsz 1024 crg 16 thr 128 write 2088.21 MB/s 16 x 130.62 = 2090.00 MB/s read 2206.04 MB/s 16 x 137.96 = 2207.34 MB/s 
total_size 33554432K rsz 1024 crg 16 thr 256 write 2085.91 MB/s 16 x 130.49 = 2087.86 MB/s read 2393.09 MB/s 16 x 149.68 = 2394.87 MB/s 
total_size 33554432K rsz 1024 crg 32 thr 32 write 2008.13 MB/s 32 x 62.81 = 2009.89 MB/s read 3197.92 MB/s 32 x 100.09 = 3202.82 MB/s 
total_size 33554432K rsz 1024 crg 32 thr 64 write 1979.69 MB/s 32 x 61.92 = 1981.51 MB/s read 3190.03 MB/s 32 x 99.79 = 3193.36 MB/s 
total_size 33554432K rsz 1024 crg 32 thr 128 write 1997.90 MB/s 32 x 62.49 = 1999.82 MB/s read 3237.11 MB/s 32 x 101.26 = 3240.36 MB/s 
total_size 33554432K rsz 1024 crg 32 thr 256 write 2005.33 MB/s 32 x 62.71 = 2006.84 MB/s read 3213.86 MB/s 32 x 100.64 = 3220.52 MB/s 
total_size 33554432K rsz 1024 crg 32 thr 512 write 1981.90 MB/s 32 x 62.00 = 1983.95 MB/s read 3277.99 MB/s 32 x 102.53 = 3280.94 MB/s 
total_size 33554432K rsz 1024 crg 64 thr 64 write 1986.19 MB/s 64 x 31.06 = 1987.92 MB/s read 3835.45 MB/s 64 x 60.01 = 3840.33 MB/s 
total_size 33554432K rsz 1024 crg 64 thr 128 write 1985.35 MB/s 64 x 31.06 = 1987.92 MB/s read 3804.40 MB/s 64 x 59.53 = 3809.81 MB/s 
total_size 33554432K rsz 1024 crg 64 thr 256 write 2009.90 MB/s 64 x 31.47 = 2014.16 MB/s read 3827.19 MB/s 64 x 59.99 = 3839.11 MB/s 
total_size 33554432K rsz 1024 crg 64 thr 512 write 1986.98 MB/s 64 x 31.09 = 1989.75 MB/s read 3876.90 MB/s 64 x 60.76 = 3888.55 MB/s 
total_size 33554432K rsz 1024 crg 64 thr 1024 write 1978.71 MB/s 64 x 30.96 = 1981.20 MB/s read 3807.05 MB/s 64 x 59.57 = 3812.26 MB/s 
total_size 33554432K rsz 1024 crg 128 thr 128 write 1999.93 MB/s 128 x 15.65 = 2003.17 MB/s read 3608.74 MB/s 128 x 28.24 = 3614.50 MB/s 
total_size 33554432K rsz 1024 crg 128 thr 256 write 2016.86 MB/s 128 x 15.80 = 2022.71 MB/s read 3600.97 MB/s 128 x 28.17 = 3605.96 MB/s 
total_size 33554432K rsz 1024 crg 128 thr 512 write 2012.85 MB/s 128 x 15.75 = 2016.60 MB/s read 3541.17 MB/s 128 x 27.71 = 3547.36 MB/s 
total_size 33554432K rsz 1024 crg 128 thr 1024 write 2012.09 MB/s 128 x 15.75 = 2016.60 MB/s read 3506.41 MB/s 128 x 27.54 = 3525.39 MB/s 
total_size 33554432K rsz 1024 crg 256 thr 256 write 1868.84 MB/s 256 x 7.61 = 1948.24 MB/s read 2809.95 MB/s 256 x 11.05 = 2829.59 MB/s 
total_size 33554432K rsz 1024 crg 256 thr 512 write 1944.84 MB/s 256 x 7.65 = 1958.01 MB/s read 2729.52 MB/s 256 x 10.76 = 2753.91 MB/s 
total_size 33554432K rsz 1024 crg 256 thr 1024 write 1960.35 MB/s 256 x 7.69 = 1967.77 MB/s read 2814.37 MB/s 256 x 11.07 = 2834.47 MB/s 
total_size 33554432K rsz 1024 crg 512 thr 512 write 1671.90 MB/s 512 x 3.29 = 1684.57 MB/s read 2378.45 MB/s 512 x 4.67 = 2392.58 MB/s 
total_size 33554432K rsz 1024 crg 512 thr 1024 write 1637.35 MB/s 512 x 3.22 = 1650.39 MB/s read 2373.27 MB/s 512 x 4.68 = 2397.46 MB/s 
total_size 33554432K rsz 1024 crg 1024 thr 1024 write 1231.95 MB/s 1024 x 1.20 = 1230.47 MB/s read 2094.72 MB/s 1024 x 2.06 = 2109.38 MB/s

Eight 15-disk LUNs

Max write ~2GB/sec , Max read ~5.5GB/sec

Mon Oct 6 11:27:15 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 from fermi-gpfs01

total_size 67108864K rsz 1024 crg 8 thr 8 write 1922.79 MB/s 8 x 240.58 = 1924.67 MB/s read 3002.11 MB/s 8 x 375.48 = 3003.85 MB/s 
total_size 67108864K rsz 1024 crg 8 thr 16 write 1954.89 MB/s 8 x 244.51 = 1956.10 MB/s read 3067.43 MB/s 8 x 383.64 = 3069.15 MB/s 
total_size 67108864K rsz 1024 crg 8 thr 32 write 1945.17 MB/s 8 x 243.23 = 1945.88 MB/s read 3554.62 MB/s 8 x 444.55 = 3556.37 MB/s 
total_size 67108864K rsz 1024 crg 8 thr 64 write 1963.69 MB/s 8 x 245.61 = 1964.87 MB/s read 3241.54 MB/s 8 x 405.60 = 3244.78 MB/s 
total_size 67108864K rsz 1024 crg 8 thr 128 write 1952.31 MB/s 8 x 244.34 = 1954.73 MB/s read 3438.20 MB/s 8 x 430.24 = 3441.93 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 16 write 2012.85 MB/s 16 x 125.86 = 2013.70 MB/s read 3145.18 MB/s 16 x 196.70 = 3147.13 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 32 write 1994.44 MB/s 16 x 124.70 = 1995.24 MB/s read 3025.56 MB/s 16 x 189.17 = 3026.73 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 64 write 2004.65 MB/s 16 x 125.36 = 2005.77 MB/s read 3004.32 MB/s 16 x 187.85 = 3005.52 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 128 write 2007.39 MB/s 16 x 125.61 = 2009.74 MB/s read 3039.40 MB/s 16 x 190.05 = 3040.77 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 256 write 1997.99 MB/s 16 x 125.04 = 2000.58 MB/s read 3112.63 MB/s 16 x 194.82 = 3117.07 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 32 write 2013.20 MB/s 32 x 62.95 = 2014.47 MB/s read 4255.10 MB/s 32 x 133.25 = 4263.92 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 64 write 2045.06 MB/s 32 x 64.00 = 2048.03 MB/s read 3722.31 MB/s 32 x 116.44 = 3726.20 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 128 write 2047.41 MB/s 32 x 64.08 = 2050.48 MB/s read 3978.66 MB/s 32 x 124.47 = 3983.15 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 256 write 2041.97 MB/s 32 x 63.89 = 2044.37 MB/s read 4020.13 MB/s 32 x 125.73 = 4023.44 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 512 write 2047.79 MB/s 32 x 64.11 = 2051.39 MB/s read 3841.14 MB/s 32 x 120.11 = 3843.38 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 64 write 2037.39 MB/s 64 x 31.90 = 2041.63 MB/s read 5316.71 MB/s 64 x 83.33 = 5333.25 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 128 write 2051.62 MB/s 64 x 32.08 = 2053.22 MB/s read 5470.01 MB/s 64 x 85.74 = 5487.67 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 256 write 2057.33 MB/s 64 x 32.18 = 2059.33 MB/s read 5499.66 MB/s 64 x 86.01 = 5504.76 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 512 write 2054.44 MB/s 64 x 32.13 = 2056.27 MB/s read 5356.79 MB/s 64 x 83.97 = 5374.15 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 1024 write 2062.91 MB/s 64 x 32.26 = 2064.82 MB/s read 5516.21 MB/s 64 x 86.26 = 5520.63 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 128 write 2067.42 MB/s 128 x 16.17 = 2070.31 MB/s read 4584.08 MB/s 128 x 35.94 = 4600.83 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 256 write 2060.90 MB/s 128 x 16.11 = 2061.77 MB/s read 4556.67 MB/s 128 x 35.69 = 4567.87 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 512 write 2065.59 MB/s 128 x 16.16 = 2067.87 MB/s read 4671.71 MB/s 128 x 36.60 = 4685.06 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 1024 write 2067.70 MB/s 128 x 16.16 = 2069.09 MB/s read 4517.05 MB/s 128 x 35.47 = 4539.79 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 2048 write 2070.00 MB/s 128 x 16.21 = 2075.20 MB/s read 4738.98 MB/s 128 x 37.17 = 4758.30 MB/s 
total_size 67108864K rsz 1024 crg 256 thr 256 write 2073.01 MB/s 256 x 8.12 = 2077.64 MB/s read 4380.17 MB/s 256 x 17.19 = 4401.86 MB/s 
total_size 67108864K rsz 1024 crg 256 thr 512 write 2071.51 MB/s 256 x 8.10 = 2072.75 MB/s read 4316.17 MB/s 256 x 16.96 = 4340.82 MB/s 
total_size 67108864K rsz 1024 crg 256 thr 1024 write 2064.41 MB/s 256 x 8.09 = 2070.31 MB/s read 4915.45 MB/s 256 x 19.34 = 4951.17 MB/s 
total_size 67108864K rsz 1024 crg 256 thr 2048 write 2064.69 MB/s 256 x 8.08 = 2067.87 MB/s read 4284.50 MB/s 256 x 16.87 = 4318.85 MB/s 
total_size 67108864K rsz 1024 crg 512 thr 512 write 2027.22 MB/s 512 x 3.96 = 2026.37 MB/s read 5133.79 MB/s 512 x 10.04 = 5141.60 MB/s 
total_size 67108864K rsz 1024 crg 512 thr 1024 write 2033.61 MB/s 512 x 3.98 = 2036.13 MB/s read 5157.11 MB/s 512 x 10.09 = 5166.02 MB/s 
total_size 67108864K rsz 1024 crg 512 thr 2048 write 2049.99 MB/s 512 x 4.01 = 2050.78 MB/s read 5086.87 MB/s 512 x 10.05 = 5146.48 MB/s 
total_size 67108864K rsz 1024 crg 1024 thr 1024 write 1813.79 MB/s 1024 x 1.77 = 1816.41 MB/s read 4298.05 MB/s 1024 x 4.28 = 4384.77 MB/s 
total_size 67108864K rsz 1024 crg 1024 thr 2048 write 1718.45 MB/s 1024 x 1.68 = 1718.75 MB/s read 4311.54 MB/s 1024 x 4.29 = 4394.53 MB/s 
total_size 67108864K rsz 1024 crg 2048 thr 2048 write 1247.56 MB/s 2048 x 0.63 = 1289.06 MB/s read 3850.62 MB/s 2048 x 1.92 = 3925.78 MB/s

Eight 15-disk LUNs without cache mirroring (NOT RECOMMENDED)

Max write ~4.2GB/sec , Max read ~5.5GB/sec

Mon Oct 6 12:07:30 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 from fermi-gpfs01
total_size 67108864K rsz 1024 crg 8 thr 8 write 4117.34 MB/s 8 x 515.06 = 4120.48 MB/s read 2449.74 MB/s 8 x 306.57 = 2452.55 MB/s 
total_size 67108864K rsz 1024 crg 8 thr 16 write 4094.32 MB/s 8 x 512.19 = 4097.52 MB/s read 2568.02 MB/s 8 x 321.88 = 2575.07 MB/s 
total_size 67108864K rsz 1024 crg 8 thr 32 write 4112.33 MB/s 8 x 514.45 = 4115.60 MB/s read 2524.38 MB/s 8 x 315.84 = 2526.70 MB/s 
total_size 67108864K rsz 1024 crg 8 thr 64 write 4092.89 MB/s 8 x 512.29 = 4098.28 MB/s read 2590.98 MB/s 8 x 324.43 = 2595.44 MB/s 
total_size 67108864K rsz 1024 crg 8 thr 128 write 4170.54 MB/s 8 x 521.89 = 4175.11 MB/s read 2190.28 MB/s 8 x 274.08 = 2192.61 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 16 write 4211.79 MB/s 16 x 263.89 = 4222.26 MB/s read 2805.23 MB/s 16 x 175.39 = 2806.24 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 32 write 4285.58 MB/s 16 x 268.08 = 4289.25 MB/s read 2899.50 MB/s 16 x 181.46 = 2903.29 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 64 write 4303.16 MB/s 16 x 269.66 = 4314.58 MB/s read 2822.25 MB/s 16 x 176.50 = 2823.94 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 128 write 4199.59 MB/s 16 x 263.23 = 4211.73 MB/s read 2749.91 MB/s 16 x 172.09 = 2753.45 MB/s 
total_size 67108864K rsz 1024 crg 16 thr 256 write 4245.41 MB/s 16 x 266.08 = 4257.20 MB/s read 2714.66 MB/s 16 x 169.85 = 2717.59 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 32 write 4186.19 MB/s 32 x 130.94 = 4190.06 MB/s read 3971.82 MB/s 32 x 124.37 = 3979.80 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 64 write 4327.97 MB/s 32 x 135.37 = 4331.97 MB/s read 4067.70 MB/s 32 x 127.22 = 4071.04 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 128 write 4297.22 MB/s 32 x 134.41 = 4301.15 MB/s read 4096.38 MB/s 32 x 128.13 = 4100.04 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 256 write 4346.15 MB/s 32 x 136.18 = 4357.60 MB/s read 4124.96 MB/s 32 x 128.97 = 4127.20 MB/s 
total_size 67108864K rsz 1024 crg 32 thr 512 write 4323.25 MB/s 32 x 135.60 = 4339.29 MB/s read 3862.30 MB/s 32 x 120.93 = 3869.63 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 64 write 4251.35 MB/s 64 x 66.79 = 4274.29 MB/s read 5469.94 MB/s 64 x 85.75 = 5488.28 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 128 write 4181.70 MB/s 64 x 65.56 = 4195.56 MB/s read 5487.44 MB/s 64 x 86.04 = 5506.59 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 256 write 4184.92 MB/s 64 x 65.63 = 4200.44 MB/s read 5498.97 MB/s 64 x 86.29 = 5522.46 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 512 write 4196.49 MB/s 64 x 65.93 = 4219.36 MB/s read 5412.16 MB/s 64 x 84.85 = 5430.30 MB/s 
total_size 67108864K rsz 1024 crg 64 thr 1024 write 4203.88 MB/s 64 x 65.94 = 4219.97 MB/s read 5278.91 MB/s 64 x 82.67 = 5291.14 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 128 write 4168.32 MB/s 128 x 32.84 = 4202.88 MB/s read 4579.70 MB/s 128 x 35.92 = 4597.17 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 256 write 4080.65 MB/s 128 x 31.93 = 4086.91 MB/s read 4590.17 MB/s 128 x 36.00 = 4608.15 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 512 write 4090.79 MB/s 128 x 32.01 = 4096.68 MB/s read 4640.93 MB/s 128 x 36.39 = 4658.20 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 1024 write 4134.32 MB/s 128 x 32.35 = 4140.62 MB/s read 4540.94 MB/s 128 x 35.59 = 4555.66 MB/s 
total_size 67108864K rsz 1024 crg 128 thr 2048 write 4153.93 MB/s 128 x 32.49 = 4158.94 MB/s read 4625.92 MB/s 128 x 36.29 = 4644.78 MB/s 
total_size 67108864K rsz 1024 crg 256 thr 256 write 4251.26 MB/s 256 x 16.74 = 4284.67 MB/s read 4465.30 MB/s 256 x 17.53 = 4487.30 MB/s 
total_size 67108864K rsz 1024 crg 256 thr 512 write 4236.42 MB/s 256 x 16.58 = 4245.61 MB/s read 4460.79 MB/s 256 x 17.50 = 4479.98 MB/s 
total_size 67108864K rsz 1024 crg 256 thr 1024 write 4238.58 MB/s 256 x 16.69 = 4272.46 MB/s read 4538.00 MB/s 256 x 17.84 = 4567.87 MB/s 
total_size 67108864K rsz 1024 crg 256 thr 2048 write 4189.18 MB/s 256 x 16.66 = 4265.14 MB/s read 4399.98 MB/s 256 x 17.24 = 4414.06 MB/s 
total_size 67108864K rsz 1024 crg 512 thr 512 write 3876.79 MB/s 512 x 7.69 = 3935.55 MB/s read 5267.24 MB/s 512 x 10.41 = 5332.03 MB/s 
total_size 67108864K rsz 1024 crg 512 thr 1024 write 4084.05 MB/s 512 x 7.99 = 4091.80 MB/s read 5179.54 MB/s 512 x 10.25 = 5249.02 MB/s 
total_size 67108864K rsz 1024 crg 512 thr 2048 write 3980.91 MB/s 512 x 7.88 = 4033.20 MB/s read 4948.02 MB/s 512 x 9.91 = 5073.24 MB/s 
total_size 67108864K rsz 1024 crg 1024 thr 1024 write 3080.92 MB/s 1024 x 3.01 = 3085.94 MB/s read 4366.60 MB/s 1024 x 4.39 = 4492.19 MB/s 
total_size 67108864K rsz 1024 crg 1024 thr 2048 write 3152.38 MB/s 1024 x 3.19 = 3271.48 MB/s read 4446.49 MB/s 1024 x 4.41 = 4511.72 MB/s 
total_size 67108864K rsz 1024 crg 2048 thr 2048 write 2296.72 MB/s 2048 x 1.14 = 2343.75 MB/s read 3883.91 MB/s 2048 x 1.98 = 4062.50 MB/s

Sixteen 15-disk LUNs (host failover scenario)

Max write ~4GB/sec , Max read ~10GB/sec

Mon Oct 6 13:32:20 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 /dev/raw/raw9 /dev/raw/raw10 /dev/raw/raw11 /dev/raw/raw12 /dev/raw/raw13 /dev/raw/raw14 /dev/raw/raw15 /dev/raw/raw16 from fermi-gpfs01
total_size 134217728K rsz 1024 crg 16 thr 16 write 3855.02 MB/s 16 x 241.03 = 3856.51 MB/s read 5777.80 MB/s 16 x 361.66 = 5786.59 MB/s 
total_size 8388608K134217728K rsz 1024 crg 116 thr 232 write 4823928.9120 MB/s 116 x 483245.1561 = 4833929.1575 MB/s read 8826414.9331 MB/s 116 x 883401.7257 = 8836425.7217 MB/s 
total_size 8388608K134217728K rsz 1024 crg 116 thr 464 write 4693926.2749 MB/s 116 x 469245.4973 = 4693931.4973 MB/s read 8716484.9325 MB/s 116 x 872405.7187 = 8726493.7199 MB/s 
total_size 8388608K134217728K rsz 1024 crg 116 thr 8128 write 4763923.8297 MB/s 116 x 477245.0573 = 4773931.0573 MB/s read 8766868.4797 MB/s 116 x 877430.2202 = 8776880.2234 MB/s 
total_size 8388608K134217728K rsz 1024 crg 116 thr 16256 write 4733924.5045 MB/s 116 x 474245.0946 = 4743927.0931 MB/s read 8696246.4378 MB/s 116 x 870390.2381 = 8706252.2390 MB/s 
total_size 8388608K134217728K rsz 1024 crg 232 thr 232 write 8134027.0776 MB/s 232 x 406126.8715 = 8134036.7387 MB/s read 4895552.0395 MB/s 232 x 244173.6572 = 4895559.2908 MB/s 
total_size 8388608K134217728K rsz 1024 crg 232 thr 464 write 8554018.2052 MB/s 232 x 427125.9889 = 8554028.9663 MB/s read 4826020.7197 MB/s 232 x 241188.4827 = 4826024.9678 MB/s 
total_size 8388608K134217728K rsz 1024 crg 232 thr 8128 write 8484018.0245 MB/s 232 x 424125.3963 = 8484020.7708 MB/s read 4935561.0738 MB/s 232 x 246174.6602 = 4935568.3254 MB/s 
total_size 8388608K134217728K rsz 1024 crg 232 thr 16256 write 8354023.9655 MB/s 232 x 418125.3879 = 8364025.7527 MB/s read 4805737.1370 MB/s 232 x 240179.1841 = 4805741.3727 MB/s 
total_size 8388608K134217728K rsz 1024 crg 232 thr 32512 write 8304006.1841 MB/s 232 x 415125.4832 = 8304010.9631 MB/s read 4875912.1517 MB/s 232 x 243184.7083 = 4875914.4061 MB/s 
total_size 8388608K134217728K rsz 1024 crg 464 thr 464 write 10424017.6039 MB/s 464 x 26062.9380 = 10434019.7017 MB/s read 6697774.9834 MB/s 464 x 167121.6281 = 6707796.4702 MB/s 
total_size 8388608K134217728K rsz 1024 crg 464 thr 8128 write 9724040.1162 MB/s 464 x 24363.2728 = 9734049.0968 MB/s read 6687571.0559 MB/s 464 x 167118.1352 = 6687585.5345 MB/s 
total_size 8388608K134217728K rsz 1024 crg 464 thr 16256 write 10194025.0483 MB/s 464 x 25563.0407 = 10204036.1625 MB/s read 6407595.7823 MB/s 464 x 160118.3091 = 6417610.2147 MB/s 
total_size 8388608K134217728K rsz 1024 crg 464 thr 32512 write 10344033.4538 MB/s 464 x 25863.9214 = 10354041.6914 MB/s read 6567665.3084 MB/s 464 x 164120.1912 = 6567687.7899 MB/s 
total_size 8388608K134217728K rsz 1024 crg 464 thr 641024 write 10164056.8754 MB/s 464 x 25463.5247 = 10184061.0789 MB/s read 6447349.3684 MB/s 464 x 161115.2107 = 6447364.8450 MB/s 
total_size 8388608K134217728K rsz 1024 crg 8128 thr 8128 write 8234020.0023 MB/s 8128 x 10231.9749 = 8234030.7576 MB/s read 8669940.2438 MB/s 8128 x 10878.3900 = 8679984.0813 MB/s 
total_size 8388608K134217728K rsz 1024 crg 8128 thr 16256 write 7884068.5199 MB/s 8128 x 9831.6686 = 7894078.2637 MB/s read 8629738.3883 MB/s 8128 x 10776.9017 = 8639749.1976 MB/s 
total_size 8388608K134217728K rsz 1024 crg 8128 thr 32512 write 7814090.6343 MB/s 8128 x 9732.7905 = 7824102.3278 MB/s read 86310002.4637 MB/s 8128 x 10878.0351 = 86410048.2683 MB/s 
total_size 8388608K134217728K rsz 1024 crg 8128 thr 641024 write 7904067.4412 MB/s 8128 x 9831.9083 = 7914074.1771 MB/s read 8689940.1219 MB/s 8128 x 10878.6300 = 8699984.0613 MB/s 
total_size 8388608K134217728K rsz 1024 crg 8128 thr 1282048 write 7874056.1036 MB/s 8128 x 9831.4983 = 7874074.8971 MB/s read 84710046.3120 MB/s 8128 x 10679.0211 = 84810125.1673 MB/s 
total_size 8388608K134217728K rsz 1024 crg 16256 thr 16256 write 6094106.9979 MB/s 16256 x 3816.1610 = 6104121.5009 MB/s read 10289031.2465 MB/s 16256 x 6435.3348 = 10299082.3603 MB/s 
total_size 8388608K134217728K rsz 1024 crg 16256 thr 32512 write 5854123.4167 MB/s 16256 x 3616.6117 = 5854140.7862 MB/s read 10188832.2380 MB/s 16256 x 6334.7270 = 10198884.5928 MB/s 
total_size 8388608K134217728K rsz 1024 crg 16256 thr 641024 write 5894056.8303 MB/s 16256 x 3615.8986 = 5904060.2106 MB/s read 10098789.7941 MB/s 16256 x 6334.1855 = 10108845.8921 MB/s 
total_size 8388608K134217728K rsz 1024 crg 16256 thr 1282048 write 5854083.7165 MB/s 16256 x 3615.6496 = 5864086.2491 MB/s read 10248863.8550 MB/s 16256 x 6434.1385 = 10268920.0090 MB/s 
total_size 8388608K134217728K rsz 1024 crg 16256 thr 2564096 write 5964066.4279 MB/s 16256 x 3715.3190 = 5964069.9282 MB/s read 10088812.5939 MB/s 16256 x 6334.1165 = 10098869.8363 MB/s 
total_size 8388608K134217728K rsz 1024 crg 32512 thr 32512 write 6674014.1648 MB/s 32512 x 207.88 = 6684033.0320 MB/s read 9468436.8908 MB/s 32512 x 2916.6260 = 9478500.8898 MB/s 
total_size 8388608K134217728K rsz 1024 crg 32512 thr 641024 write 6954074.5740 MB/s 32512 x 217.7699 = 6964091.4180 MB/s read 9608229.6117 MB/s 32512 x 3016.0622 = 9618305.9166 MB/s 
total_size 8388608K134217728K rsz 1024 crg 32512 thr 1282048 write 6944071.3195 MB/s 32512 x 217.7298 = 6954086.1991 MB/s read 9568453.8805 MB/s 32512 x 2916.9467 = 9578535.9516 MB/s 
total_size 8388608K134217728K rsz 1024 crg 32512 thr 2564096 write 6824094.6775 MB/s 32512 x 218.3609 = 6834140.5962 MB/s read 9518294.7224 MB/s 32512 x 2916.7937 = 9538378.3791 MB/s 
total_size 8388608K134217728K rsz 1024 crg 641024 thr 641024 write 6594085.2938 MB/s 641024 x 104.3202 = 6604121.4009 MB/s read 7579709.7957 MB/s 641024 x 119.8579 = 75810029.6730 MB/s 
total_size 8388608K134217728K rsz 1024 crg 641024 thr 1282048 write 6564019.7482 MB/s 641024 x 104.2801 = 6574111.9633 MB/s read 7579815.1334 MB/s 641024 x 119.8487 = 75810107.0642 MB/s 
total_size 8388608K134217728K rsz 1024 crg 641024 thr 2564096 write 6443844.7914 MB/s 641024 x 103.1085 = 6463945.3631 MB/s read 7569757.0380 MB/s 641024 x 119.8478 = 75710019.4553 MB/s 
total_size 8388608K134217728K rsz 1024 crg 1282048 thr 1282048 write 6023412.8825 MB/s 1282048 x 41.73 = 6053535.4716 MB/s read 6387988.5500 MB/s 1282048 x 54.0117 = 6408535.8716 MB/s 
total_size 8388608K134217728K rsz 1024 crg 1282048 thr 2564096 write 5883212.5593 MB/s 1282048 x 41.6167 = 5893417.6097 MB/s read 6237873.1892 MB/s 1282048 x 4.8810 = 6258398.0044 MB/s 
total_size 8388608K134217728K rsz 1024 crg 2564096 thr 2564096 write 4702423.9829 MB/s 2564096 x 10.8463 = 4712578.1913 MB/s read 5646793.4041 MB/s 2564096 x 21.2291 = 5687812.8550 MB/s

...