The Dell MD3460 controller array can support up to 120 drives with an MD3060e expansion tray attached. Single drive capacities have been steadily increasing, but transfer speeds have remained fairly constant. As a result, RAID6 rebuild times can exceed 24hrs when using 4TB drives. This is the main reason for choosing Dynamic Disk Pools over RAID6 - (DDP) provide lower rebuild times are possible since the RAID parity data is distributed compared to RAID6 because the hotspare space is spread across all drives in the pool. For the latest Fermi storage configuration, we are benchmarking eight 15-disk DDP volumes on each 120-drive MD3460+MD3060e storage config. Both arrays are dual-attached to a failover pair of servers (fermi-gpfs01,2). In a failover scenario, one machine could serve all sixteen DDP volumes.
The sgpdd-survey benchmark simulates performs parallel I/O O using raw unformatted LUNs. The tests spawn and an increasing number of threads (thr) that write and read to an increasing number of concurrent regions (crg). As the tests progress, I/O becomes more random. This set of tests transfer data in The transfer operations use 1MB records/chunks (rsz).
One 15-disk LUNs
. This is a local test that does not require any networking or formatted filesystem. The purpose is to understand the performance limits of the storage hardware before we create filesystems (GPFS, Lustre, etc.). The benchmark results show the MD3460 is capable of sustaining throughput well above 2GB/sec. In order to avoid networking bottlenecks, we may use 4x10Gb ethernet link aggregation instead of the usual 2x10Gb. We also have a limited number of 40Gb ports on the SLAC network.
One 15-disk LUN
Max write ~1GB/sec , Max read Max write ~1GB/sec , Max read ~1GB/sec
Mon Oct 6 10:26:06 PDT 2014 sgpdd-survey on /dev/raw/raw1 from fermi-gpfs01
total_size 8388608K rsz 1024 crg 1 thr 1 write 438.64 MB/s 1 x 438.94 = 438.94 MB/s read 893.60 MB/s 1 x 894.41 = 894.41 MB/s
total_size 8388608K rsz 1024 crg 1 thr 2 write 482.91 MB/s 1 x 483.15 = 483.15 MB/s read 882.93 MB/s 1 x 883.72 = 883.72 MB/s
total_size 8388608K rsz 1024 crg 1 thr 4 write 469.27 MB/s 1 x 469.49 = 469.49 MB/s read 871.93 MB/s 1 x 872.71 = 872.71 MB/s
total_size 8388608K rsz 1024 crg 1 thr 8 write 476.82 MB/s 1 x 477.05 = 477.05 MB/s read 876.47 MB/s 1 x 877.22 = 877.22 MB/s
total_size 8388608K rsz 1024 crg 1 thr 16 write 473.50 MB/s 1 x 474.09 = 474.09 MB/s read 869.43 MB/s 1 x 870.23 = 870.23 MB/s
total_size 8388608K rsz 1024 crg 2 thr 2 write 813.07 MB/s 2 x 406.87 = 813.73 MB/s read 489.03 MB/s 2 x 244.65 = 489.29 MB/s
total_size 8388608K rsz 1024 crg 2 thr 4 write 855.20 MB/s 2 x 427.98 = 855.96 MB/s read 482.71 MB/s 2 x 241.48 = 482.96 MB/s
total_size 8388608K rsz 1024 crg 2 thr 8 write 848.02 MB/s 2 x 424.39 = 848.77 MB/s read 493.07 MB/s 2 x 246.66 = 493.32 MB/s
total_size 8388608K rsz 1024 crg 2 thr 16 write 835.96 MB/s 2 x 418.38 = 836.75 MB/s read 480.13 MB/s 2 x 240.18 = 480.37 MB/s
total_size 8388608K rsz 1024 crg 2 thr 32 write 830.18 MB/s 2 x 415.48 = 830.96 MB/s read 487.15 MB/s 2 x 243.70 = 487.40 MB/s
total_size 8388608K rsz 1024 crg 4 thr 4 write 1042.60 MB/s 4 x 260.93 = 1043.70 MB/s read 669.98 MB/s 4 x 167.62 = 670.47 MB/s
total_size 8388608K rsz 1024 crg 4 thr 8 write 972.11 MB/s 4 x 243.27 = 973.09 MB/s read 668.05 MB/s 4 x 167.13 = 668.53 MB/s
total_size 8388608K rsz 1024 crg 4 thr 16 write 1019.04 MB/s 4 x 255.04 = 1020.16 MB/s read 640.78 MB/s 4 x 160.30 = 641.21 MB/s
total_size 8388608K rsz 1024 crg 4 thr 32 write 1034.45 MB/s 4 x 258.92 = 1035.69 MB/s read 656.30 MB/s 4 x 164.19 = 656.78 MB/s
total_size 8388608K rsz 1024 crg 4 thr 64 write 1016.87 MB/s 4 x 254.52 = 1018.07 MB/s read 644.36 MB/s 4 x 161.21 = 644.84 MB/s
total_size 8388608K rsz 1024 crg 8 thr 8 write 823.00 MB/s 8 x 102.97 = 823.75 MB/s read 866.24 MB/s 8 x 108.39 = 867.08 MB/s
total_size 8388608K rsz 1024 crg 8 thr 16 write 788.51 MB/s 8 x 98.66 = 789.26 MB/s read 862.38 MB/s 8 x 107.90 = 863.19 MB/s
total_size 8388608K rsz 1024 crg 8 thr 32 write 781.63 MB/s 8 x 97.79 = 782.32 MB/s read 863.46 MB/s 8 x 108.03 = 864.26 MB/s
total_size 8388608K rsz 1024 crg 8 thr 64 write 790.44 MB/s 8 x 98.90 = 791.17 MB/s read 868.12 MB/s 8 x 108.63 = 869.06 MB/s
total_size 8388608K rsz 1024 crg 8 thr 128 write 787.10 MB/s 8 x 98.49 = 787.89 MB/s read 847.31 MB/s 8 x 106.02 = 848.16 MB/s
total_size 8388608K rsz 1024 crg 16 thr 16 write 609.99 MB/s 16 x 38.16 = 610.50 MB/s read 1028.24 MB/s 16 x 64.33 = 1029.36 MB/s
total_size 8388608K rsz 1024 crg 16 thr 32 write 585.41 MB/s 16 x 36.61 = 585.78 MB/s read 1018.23 MB/s 16 x 63.72 = 1019.59 MB/s
total_size 8388608K rsz 1024 crg 16 thr 64 write 589.83 MB/s 16 x 36.89 = 590.21 MB/s read 1009.79 MB/s 16 x 63.18 = 1010.89 MB/s
total_size 8388608K rsz 1024 crg 16 thr 128 write 585.71 MB/s 16 x 36.64 = 586.24 MB/s read 1024.85 MB/s 16 x 64.13 = 1026.00 MB/s
total_size 8388608K rsz 1024 crg 16 thr 256 write 596.42 MB/s 16 x 37.31 = 596.92 MB/s read 1008.59 MB/s 16 x 63.11 = 1009.83 MB/s
total_size 8388608K rsz 1024 crg 32 thr 32 write 667.16 MB/s 32 x 20.88 = 668.03 MB/s read 946.89 MB/s 32 x 29.62 = 947.88 MB/s
total_size 8388608K rsz 1024 crg 32 thr 64 write 695.57 MB/s 32 x 21.76 = 696.41 MB/s read 960.61 MB/s 32 x 30.06 = 961.91 MB/s
total_size 8388608K rsz 1024 crg 32 thr 128 write 694.31 MB/s 32 x 21.72 = 695.19 MB/s read 956.88 MB/s 32 x 29.94 = 957.95 MB/s
total_size 8388608K rsz 1024 crg 32 thr 256 write 682.67 MB/s 32 x 21.36 = 683.59 MB/s read 951.72 MB/s 32 x 29.79 = 953.37 MB/s
total_size 8388608K rsz 1024 crg 64 thr 64 write 659.29 MB/s 64 x 10.32 = 660.40 MB/s read 757.79 MB/s 64 x 11.85 = 758.67 MB/s
total_size 8388608K rsz 1024 crg 64 thr 128 write 656.74 MB/s 64 x 10.28 = 657.96 MB/s read 757.13 MB/s 64 x 11.84 = 758.06 MB/s
total_size 8388608K rsz 1024 crg 64 thr 256 write 644.79 MB/s 64 x 10.10 = 646.36 MB/s read 756.03 MB/s 64 x 11.84 = 757.45 MB/s
total_size 8388608K rsz 1024 crg 128 thr 128 write 602.88 MB/s 128 x 4.73 = 605.47 MB/s read 638.55 MB/s 128 x 5.01 = 640.87 MB/s
total_size 8388608K rsz 1024 crg 128 thr 256 write 588.55 MB/s 128 x 4.61 = 589.60 MB/s read 623.18 MB/s 128 x 4.88 = 625.00 MB/s
total_size 8388608K rsz 1024 crg 256 thr 256 write 470.98 MB/s 256 x 1.84 = 471.19 MB/s read 564.40 MB/s 256 x 2.22 = 568.85 MB/s
Four 15-disk LUNs
Max write ~2GB/sec , Max read ~3.7GB8GB/sec
total_sizeMon 33554432KOct rsz 1024 crg 4 thr 4 write 1328.92 MB/s 4 x 332.39 = 1329.57 MB/s read 2911.00 MB/s 4 x 728.76 = 2915.046 11:06:36 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 from fermi-gpfs01
total_size 33554432K rsz 1024 crg 4 thr 4 write 1328.92 MB/s 4 x 332.39 = 1329.57 MB/s read 2911.00 MB/s 4 x 728.76 = 2915.04 MB/s
total_size 33554432K rsz 1024 crg 4 thr 8 write 1454.79 MB/s 4 x 364.14 = 1456.57 MB/s read 2895.55 MB/s 4 x 724.64 = 2898.56 MB/s
total_size 33554432K rsz 1024 crg 4 thr 16 write 1456.28 MB/s 4 x 364.27 = 1457.10 MB/s read 2458.23 MB/s 4 x 615.12 = 2460.48 MB/s
total_size 33554432K rsz 1024 crg 4 thr 32 write 1445.62 MB/s 4 x 361.72 = 1446.88 MB/s read 2994.64 MB/s 4 x 749.94 = 2999.76 MB/s
total_size 33554432K rsz 1024 crg 4 thr 64 write 1434.84 MB/s 4 x 358.97 = 1435.89 MB/s read 2740.28 MB/s 4 x 685.58 = 2742.31 MB/s
total_size 33554432K rsz 1024 crg 8 thr 8 write 2001.29 MB/s 8 x 250.36 = 2002.87 MB/s read 1639.01 MB/s 8 x 204.96 = 1639.71 MB/s
total_size 33554432K rsz 1024 crg 8 thr 16 write 1980.60 MB/s 8 x 247.88 = 1983.03 MB/s read 1440.08 MB/s 8 x 180.07 = 1440.58 MB/s
total_size 33554432K rsz 1024 crg 8 thr 32 write 1947.07 MB/s 8 x 243.69 = 1949.54 MB/s read 1661.00 MB/s 8 x 207.72 = 1661.76 MB/s
total_size 33554432K rsz 1024 crg 8 thr 64 write 2016.98 MB/s 8 x 252.47 = 2019.73 MB/s read 1715.79 MB/s 8 x 214.57 = 1716.54 MB/s
total_size 33554432K rsz 1024 crg 8 thr 128 write 1961.61 MB/s 8 x 245.53 = 1964.26 MB/s read 1682.21 MB/s 8 x 210.47 = 1683.73 MB/s
total_size 33554432K rsz 1024 crg 16 thr 16 write 2097.46 MB/s 16 x 131.28 = 2100.52 MB/s read 2379.03 MB/s 16 x 148.78 = 2380.52 MB/s
total_size 33554432K rsz 1024 crg 16 thr 32 write 2078.31 MB/s 16 x 130.00 = 2080.08 MB/s read 2389.99 MB/s 16 x 149.53 = 2392.43 MB/s
total_size 33554432K rsz 1024 crg 16 thr 64 write 2075.24 MB/s 16 x 129.81 = 2077.03 MB/s read 2193.77 MB/s 16 x 137.20 = 2195.13 MB/s
total_size 33554432K rsz 1024 crg 16 thr 128 write 2088.21 MB/s 16 x 130.62 = 2090.00 MB/s read 2206.04 MB/s 16 x 137.96 = 2207.34 MB/s
total_size 33554432K rsz 1024 crg 16 thr 256 write 2085.91 MB/s 16 x 130.49 = 2087.86 MB/s read 2393.09 MB/s 16 x 149.68 = 2394.87 MB/s
total_size 33554432K rsz 1024 crg 32 thr 32 write 2008.13 MB/s 32 x 62.81 = 2009.89 MB/s read 3197.92 MB/s 32 x 100.09 = 3202.82 MB/s
total_size 33554432K rsz 1024 crg 32 thr 64 write 1979.69 MB/s 32 x 61.92 = 1981.51 MB/s read 3190.03 MB/s 32 x 99.79 = 3193.36 MB/s
total_size 33554432K rsz 1024 crg 32 thr 128 write 1997.90 MB/s 32 x 62.49 = 1999.82 MB/s read 3237.11 MB/s 32 x 101.26 = 3240.36 MB/s
total_size 33554432K rsz 1024 crg 32 thr 256 write 2005.33 MB/s 32 x 62.71 = 2006.84 MB/s read 3213.86 MB/s 32 x 100.64 = 3220.52 MB/s
total_size 33554432K rsz 1024 crg 32 thr 512 write 1981.90 MB/s 32 x 62.00 = 1983.95 MB/s read 3277.99 MB/s 32 x 102.53 = 3280.94 MB/s
total_size 33554432K rsz 1024 crg 64 thr 64 write 1986.19 MB/s 64 x 31.06 = 1987.92 MB/s read 3835.45 MB/s 64 x 60.01 = 3840.33 MB/s
total_size 33554432K rsz 1024 crg 64 thr 128 write 1985.35 MB/s 64 x 31.06 = 1987.92 MB/s read 3804.40 MB/s 64 x 59.53 = 3809.81 MB/s
total_size 33554432K rsz 1024 crg 64 thr 256 write 2009.90 MB/s 64 x 31.47 = 2014.16 MB/s read 3827.19 MB/s 64 x 59.99 = 3839.11 MB/s
total_size 33554432K rsz 1024 crg 64 thr 512 write 1986.98 MB/s 64 x 31.09 = 1989.75 MB/s read 3876.90 MB/s 64 x 60.76 = 3888.55 MB/s
total_size 33554432K rsz 1024 crg 64 thr 1024 write 1978.71 MB/s 64 x 30.96 = 1981.20 MB/s read 3807.05 MB/s 64 x 59.57 = 3812.26 MB/s
total_size 33554432K rsz 1024 crg 128 thr 128 write 1999.93 MB/s 128 x 15.65 = 2003.17 MB/s read 3608.74 MB/s 128 x 28.24 = 3614.50 MB/s
total_size 33554432K rsz 1024 crg 128 thr 256 write 2016.86 MB/s 128 x 15.80 = 2022.71 MB/s read 3600.97 MB/s 128 x 28.17 = 3605.96 MB/s
total_size 33554432K rsz 1024 crg 128 thr 512 write 2012.85 MB/s 128 x 15.75 = 2016.60 MB/s read 3541.17 MB/s 128 x 27.71 = 3547.36 MB/s
total_size 33554432K rsz 1024 crg 128 thr 1024 write 2012.09 MB/s 128 x 15.75 = 2016.60 MB/s read 3506.41 MB/s 128 x 27.54 = 3525.39 MB/s
total_size 33554432K rsz 1024 crg 256 thr 256 write 1868.84 MB/s 256 x 7.61 = 1948.24 MB/s read 2809.95 MB/s 256 x 11.05 = 2829.59 MB/s
total_size 33554432K rsz 1024 crg 256 thr 512 write 1944.84 MB/s 256 x 7.65 = 1958.01 MB/s read 2729.52 MB/s 256 x 10.76 = 2753.91 MB/s
total_size 33554432K rsz 1024 crg 256 thr 1024 write 1960.35 MB/s 256 x 7.69 = 1967.77 MB/s read 2814.37 MB/s 256 x 11.07 = 2834.47 MB/s
total_size 33554432K rsz 1024 crg 512 thr 512 write 1671.90 MB/s 512 x 3.29 = 1684.57 MB/s read 2378.45 MB/s 512 x 4.67 = 2392.58 MB/s
total_size 33554432K rsz 1024 crg 512 thr 1024 write 1637.35 MB/s 512 x 3.22 = 1650.39 MB/s read 2373.27 MB/s 512 x 4.68 = 2397.46 MB/s
total_size 33554432K rsz 1024 crg 1024 thr 1024 write 1231.95 MB/s 1024 x 1.20 = 1230.47 MB/s read 2094.72 MB/s 1024 x 2.06 = 2109.38 MB/s
Eight 15-disk LUNs
Max write ~2GB/sec , Max read ~5.5GB/sec
Mon Oct 6 11:27:15 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 from fermi-gpfs01total_size 67108864K rsz 1024 crg 8 thr 8 write 1922.79 MB/s 8 x 240.58 = 1924.67 MB/s read 3002.11 MB/s 8 x 375.48 = 3003.85 MB/s
total_size 67108864K rsz 1024 crg 8 thr 16 write 1954.89 MB/s 8 x 244.51 = 1956.10 MB/s read 3067.43 MB/s 8 x 383.64 = 3069.15 MB/s
total_size 67108864K rsz 1024 crg 8 thr 32 write 1945.17 MB/s 8 x 243.23 = 1945.88 MB/s read 3554.62 MB/s 8 x 444.55 = 3556.37 MB/s
total_size 67108864K rsz 1024 crg 8 thr 64 write 1963.69 MB/s 8 x 245.61 = 1964.87 MB/s read 3241.54 MB/s 8 x 405.60 = 3244.78 MB/s
total_size 67108864K rsz 1024 crg 8 thr 128 write 1952.31 MB/s 8 x 244.34 = 1954.73 MB/s read 3438.20 MB/s 8 x 430.24 = 3441.93 MB/s
total_size 67108864K rsz 1024 crg 16 thr 16 write 2012.85 MB/s 16 x 125.86 = 2013.70 MB/s read 3145.18 MB/s 16 x 196.70 = 3147.13 MB/s
total_size 67108864K rsz 1024 crg 16 thr 32 write 1994.44 MB/s 16 x 124.70 = 1995.24 MB/s read 3025.56 MB/s 16 x 189.17 = 3026.73 MB/s
total_size 67108864K rsz 1024 crg 16 thr 64 write 2004.65 MB/s 16 x 125.36 = 2005.77 MB/s read 3004.32 MB/s 16 x 187.85 = 3005.52 MB/s
total_size 67108864K rsz 1024 crg 16 thr 128 write 2007.39 MB/s 16 x 125.61 = 2009.74 MB/s read 3039.40 MB/s 16 x 190.05 = 3040.77 MB/s
total_size 67108864K rsz 1024 crg 16 thr 256 write 1997.99 MB/s 16 x 125.04 = 2000.58 MB/s read 3112.63 MB/s 16 x 194.82 = 3117.07 MB/s
total_size 67108864K rsz 1024 crg 32 thr 32 write 2013.20 MB/s 32 x 62.95 = 2014.47 MB/s read 4255.10 MB/s 32 x 133.25 = 4263.92 MB/s
total_size 67108864K rsz 1024 crg 32 thr 64 write 2045.06 MB/s 32 x 64.00 = 2048.03 MB/s read 3722.31 MB/s 32 x 116.44 = 3726.20 MB/s
total_size 67108864K rsz 1024 crg 32 thr 128 write 2047.41 MB/s 32 x 64.08 = 2050.48 MB/s read 3978.66 MB/s 32 x 124.47 = 3983.15 MB/s
total_size 67108864K rsz 1024 crg 32 thr 256 write 2041.97 MB/s 32 x 63.89 = 2044.37 MB/s read 4020.13 MB/s 32 x 125.73 = 4023.44 MB/s
total_size 67108864K rsz 1024 crg 32 thr 512 write 2047.79 MB/s 32 x 64.11 = 2051.39 MB/s read 3841.14 MB/s 32 x 120.11 = 3843.38 MB/s
total_size 67108864K rsz 1024 crg 64 thr 64 write 2037.39 MB/s 64 x 31.90 = 2041.63 MB/s read 5316.71 MB/s 64 x 83.33 = 5333.25 MB/s
total_size 67108864K rsz 1024 crg 64 thr 128 write 2051.62 MB/s 64 x 32.08 = 2053.22 MB/s read 5470.01 MB/s 64 x 85.74 = 5487.67 MB/s
total_size 67108864K rsz 1024 crg 64 thr 256 write 2057.33 MB/s 64 x 32.18 = 2059.33 MB/s read 5499.66 MB/s 64 x 86.01 = 5504.76 MB/s
total_size 67108864K rsz 1024 crg 64 thr 512 write 2054.44 MB/s 64 x 32.13 = 2056.27 MB/s read 5356.79 MB/s 64 x 83.97 = 5374.15 MB/s
total_size 67108864K rsz 1024 crg 64 thr 1024 write 2062.91 MB/s 64 x 32.26 = 2064.82 MB/s read 5516.21 MB/s 64 x 86.26 = 5520.63 MB/s
total_size 67108864K rsz 1024 crg 128 thr 128 write 2067.42 MB/s 128 x 16.17 = 2070.31 MB/s read 4584.08 MB/s 128 x 35.94 = 4600.83 MB/s
total_size 67108864K rsz 1024 crg 128 thr 256 write 2060.90 MB/s 128 x 16.11 = 2061.77 MB/s read 4556.67 MB/s 128 x 35.69 = 4567.87 MB/s
total_size 67108864K rsz 1024 crg 128 thr 512 write 2065.59 MB/s 128 x 16.16 = 2067.87 MB/s read 4671.71 MB/s 128 x 36.60 = 4685.06 MB/s
total_size 67108864K rsz 1024 crg 128 thr 1024 write 2067.70 MB/s 128 x 16.16 = 2069.09 MB/s read 4517.05 MB/s 128 x 35.47 = 4539.79 MB/s
total_size 67108864K rsz 1024 crg 128 thr 2048 write 2070.00 MB/s 128 x 16.21 = 2075.20 MB/s read 4738.98 MB/s 128 x 37.17 = 4758.30 MB/s
total_size 67108864K rsz 1024 crg 256 thr 256 write 2073.01 MB/s 256 x 8.12 = 2077.64 MB/s read 4380.17 MB/s 256 x 17.19 = 4401.86 MB/s
total_size 67108864K rsz 1024 crg 256 thr 512 write 2071.51 MB/s 256 x 8.10 = 2072.75 MB/s read 4316.17 MB/s 256 x 16.96 = 4340.82 MB/s
total_size 67108864K rsz 1024 crg 256 thr 1024 write 2064.41 MB/s 256 x 8.09 = 2070.31 MB/s read 4915.45 MB/s 256 x 19.34 = 4951.17 MB/s
total_size 67108864K rsz 1024 crg 256 thr 2048 write 2064.69 MB/s 256 x 8.08 = 2067.87 MB/s read 4284.50 MB/s 256 x 16.87 = 4318.85 MB/s
total_size 67108864K rsz 1024 crg 512 thr 512 write 2027.22 MB/s 512 x 3.96 = 2026.37 MB/s read 5133.79 MB/s 512 x 10.04 = 5141.60 MB/s
total_size 67108864K rsz 1024 crg 512 thr 1024 write 2033.61 MB/s 512 x 3.98 = 2036.13 MB/s read 5157.11 MB/s 512 x 10.09 = 5166.02 MB/s
total_size 67108864K rsz 1024 crg 512 thr 2048 write 2049.99 MB/s 512 x 4.01 = 2050.78 MB/s read 5086.87 MB/s 512 x 10.05 = 5146.48 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 1024 write 1813.79 MB/s 1024 x 1.77 = 1816.41 MB/s read 4298.05 MB/s 1024 x 4.28 = 4384.77 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 2048 write 1718.45 MB/s 1024 x 1.68 = 1718.75 MB/s read 4311.54 MB/s 1024 x 4.29 = 4394.53 MB/s
total_size 67108864K rsz 1024 crg 2048 thr 2048 write 1247.56 MB/s 2048 x 0.63 = 1289.06 MB/s read 3850.62 MB/s 2048 x 1.92 = 3925.78 MB/s
Eight 15-disk LUNs without cache mirroring (NOT RECOMMENDED)
Max write ~4.2GB/sec , Max read ~5.5GB/sec
Mon Oct 6 12:07:30 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 from fermi-gpfs01
total_size 67108864K rsz 1024 crg 8 thr 8 write 4117.34 MB/s 8 x 515.06 = 4120.48 MB/s read 2449.74 MB/s 8 x 306.57 = 2452.55 MB/s
total_size 67108864K rsz 1024 crg 8 thr 16 write 4094.32 MB/s 8 x 512.19 = 4097.52 MB/s read 2568.02 MB/s 8 x 321.88 = 2575.07 MB/s
total_size 67108864K rsz 1024 crg 8 thr 32 write 4112.33 MB/s 8 x 514.45 = 4115.60 MB/s read 2524.38 MB/s 8 x 315.84 = 2526.70 MB/s
total_size 67108864K rsz 1024 crg 8 thr 64 write 4092.89 MB/s 8 x 512.29 = 4098.28 MB/s read 2590.98 MB/s 8 x 324.43 = 2595.44 MB/s
total_size 67108864K rsz 1024 crg 8 thr 128 write 4170.54 MB/s 8 x 521.89 = 4175.11 MB/s read 2190.28 MB/s 8 x 274.08 = 2192.61 MB/s
total_size 67108864K rsz 1024 crg 16 thr 16 write 4211.79 MB/s 16 x 263.89 = 4222.26 MB/s read 2805.23 MB/s 16 x 175.39 = 2806.24 MB/s
total_size 67108864K rsz 1024 crg 16 thr 32 write 4285.58 MB/s 16 x 268.08 = 4289.25 MB/s read 2899.50 MB/s 16 x 181.46 = 2903.29 MB/s
total_size 67108864K rsz 1024 crg 16 thr 64 write 4303.16 MB/s 16 x 269.66 = 4314.58 MB/s read 2822.25 MB/s 16 x 176.50 = 2823.94 MB/s
total_size 67108864K rsz 1024 crg 16 thr 128 write 4199.59 MB/s 16 x 263.23 = 4211.73 MB/s read 2749.91 MB/s 16 x 172.09 = 2753.45 MB/s
total_size 67108864K rsz 1024 crg 16 thr 256 write 4245.41 MB/s 16 x 266.08 = 4257.20 MB/s read 2714.66 MB/s 16 x 169.85 = 2717.59 MB/s
total_size 67108864K rsz 1024 crg 32 thr 32 write 4186.19 MB/s 32 x 130.94 = 4190.06 MB/s read 3971.82 MB/s 32 x 124.37 = 3979.80 MB/s
total_size 67108864K rsz 1024 crg 32 thr 64 write 4327.97 MB/s 32 x 135.37 = 4331.97 MB/s read 4067.70 MB/s 32 x 127.22 = 4071.04 MB/s
total_size 67108864K rsz 1024 crg 32 thr 128 write 4297.22 MB/s 32 x 134.41 = 4301.15 MB/s read 4096.38 MB/s 32 x 128.13 = 4100.04 MB/s
total_size 67108864K rsz 1024 crg 32 thr 256 write 4346.15 MB/s 32 x 136.18 = 4357.60 MB/s read 4124.96 MB/s 32 x 128.97 = 4127.20 MB/s
total_size 67108864K rsz 1024 crg 32 thr 512 write 4323.25 MB/s 32 x 135.60 = 4339.29 MB/s read 3862.30 MB/s 32 x 120.93 = 3869.63 MB/s
total_size 67108864K rsz 1024 crg 64 thr 64 write 4251.35 MB/s 64 x 66.79 = 4274.29 MB/s read 5469.94 MB/s 64 x 85.75 = 5488.28 MB/s
total_size 67108864K rsz 1024 crg 64 thr 128 write 4181.70 MB/s 64 x 65.56 = 4195.56 MB/s read 5487.44 MB/s 64 x 86.04 = 5506.59 MB/s
total_size 67108864K rsz 1024 crg 64 thr 256 write 4184.92 MB/s 64 x 65.63 = 4200.44 MB/s read 5498.97 MB/s 64 x 86.29 = 5522.46 MB/s
total_size 67108864K rsz 1024 crg 64 thr 512 write 4196.49 MB/s 64 x 65.93 = 4219.36 MB/s read 5412.16 MB/s 64 x 84.85 = 5430.30 MB/s
total_size 67108864K rsz 1024 crg 64 thr 1024 write 4203.88 MB/s 64 x 65.94 = 4219.97 MB/s read 5278.91 MB/s 64 x 82.67 = 5291.14 MB/s
total_size 67108864K rsz 1024 crg 128 thr 128 write 4168.32 MB/s 128 x 32.84 = 4202.88 MB/s read 4579.70 MB/s 128 x 35.92 = 4597.17 MB/s
total_size 67108864K rsz 1024 crg 128 thr 256 write 4080.65 MB/s 128 x 31.93 = 4086.91 MB/s read 4590.17 MB/s 128 x 36.00 = 4608.15 MB/s
total_size 67108864K rsz 1024 crg 128 thr 512 write 4090.79 MB/s 128 x 32.01 = 4096.68 MB/s read 4640.93 MB/s 128 x 36.39 = 4658.20 MB/s
total_size 67108864K rsz 1024 crg 128 thr 1024 write 4134.32 MB/s 128 x 32.35 = 4140.62 MB/s read 4540.94 MB/s 128 x 35.59 = 4555.66 MB/s
total_size 67108864K rsz 1024 crg 128 thr 2048 write 4153.93 MB/s 128 x 32.49 = 4158.94 MB/s read 4625.92 MB/s 128 x 36.29 = 4644.78 MB/s
total_size 67108864K rsz 1024 crg 256 thr 256 write 4251.26 MB/s 256 x 16.74 = 4284.67 MB/s read 4465.30 MB/s 256 x 17.53 = 4487.30 MB/s
total_size 67108864K rsz 1024 crg 256 thr 512 write 4236.42 MB/s 256 x 16.58 = 4245.61 MB/s read 4460.79 MB/s 256 x 17.50 = 4479.98 MB/s
total_size 67108864K rsz 1024 crg 256 thr 1024 write 4238.58 MB/s 256 x 16.69 = 4272.46 MB/s read 4538.00 MB/s 256 x 17.84 = 4567.87 MB/s
total_size 67108864K rsz 1024 crg 256 thr 2048 write 4189.18 MB/s 256 x 16.66 = 4265.14 MB/s read 4399.98 MB/s 256 x 17.24 = 4414.06 MB/s
total_size 67108864K rsz 1024 crg 512 thr 512 write 3876.79 MB/s 512 x 7.69 = 3935.55 MB/s read 5267.24 MB/s 512 x 10.41 = 5332.03 MB/s
total_size 67108864K rsz 1024 crg 512 thr 1024 write 4084.05 MB/s 512 x 7.99 = 4091.80 MB/s read 5179.54 MB/s 512 x 10.25 = 5249.02 MB/s
total_size 67108864K rsz 1024 crg 512 thr 2048 write 3980.91 MB/s 512 x 7.88 = 4033.20 MB/s read 4948.02 MB/s 512 x 9.91 = 5073.24 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 1024 write 3080.92 MB/s 1024 x 3.01 = 3085.94 MB/s read 4366.60 MB/s 1024 x 4.39 = 4492.19 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 2048 write 3152.38 MB/s 1024 x 3.19 = 3271.48 MB/s read 4446.49 MB/s 1024 x 4.41 = 4511.72 MB/s
total_size 67108864K rsz 1024 crg 2048 thr 2048 write 2296.72 MB/s 2048 x 1.14 = 2343.75 MB/s read 3883.91 MB/s 2048 x 1.98 = 4062.50 MB/s
Sixteen 15-disk LUNs (host failover scenario)
Max write ~4GB/sec , Max read ~10GB/sec
Mon Oct 6 13:32:20 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 /dev/raw/raw9 /dev/raw/raw10 /dev/raw/raw11 /dev/raw/raw12 /dev/raw/raw13 /dev/raw/raw14 /dev/raw/raw15 /dev/raw/raw16 from fermi-gpfs01
total_size 134217728K rsz 1024 crg 16 thr 16 write 3855.02 MB/s 16 x 241.03 = 3856.51 MB/s read 5777.80 MB/s 16 x 361.66 = 5786.59 MB/s
total_size 33554432K134217728K rsz 1024 crg 416 thr 832 write 14543928.7920 MB/s 416 x 364245.1461 = 14563929.5775 MB/s read 28956414.5531 MB/s 416 x 724401.6457 = 28986425.5617 MB/s
total_size 33554432K134217728K rsz 1024 crg 416 thr 1664 write 14563926.2849 MB/s 416 x 364245.2773 = 14573931.1073 MB/s read 24586484.2325 MB/s 416 x 615405.1287 = 24606493.4899 MB/s
total_size 33554432K134217728K rsz 1024 crg 416 thr 32128 write 14453923.6297 MB/s 416 x 361245.7273 = 14463931.8873 MB/s read 29946868.6497 MB/s 416 x 749430.9402 = 29996880.7634 MB/s
total_size 33554432K134217728K rsz 1024 crg 416 thr 64256 write 14343924.8445 MB/s 416 x 358245.9746 = 14353927.8931 MB/s read 27406246.2878 MB/s 416 x 685390.5881 = 27426252.3190 MB/s
total_size 33554432K134217728K rsz 1024 crg 832 thr 832 write 20014027.2976 MB/s 832 x 250126.3615 = 20024036.87 MB/s read 16395552.0195 MB/s 832 x 204173.9672 = 16395559.7108 MB/s
total_size 33554432K134217728K rsz 1024 crg 832 thr 1664 write 19804018.6052 MB/s 832 x 247125.8889 = 19834028.0363 MB/s read 14406020.0897 MB/s 832 x 180188.0727 = 14406024.5878 MB/s
total_size 33554432K134217728K rsz 1024 crg 832 thr 32128 write 19474018.0745 MB/s 832 x 243125.6963 = 19494020.5408 MB/s read 16615561.0038 MB/s 832 x 207174.7202 = 16615568.7654 MB/s
total_size 33554432K134217728K rsz 1024 crg 832 thr 64256 write 20164023.9855 MB/s 832 x 252125.4779 = 20194025.7327 MB/s read 17155737.7970 MB/s 832 x 214179.5741 = 17165741.5427 MB/s
total_size 33554432K134217728K rsz 1024 crg 832 thr 128512 write 19614006.6141 MB/s 832 x 245125.5332 = 19644010.2631 MB/s read 16825912.2117 MB/s 832 x 210184.4783 = 16835914.7361 MB/s
total_size 33554432K134217728K rsz 1024 crg 1664 thr 1664 write 20974017.4639 MB/s 1664 x 13162.2880 = 21004019.5217 MB/s read 23797774.0334 MB/s 1664 x 148121.7881 = 23807796.5202 MB/s
total_size 33554432K134217728K rsz 1024 crg 1664 thr 32128 write 20784040.3162 MB/s 1664 x 13063.0028 = 20804049.0868 MB/s read 23897571.9959 MB/s 1664 x 149118.5352 = 23927585.4345 MB/s
total_size 33554432K134217728K rsz 1024 crg 1664 thr 64256 write 20754025.2483 MB/s 1664 x 12963.8107 = 20774036.0325 MB/s read 21937595.7723 MB/s 1664 x 137118.2091 = 21957610.1347 MB/s
total_size 33554432K134217728K rsz 1024 crg 1664 thr 128512 write 20884033.2138 MB/s 1664 x 13063.6214 = 20904041.0014 MB/s read 22067665.0484 MB/s 1664 x 137120.9612 = 22077687.3499 MB/s
total_size 33554432K134217728K rsz 1024 crg 1664 thr 2561024 write 20854056.9154 MB/s 1664 x 13063.4947 = 20874061.8689 MB/s read 23937349.0984 MB/s 1664 x 149115.6807 = 23947364.8750 MB/s
total_size 33554432K134217728K rsz 1024 crg 32128 thr 32128 write 20084020.1323 MB/s 32128 x 6231.8149 = 20094030.8976 MB/s read 31979940.9238 MB/s 32128 x 10078.0900 = 32029984.8213 MB/s
total_size 33554432K134217728K rsz 1024 crg 32128 thr 64256 write 19794068.6999 MB/s 32128 x 6131.9286 = 19814078.5137 MB/s read 31909738.0383 MB/s 32128 x 9976.7917 = 31939749.3676 MB/s
total_size 33554432K134217728K rsz 1024 crg 32128 thr 128512 write 19974090.9043 MB/s 32128 x 6232.4905 = 19994102.8278 MB/s read 323710002.1137 MB/s 32128 x 10178.2651 = 324010048.3683 MB/s
total_size 33554432K134217728K rsz 1024 crg 32128 thr 2561024 write 20054067.3312 MB/s 32128 x 6231.7183 = 20064074.8471 MB/s read 32139940.8619 MB/s 32128 x 10078.6400 = 32209984.5213 MB/s
total_size 33554432K134217728K rsz 1024 crg 32128 thr 5122048 write 19814056.9036 MB/s 32128 x 6231.0083 = 19834074.9571 MB/s read 327710046.9920 MB/s 32128 x 10279.5311 = 328010125.9473 MB/s
total_size 33554432K134217728K rsz 1024 crg 64256 thr 64256 write 19864106.1979 MB/s 64256 x 3116.0610 = 19874121.9209 MB/s read 38359031.4565 MB/s 64256 x 6035.0148 = 38409082.3303 MB/s
total_size 33554432K134217728K rsz 1024 crg 64256 thr 128512 write 19854123.3567 MB/s 64256 x 3116.0617 = 19874140.9262 MB/s read 38048832.4080 MB/s 64256 x 5934.5370 = 38098884.8128 MB/s
total_size 33554432K134217728K rsz 1024 crg 64256 thr 2561024 write 20094056.9003 MB/s 64256 x 3115.4786 = 20144060.1606 MB/s read 38278789.1941 MB/s 64256 x 5934.9955 = 38398845.1121 MB/s
total_size 33554432K134217728K rsz 1024 crg 64256 thr 5122048 write 19864083.9865 MB/s 64256 x 3115.0996 = 19894086.7591 MB/s read 38768863.9050 MB/s 64256 x 6034.7685 = 38888920.5590 MB/s
total_size 33554432K134217728K rsz 1024 crg 64256 thr 10244096 write 19784066.7179 MB/s 64256 x 3015.9690 = 19814069.2082 MB/s read 38078812.0539 MB/s 64256 x 5934.5765 = 38128869.2663 MB/s
total_size 33554432K134217728K rsz 1024 crg 128512 thr 128512 write 19994014.9348 MB/s 128512 x 157.6588 = 20034033.1720 MB/s read 36088436.7408 MB/s 128512 x 2816.2460 = 36148500.5098 MB/s
total_size 33554432K134217728K rsz 1024 crg 128512 thr 2561024 write 20164074.8640 MB/s 128512 x 157.8099 = 20224091.7180 MB/s read 36008229.9717 MB/s 128512 x 2816.1722 = 36058305.9666 MB/s
total_size 33554432K134217728K rsz 1024 crg 128512 thr 5122048 write 20124071.8595 MB/s 128512 x 157.7598 = 20164086.6091 MB/s read 35418453.1705 MB/s 128512 x 2716.7167 = 35478535.3616 MB/s
total_size 33554432K134217728K rsz 1024 crg 128512 thr 10244096 write 20124094.0975 MB/s 128512 x 158.7509 = 20164140.6062 MB/s read 35068294.4124 MB/s 128512 x 2716.5437 = 35258378.3991 MB/s
total_size 33554432K134217728K rsz 1024 crg 2561024 thr 2561024 write 18684085.8438 MB/s 2561024 x 74.6102 = 19484121.2409 MB/s read 28099709.9557 MB/s 2561024 x 119.0579 = 282910029.5930 MB/s
total_size 33554432K134217728K rsz 1024 crg 2561024 thr 5122048 write 19444019.8482 MB/s 2561024 x 74.6501 = 19584111.0133 MB/s read 27299815.5234 MB/s 2561024 x 109.7687 = 275310107.9142 MB/s
total_size 33554432K134217728K rsz 1024 crg 2561024 thr 10244096 write 19603844.3514 MB/s 2561024 x 73.6985 = 19673945.7731 MB/s read 28149757.3780 MB/s 2561024 x 119.0778 = 283410019.4753 MB/s
total_size 33554432K134217728K rsz 1024 crg 5122048 thr 5122048 write 16713412.9025 MB/s 5122048 x 31.2973 = 16843535.5716 MB/s read 23787988.4500 MB/s 5122048 x 4.6717 = 23928535.5816 MB/s
total_size 33554432K134217728K rsz 1024 crg 5122048 thr 10244096 write 16373212.3593 MB/s 5122048 x 31.2267 = 16503417.3997 MB/s read 23737873.2792 MB/s 5122048 x 4.6810 = 23978398.4644 MB/s
total_size 33554432K134217728K rsz 1024 crg 10244096 thr 10244096 write 12312423.9529 MB/s 10244096 x 10.2063 = 12302578.4713 MB/s read 20946793.7241 MB/s 10244096 x 21.0691 = 21097812.3850 MB/s
...