The Dell MD3460 controller array can support up to 120 drives with an MD3060e expansion tray attached. Single drive capacities have been steadily increasing, but transfer speeds have remained fairly constant. As a result, RAID6 rebuild times can exceed 24hrs when using 4TB drives. This is the main reason for choosing Dynamic Disk Pools over RAID6 - (DDP) provide lower rebuild times are possible since the RAID parity data is distributed compared to RAID6 because the hotspare space is spread across all drives in the pool. For the latest Fermi storage configuration, we are benchmarking eight 15-disk DDP volumes on each 120-drive MD3460+MD3060e storage config. Both arrays are dual-attached to a failover pair of servers (fermi-gpfs01,2). In a failover scenario, one machine could serve all sixteen DDP volumes.
The sgpdd-survey benchmark simulates performs parallel I/O O using raw unformatted LUNs. The tests spawn and an increasing number of threads (thr) that write and read to an increasing number of concurrent regions (crg). As the tests progress, I/O becomes more random. This set of tests transfer data in The transfer operations use 1MB records/chunks (rsz).
One 15-disk LUNs
Max write ~1GB/sec , Max read ~1GB/sec
This is a local test that does not require any networking or formatted filesystem. The purpose is to understand the performance limits of the storage hardware before we create filesystems (GPFS, Lustre, etc.). The benchmark results show the MD3460 is capable of sustaining throughput well above 2GB/sec. In order to avoid networking bottlenecks, we may use 4x10Gb ethernet link aggregation instead of the usual 2x10Gb. We also have a limited number of 40Gb ports on the SLAC network.
One 15-disk LUN
Max write ~1GB/sec , Max read ~1GB/sec
Mon OctMon Oct 6 10:26:06 PDT 2014 sgpdd-survey on /dev/raw/raw1 from fermi-gpfs01
total_size 8388608K rsz 1024 crg 1 thr 1 write 438.64 MB/s 1 x 438.94 = 438.94 MB/s read 893.60 MB/s 1 x 894.41 = 894.41 MB/s
total_size 8388608K rsz 1024 crg 1 thr 2 write 482.91 MB/s 1 x 483.15 = 483.15 MB/s read 882.93 MB/s 1 x 883.72 = 883.72 MB/s
total_size 8388608K rsz 1024 crg 1 thr 4 write 469.27 MB/s 1 x 469.49 = 469.49 MB/s read 871.93 MB/s 1 x 872.71 = 872.71 MB/s
total_size 8388608K rsz 1024 crg 1 thr 8 write 476.82 MB/s 1 x 477.05 = 477.05 MB/s read 876.47 MB/s 1 x 877.22 = 877.22 MB/s
total_size 8388608K rsz 1024 crg 1 thr 16 write 473.50 MB/s 1 x 474.09 = 474.09 MB/s read 869.43 MB/s 1 x 870.23 = 870.23 MB/s
total_size 8388608K rsz 1024 crg 2 thr 2 write 813.07 MB/s 2 x 406.87 = 813.73 MB/s read 489.03 MB/s 2 x 244.65 = 489.29 MB/s
total_size 8388608K rsz 1024 crg 2 thr 4 write 855.20 MB/s 2 x 427.98 = 855.96 MB/s read 482.71 MB/s 2 x 241.48 = 482.96 MB/s
total_size 8388608K rsz 1024 crg 2 thr 8 write 848.02 MB/s 2 x 424.39 = 848.77 MB/s read 493.07 MB/s 2 x 246.66 = 493.32 MB/s
total_size 8388608K rsz 1024 crg 2 thr 16 write 835.96 MB/s 2 x 418.38 = 836.75 MB/s read 480.13 MB/s 2 x 240.18 = 480.37 MB/s
total_size 8388608K rsz 1024 crg 2 thr 32 write 830.18 MB/s 2 x 415.48 = 830.96 MB/s read 487.15 MB/s 2 x 243.70 = 487.40 MB/s
total_size 8388608K rsz 1024 crg 4 thr 4 write 1042.60 MB/s 4 x 260.93 = 1043.70 MB/s read 669.98 MB/s 4 x 167.62 = 670.47 MB/s
total_size 8388608K rsz 1024 crg 4 thr 8 write 972.11 MB/s 4 x 243.27 = 973.09 MB/s read 668.05 MB/s 4 x 167.13 = 668.53 MB/s
total_size 8388608K rsz 1024 crg 4 thr 16 write 1019.04 MB/s 4 x 255.04 = 1020.16 MB/s read 640.78 MB/s 4 x 160.30 = 641.21 MB/s
total_size 8388608K rsz 1024 crg 4 thr 32 write 1034.45 MB/s 4 x 258.92 = 1035.69 MB/s read 656.30 MB/s 4 x 164.19 = 656.78 MB/s
total_size 8388608K rsz 1024 crg 4 thr 64 write 1016.87 MB/s 4 x 254.52 = 1018.07 MB/s read 644.36 MB/s 4 x 161.21 = 644.84 MB/s
total_size 8388608K rsz 1024 crg 8 thr 8 write 823.00 MB/s 8 x 102.97 = 823.75 MB/s read 866.24 MB/s 8 x 108.39 = 867.08 MB/s
total_size 8388608K rsz 1024 crg 8 thr 16 write 788.51 MB/s 8 x 98.66 = 789.26 MB/s read 862.38 MB/s 8 x 107.90 = 863.19 MB/s
total_size 8388608K rsz 1024 crg 8 thr 32 write 781.63 MB/s 8 x 97.79 = 782.32 MB/s read 863.46 MB/s 8 x 108.03 = 864.26 MB/s
total_size 8388608K rsz 1024 crg 8 thr 64 write 790.44 MB/s 8 x 98.90 = 791.17 MB/s read 868.12 MB/s 8 x 108.63 = 869.06 MB/s
total_size 8388608K rsz 1024 crg 8 thr 128 write 787.10 MB/s 8 x 98.49 = 787.89 MB/s read 847.31 MB/s 8 x 106.02 = 848.16 MB/s
total_size 8388608K rsz 1024 crg 16 thr 16 write 609.99 MB/s 16 x 38.16 = 610.50 MB/s read 1028.24 MB/s 16 x 64.33 = 1029.36 MB/s
total_size 8388608K rsz 1024 crg 16 thr 32 write 585.41 MB/s 16 x 36.61 = 585.78 MB/s read 1018.23 MB/s 16 x 63.72 = 1019.59 MB/s
total_size 8388608K rsz 1024 crg 16 thr 64 write 589.83 MB/s 16 x 36.89 = 590.21 MB/s read 1009.79 MB/s 16 x 63.18 = 1010.89 MB/s
total_size 8388608K rsz 1024 crg 16 thr 128 write 585.71 MB/s 16 x 36.64 = 586.24 MB/s read 1024.85 MB/s 16 x 64.13 = 1026.00 MB/s
total_size 8388608K rsz 1024 crg 16 thr 256 write 596.42 MB/s 16 x 37.31 = 596.92 MB/s read 1008.59 MB/s 16 x 63.11 = 1009.83 MB/s
total_size 8388608K rsz 1024 crg 32 thr 32 write 667.16 MB/s 32 x 20.88 = 668.03 MB/s read 946.89 MB/s 32 x 29.62 = 947.88 MB/s
total_size 8388608K rsz 1024 crg 32 thr 64 write 695.57 MB/s 32 x 21.76 = 696.41 MB/s read 960.61 MB/s 32 x 30.06 = 961.91 MB/s
total_size 8388608K rsz 1024 crg 32 thr 128 write 694.31 MB/s 32 x 21.72 = 695.19 MB/s read 956.88 MB/s 32 x 29.94 = 957.95 MB/s
total_size 8388608K rsz 1024 crg 32 thr 256 write 682.67 MB/s 32 x 21.36 = 683.59 MB/s read 951.72 MB/s 32 x 29.79 = 953.37 MB/s
total_size 8388608K rsz 1024 crg 64 thr 64 write 659.29 MB/s 64 x 10.32 = 660.40 MB/s read 757.79 MB/s 64 x 11.85 = 758.67 MB/s
total_size 8388608K rsz 1024 crg 64 thr 128 write 656.74 MB/s 64 x 10.28 = 657.96 MB/s read 757.13 MB/s 64 x 11.84 = 758.06 MB/s
total_size 8388608K rsz 1024 crg 64 thr 256 write 644.79 MB/s 64 x 10.10 = 646.36 MB/s read 756.03 MB/s 64 x 11.84 = 757.45 MB/s
total_size 8388608K rsz 1024 crg 128 thr 128 write 602.88 MB/s 128 x 4.73 = 605.47 MB/s read 638.55 MB/s 128 x 5.01 = 640.87 MB/s
total_size 8388608K rsz 1024 crg 128 thr 256 write 588.55 MB/s 128 x 4.61 = 589.60 MB/s read 623.18 MB/s 128 x 4.88 = 625.00 MB/s
total_size 8388608K rsz 1024 crg 256 thr 256 write 470.98 MB/s 256 x 1.84 = 471.19 MB/s read 564.40 MB/s 256 x 2.22 = 568.85 MB/s
Four 15-disk LUNs
Max write ~2GB/sec , Max read ~3.8GB/sec
Mon Oct 6 11:06:36 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 from fermi-gpfs01
total_size 33554432K rsz 1024 crg 4 thr 4 write 1328.92 MB/s 4 x 332.39 = 1329.57 MB/s read 2911.00 MB/s 4 x 728.76 = 2915.04 MB/s
total_size 33554432K rsz 1024 crg 4 thr 8 write 1454.79 MB/s 4 x 364.14 = 1456.57 MB/s read 2895.55 MB/s 4 x 724.64 = 2898.56 MB/s
total_size 33554432K rsz 1024 crg 4 thr 16 write 1456.28 MB/s 4 x 364.27 = 1457.10 MB/s read 2458.23 MB/s 4 x 615.12 = 2460.48 MB/s
total_size 33554432K rsz 1024 crg 4 thr 32 write 1445.62 MB/s 4 x 361.72 = 1446.88 MB/s read 2994.64 MB/s 4 x 749.94 = 2999.76 MB/s
total_size 33554432K rsz 1024 crg 4 thr 64 write 1434.84 MB/s 4 x 358.97 = 1435.89 MB/s read 2740.28 MB/s 4 x 685.58 = 2742.31 MB/s
total_size 33554432K rsz 1024 crg 8 thr 8 write 2001.29 MB/s 8 x 250.36 = 2002.87 MB/s read 1639.01 MB/s 8 x 204.96 = 1639.71 MB/s
total_size 33554432K rsz 1024 crg 8 thr 16 write 1980.60 MB/s 8 x 247.88 = 1983.03 MB/s read 1440.08 MB/s 8 x 180.07 = 1440.58 MB/s
total_size 33554432K rsz 1024 crg 8 thr 32 write 1947.07 MB/s 8 x 243.69 = 1949.54 MB/s read 1661.00 MB/s 8 x 207.72 = 1661.76 MB/s
total_size 33554432K rsz 1024 crg 8 thr 64 write 2016.98 MB/s 8 x 252.47 = 2019.73 MB/s read 1715.79 MB/s 8 x 214.57 = 1716.54 MB/s
total_size 33554432K rsz 1024 crg 8 thr 128 write 1961.61 MB/s 8 x 245.53 = 1964.26 MB/s read 1682.21 MB/s 8 x 210.47 = 1683.73 MB/s
total_size 33554432K rsz 1024 crg 16 thr 16 write 2097.46 MB/s 16 x 131.28 = 2100.52 MB/s read 2379.03 MB/s 16 x 148.78 = 2380.52 MB/s
total_size 33554432K rsz 1024 crg 16 thr 32 write 2078.31 MB/s 16 x 130.00 = 2080.08 MB/s read 2389.99 MB/s 16 x 149.53 = 2392.43 MB/s
total_size 33554432K rsz 1024 crg 16 thr 64 write 2075.24 MB/s 16 x 129.81 = 2077.03 MB/s read 2193.77 MB/s 16 x 137.20 = 2195.13 MB/s
total_size 33554432K rsz 1024 crg 16 thr 128 write 2088.21 MB/s 16 x 130.62 = 2090.00 MB/s read 2206.04 MB/s 16 x 137.96 = 2207.34 MB/s
total_size 33554432K rsz 1024 crg 16 thr 256 write 2085.91 MB/s 16 x 130.49 = 2087.86 MB/s read 2393.09 MB/s 16 x 149.68 = 2394.87 MB/s
total_size 33554432K rsz 1024 crg 32 thr 32 write 2008.13 MB/s 32 x 62.81 = 2009.89 MB/s read 3197.92 MB/s 32 x 100.09 = 3202.82 MB/s
total_size 33554432K rsz 1024 crg 32 thr 64 write 1979.69 MB/s 32 x 61.92 = 1981.51 MB/s read 3190.03 MB/s 32 x 99.79 = 3193.36 MB/s
total_size 33554432K rsz 1024 crg 32 thr 128 write 1997.90 MB/s 32 x 62.49 = 1999.82 MB/s read 3237.11 MB/s 32 x 101.26 = 3240.36 MB/s
total_size 33554432K rsz 1024 crg 32 thr 256 write 2005.33 MB/s 32 x 62.71 = 2006.84 MB/s read 3213.86 MB/s 32 x 100.64 = 3220.52 MB/s
total_size 33554432K rsz 1024 crg 32 thr 512 write 1981.90 MB/s 32 x 62.00 = 1983.95 MB/s read 3277.99 MB/s 32 x 102.53 = 3280.94 MB/s
total_size 33554432K rsz 1024 crg 64 thr 64 write 1986.19 MB/s 64 x 31.06 = 1987.92 MB/s read 3835.45 MB/s 64 x 60.01 = 3840.33 MB/s
total_size 33554432K rsz 1024 crg 64 thr 128 write 1985.35 MB/s 64 x 31.06 = 1987.92 MB/s read 3804.40 MB/s 64 x 59.53 = 3809.81 MB/s
total_size 33554432K rsz 1024 crg 64 thr 256 write 2009.90 MB/s 64 x 31.47 = 2014.16 MB/s read 3827.19 MB/s 64 x 59.99 = 3839.11 MB/s
total_size 33554432K rsz 1024 crg 64 thr 512 write 1986.98 MB/s 64 x 31.09 = 1989.75 MB/s read 3876.90 MB/s 64 x 60.76 = 3888.55 MB/s
total_size 33554432K rsz 1024 crg 64 thr 1024 write 1978.71 MB/s 64 x 30.96 = 1981.20 MB/s read 3807.05 MB/s 64 x 59.57 = 3812.26 MB/s
total_size 33554432K rsz 1024 crg 128 thr 128 write 1999.93 MB/s 128 x 15.65 = 2003.17 MB/s read 3608.74 MB/s 128 x 28.24 = 3614.50 MB/s
total_size 33554432K rsz 1024 crg 128 thr 256 write 2016.86 MB/s 128 x 15.80 = 2022.71 MB/s read 3600.97 MB/s 128 x 28.17 = 3605.96 MB/s
total_size 33554432K rsz 1024 crg 128 thr 512 write 2012.85 MB/s 128 x 15.75 = 2016.60 MB/s read 3541.17 MB/s 128 x 27.71 = 3547.36 MB/s
total_size 33554432K rsz 1024 crg 128 thr 1024 write 2012.09 MB/s 128 x 15.75 = 2016.60 MB/s read 3506.41 MB/s 128 x 27.54 = 3525.39 MB/s
total_size 33554432K rsz 1024 crg 256 thr 256 write 1868.84 MB/s 256 x 7.61 = 1948.24 MB/s read 2809.95 MB/s 256 x 11.05 = 2829.59 MB/s
total_size 33554432K rsz 1024 crg 256 thr 512 write 1944.84 MB/s 256 x 7.65 = 1958.01 MB/s read 2729.52 MB/s 256 x 10.76 = 2753.91 MB/s
total_size 33554432K rsz 1024 crg 256 thr 1024 write 1960.35 MB/s 256 x 7.69 = 1967.77 MB/s read 2814.37 MB/s 256 x 11.07 = 2834.47 MB/s
total_size 33554432K rsz 1024 crg 512 thr 512 write 1671.90 MB/s 512 x 3.29 = 1684.57 MB/s read 2378.45 MB/s 512 x 4.67 = 2392.58 MB/s
total_size 33554432K rsz 1024 crg 512 thr 1024 write 1637.35 MB/s 512 x 3.22 = 1650.39 MB/s read 2373.27 MB/s 512 x 4.68 = 2397.46 MB/s
total_size 33554432K rsz 1024 crg 1024 thr 1024 write 1231.95 MB/s 1024 x 1.20 = 1230.47 MB/s read 2094.72 MB/s 1024 x 2.06 = 2109.38 MB/s
Eight
...
15-disk
...
LUNs
Max write ~2GB/sec , Max read ~5.5GB/sec
Mon Oct 6 11:27:15 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 from fermi-gpfs01total_size 67108864K rsz 1024 crg 8 thr 8 write 1922.79 MB/s 8 x 240.58 = 1924.67 MB/s read 3002.11 MB/s 8 x 375.48 = 3003.85 MB/s
total_size 67108864K rsz 1024 crg 8 thr 16 write 1954.89 MB/s 8 x 244.51 = 1956.10 MB/s read 3067.43 MB/s 8 x 383.64 = 3069.15 MB/s
total_size 67108864K rsz 1024 crg 8 thr 32 write 1945.17 MB/s 8 x 243.23 = 1945.88 MB/s read 3554.62 MB/s 8 x 444.55 = 3556.37 MB/s
total_size 67108864K rsz 1024 crg 8 thr 64 write 1963.69 MB/s 8 x 245.61 = 1964.87 MB/s read 3241.54 MB/s 8 x 405.60 = 3244.78 MB/s
total_size 67108864K rsz 1024 crg 8 thr 128 write 1952.31 MB/s 8 x 244.34 = 1954.73 MB/s read 3438.20 MB/s 8 x 430.24 = 3441.93 MB/s
total_size 67108864K rsz 1024 crg 16 thr 16 write 2012.85 MB/s 16 x 125.86 = 2013.70 MB/s read 3145.18 MB/s 16 x 196.70 = 3147.13 MB/s
total_size 67108864K rsz 1024 crg 16 thr 32 write 1994.44 MB/s 16 x 124.70 = 1995.24 MB/s read 3025.56 MB/s 16 x 189.17 = 3026.73 MB/s
total_size 67108864K rsz 1024 crg 16 thr 64 write 2004.65 MB/s 16 x 125.36 = 2005.77 MB/s read 3004.32 MB/s 16 x 187.85 = 3005.52 MB/s
total_size 67108864K rsz 1024 crg 16 thr 128 write 2007.39 MB/s 16 x 125.61 = 2009.74 MB/s read 3039.40 MB/s 16 x 190.05 = 3040.77 MB/s
total_size 67108864K rsz 1024 crg 16 thr 256 write 1997.99 MB/s 16 x 125.04 = 2000.58 MB/s read 3112.63 MB/s 16 x 194.82 = 3117.07 MB/s
total_size 67108864K rsz 1024 crg 32 thr 32 write 2013.20 MB/s 32 x 62.95 = 2014.47 MB/s read 4255.10 MB/s 32 x 133.25 = 4263.92 MB/s
total_size 67108864K rsz 1024 crg 32 thr 64 write 2045.06 MB/s 32 x 64.00 = 2048.03 MB/s read 3722.31 MB/s 32 x 116.44 = 3726.20 MB/s
total_size 67108864K rsz 1024 crg 32 thr 128 write 2047.41 MB/s 32 x 64.08 = 2050.48 MB/s read 3978.66 MB/s 32 x 124.47 = 3983.15 MB/s
total_size 67108864K rsz 1024 crg 32 thr 256 write 2041.97 MB/s 32 x 63.89 = 2044.37 MB/s read 4020.13 MB/s 32 x 125.73 = 4023.44 MB/s
total_size 67108864K rsz 1024 crg 32 thr 512 write 2047.79 MB/s 32 x 64.11 = 2051.39 MB/s read 3841.14 MB/s 32 x 120.11 = 3843.38 MB/s
total_size 67108864K rsz 1024 crg 64 thr 64 write 2037.39 MB/s 64 x 31.90 = 2041.63 MB/s read 5316.71 MB/s 64 x 83.33 = 5333.25 MB/s
total_size 67108864K rsz 1024 crg 64 thr 128 write 2051.62 MB/s 64 x 32.08 = 2053.22 MB/s read 5470.01 MB/s 64 x 85.74 = 5487.67 MB/s
total_size 67108864K rsz 1024 crg 64 thr 256 write 2057.33 MB/s 64 x 32.18 = 2059.33 MB/s read 5499.66 MB/s 64 x 86.01 = 5504.76 MB/s
total_size 67108864K rsz 1024 crg 64 thr 512 write 2054.44 MB/s 64 x 32.13 = 2056.27 MB/s read 5356.79 MB/s 64 x 83.97 = 5374.15 MB/s
total_size 67108864K rsz 1024 crg 64 thr 1024 write 2062.91 MB/s 64 x 32.26 = 2064.82 MB/s read 5516.21 MB/s 64 x 86.26 = 5520.63 MB/s
total_size 67108864K rsz 1024 crg 128 thr 128 write 2067.42 MB/s 128 x 16.17 = 2070.31 MB/s read 4584.08 MB/s 128 x 35.94 = 4600.83 MB/s
total_size 67108864K rsz 1024 crg 128 thr 256 write 2060.90 MB/s 128 x 16.11 = 2061.77 MB/s read 4556.67 MB/s 128 x 35.69 = 4567.87 MB/s
total_size 67108864K rsz 1024 crg 128 thr 512 write 2065.59 MB/s 128 x 16.16 = 2067.87 MB/s read 4671.71 MB/s 128 x 36.60 = 4685.06 MB/s
total_size 67108864K rsz 1024 crg 128 thr 1024 write 2067.70 MB/s 128 x 16.16 = 2069.09 MB/s read 4517.05 MB/s 128 x 35.47 = 4539.79 MB/s
total_size 67108864K rsz 1024 crg 128 thr 2048 write 2070.00 MB/s 128 x 16.21 = 2075.20 MB/s read 4738.98 MB/s 128 x 37.17 = 4758.30 MB/s
total_size 67108864K rsz 1024 crg 256 thr 256 write 2073.01 MB/s 256 x 8.12 = 2077.64 MB/s read 4380.17 MB/s 256 x 17.19 = 4401.86 MB/s
total_size 67108864K rsz 1024 crg 256 thr 512 write 2071.51 MB/s 256 x 8.10 = 2072.75 MB/s read 4316.17 MB/s 256 x 16.96 = 4340.82 MB/s
total_size 67108864K rsz 1024 crg 256 thr 1024 write 2064.41 MB/s 256 x 8.09 = 2070.31 MB/s read 4915.45 MB/s 256 x 19.34 = 4951.17 MB/s
total_size 67108864K rsz 1024 crg 256 thr 2048 write 2064.69 MB/s 256 x 8.08 = 2067.87 MB/s read 4284.50 MB/s 256 x 16.87 = 4318.85 MB/s
total_size 67108864K rsz 1024 crg 512 thr 512 write 2027.22 MB/s 512 x 3.96 = 2026.37 MB/s read 5133.79 MB/s 512 x 10.04 = 5141.60 MB/s
total_size 67108864K rsz 1024 crg 512 thr 1024 write 2033.61 MB/s 512 x 3.98 = 2036.13 MB/s read 5157.11 MB/s 512 x 10.09 = 5166.02 MB/s
total_size 67108864K rsz 1024 crg 512 thr 2048 write 2049.99 MB/s 512 x 4.01 = 2050.78 MB/s read 5086.87 MB/s 512 x 10.05 = 5146.48 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 1024 write 1813.79 MB/s 1024 x 1.77 = 1816.41 MB/s read 4298.05 MB/s 1024 x 4.28 = 4384.77 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 2048 write 1718.45 MB/s 1024 x 1.68 = 1718.75 MB/s read 4311.54 MB/s 1024 x 4.29 = 4394.53 MB/s
total_size 67108864K rsz 1024 crg 2048 thr 2048 write 1247.56 MB/s 2048 x 0.63 = 1289.06 MB/s read 3850.62 MB/s 2048 x 1.92 = 3925.78 MB/s
Eight
...
15-disk
...
LUNs
...
without cache
...
mirroring
...
(NOT
...
RECOMMENDED
...
)
Max write ~4.2GB/sec , Max read ~5.5GB/sec
Mon Oct 6 12:07:30 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 from fermi-gpfs01
total_size 67108864K rsz 1024 crg 8 thr 8 write 4117.34 MB/s 8 x 515.06 = 4120.48 MB/s read 2449.74 MB/s 8 x 306.57 = 2452.55 MB/s
total_size 67108864K rsz 1024 crg 8 thr 16 write 4094.32 MB/s 8 x 512.19 = 4097.52 MB/s read 2568.02 MB/s 8 x 321.88 = 2575.07 MB/s
total_size 67108864K rsz 1024 crg 8 thr 32 write 4112.33 MB/s 8 x 514.45 = 4115.60 MB/s read 2524.38 MB/s 8 x 315.84 = 2526.70 MB/s
total_size 67108864K rsz 1024 crg 8 thr 64 write 4092.89 MB/s 8 x 512.29 = 4098.28 MB/s read 2590.98 MB/s 8 x 324.43 = 2595.44 MB/s
total_size 67108864K rsz 1024 crg 8 thr 128 write 4170.54 MB/s 8 x 521.89 = 4175.11 MB/s read 2190.28 MB/s 8 x 274.08 = 2192.61 MB/s
total_size 67108864K rsz 1024 crg 16 thr 16 write 4211.79 MB/s 16 x 263.89 = 4222.26 MB/s read 2805.23 MB/s 16 x 175.39 = 2806.24 MB/s
total_size 67108864K rsz 1024 crg 16 thr 32 write 4285.58 MB/s 16 x 268.08 = 4289.25 MB/s read 2899.50 MB/s 16 x 181.46 = 2903.29 MB/s
total_size 67108864K rsz 1024 crg 16 thr 64 write 4303.16 MB/s 16 x 269.66 = 4314.58 MB/s read 2822.25 MB/s 16 x 176.50 = 2823.94 MB/s
total_size 67108864K rsz 1024 crg 16 thr 128 write 4199.59 MB/s 16 x 263.23 = 4211.73 MB/s read 2749.91 MB/s 16 x 172.09 = 2753.45 MB/s
total_size 67108864K rsz 1024 crg 16 thr 256 write 4245.41 MB/s 16 x 266.08 = 4257.20 MB/s read 2714.66 MB/s 16 x 169.85 = 2717.59 MB/s
total_size 67108864K rsz 1024 crg 32 thr 32 write 4186.19 MB/s 32 x 130.94 = 4190.06 MB/s read 3971.82 MB/s 32 x 124.37 = 3979.80 MB/s
total_size 67108864K rsz 1024 crg 32 thr 64 write 4327.97 MB/s 32 x 135.37 = 4331.97 MB/s read 4067.70 MB/s 32 x 127.22 = 4071.04 MB/s
total_size 67108864K rsz 1024 crg 32 thr 128 write 4297.22 MB/s 32 x 134.41 = 4301.15 MB/s read 4096.38 MB/s 32 x 128.13 = 4100.04 MB/s
total_size 67108864K rsz 1024 crg 32 thr 256 write 4346.15 MB/s 32 x 136.18 = 4357.60 MB/s read 4124.96 MB/s 32 x 128.97 = 4127.20 MB/s
total_size 67108864K rsz 1024 crg 32 thr 512 write 4323.25 MB/s 32 x 135.60 = 4339.29 MB/s read 3862.30 MB/s 32 x 120.93 = 3869.63 MB/s
total_size 67108864K rsz 1024 crg 64 thr 64 write 4251.35 MB/s 64 x 66.79 = 4274.29 MB/s read 5469.94 MB/s 64 x 85.75 = 5488.28 MB/s
total_size 67108864K rsz 1024 crg 64 thr 128 write 4181.70 MB/s 64 x 65.56 = 4195.56 MB/s read 5487.44 MB/s 64 x 86.04 = 5506.59 MB/s
total_size 67108864K rsz 1024 crg 64 thr 256 write 4184.92 MB/s 64 x 65.63 = 4200.44 MB/s read 5498.97 MB/s 64 x 86.29 = 5522.46 MB/s
total_size 67108864K rsz 1024 crg 64 thr 512 write 4196.49 MB/s 64 x 65.93 = 4219.36 MB/s read 5412.16 MB/s 64 x 84.85 = 5430.30 MB/s
total_size 67108864K rsz 1024 crg 64 thr 1024 write 4203.88 MB/s 64 x 65.94 = 4219.97 MB/s read 5278.91 MB/s 64 x 82.67 = 5291.14 MB/s
total_size 67108864K rsz 1024 crg 128 thr 128 write 4168.32 MB/s 128 x 32.84 = 4202.88 MB/s read 4579.70 MB/s 128 x 35.92 = 4597.17 MB/s
total_size 67108864K rsz 1024 crg 128 thr 256 write 4080.65 MB/s 128 x 31.93 = 4086.91 MB/s read 4590.17 MB/s 128 x 36.00 = 4608.15 MB/s
total_size 67108864K rsz 1024 crg 128 thr 512 write 4090.79 MB/s 128 x 32.01 = 4096.68 MB/s read 4640.93 MB/s 128 x 36.39 = 4658.20 MB/s
total_size 67108864K rsz 1024 crg 128 thr 1024 write 4134.32 MB/s 128 x 32.35 = 4140.62 MB/s read 4540.94 MB/s 128 x 35.59 = 4555.66 MB/s
total_size 67108864K rsz 1024 crg 128 thr 2048 write 4153.93 MB/s 128 x 32.49 = 4158.94 MB/s read 4625.92 MB/s 128 x 36.29 = 4644.78 MB/s
total_size 67108864K rsz 1024 crg 256 thr 256 write 4251.26 MB/s 256 x 16.74 = 4284.67 MB/s read 4465.30 MB/s 256 x 17.53 = 4487.30 MB/s
total_size 67108864K rsz 1024 crg 256 thr 512 write 4236.42 MB/s 256 x 16.58 = 4245.61 MB/s read 4460.79 MB/s 256 x 17.50 = 4479.98 MB/s
total_size 67108864K rsz 1024 crg 256 thr 1024 write 4238.58 MB/s 256 x 16.69 = 4272.46 MB/s read 4538.00 MB/s 256 x 17.84 = 4567.87 MB/s
total_size 67108864K rsz 1024 crg 256 thr 2048 write 4189.18 MB/s 256 x 16.66 = 4265.14 MB/s read 4399.98 MB/s 256 x 17.24 = 4414.06 MB/s
total_size 67108864K rsz 1024 crg 512 thr 512 write 3876.79 MB/s 512 x 7.69 = 3935.55 MB/s read 5267.24 MB/s 512 x 10.41 = 5332.03 MB/s
total_size 67108864K rsz 1024 crg 512 thr 1024 write 4084.05 MB/s 512 x 7.99 = 4091.80 MB/s read 5179.54 MB/s 512 x 10.25 = 5249.02 MB/s
total_size 67108864K rsz 1024 crg 512 thr 2048 write 3980.91 MB/s 512 x 7.88 = 4033.20 MB/s read 4948.02 MB/s 512 x 9.91 = 5073.24 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 1024 write 3080.92 MB/s 1024 x 3.01 = 3085.94 MB/s read 4366.60 MB/s 1024 x 4.39 = 4492.19 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 2048 write 3152.38 MB/s 1024 x 3.19 = 3271.48 MB/s read 4446.49 MB/s 1024 x 4.41 = 4511.72 MB/s
total_size 67108864K rsz 1024 crg 2048 thr 2048 write 2296.72 MB/s 2048 x 1.14 = 2343.75 MB/s read 3883.91 MB/s 2048 x 1.98 = 4062.50 MB/s
Sixteen 15-disk LUNs (host failover scenario)
Max write ~4GB/sec , Max read ~10GB/sec
Mon Oct 6 13:32:20 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 /dev/raw/raw9 /dev/raw/raw10 /dev/raw/raw11 /dev/raw/raw12 /dev/raw/raw13 /dev/raw/raw14 /dev/raw/raw15 /dev/raw/raw16 from fermi-gpfs01
total_size 67108864K134217728K rsz 1024 crg 816 thr 816 write 41173855.3402 MB/s 816 x 515241.0603 = 41203856.4851 MB/s read 24495777.7480 MB/s 816 x 306361.5766 = 24525786.5559 MB/s
total_size 67108864K134217728K rsz 1024 crg 816 thr 1632 write 40943928.3220 MB/s 816 x 512245.1961 = 40973929.5275 MB/s read 25686414.0231 MB/s 816 x 321401.8857 = 25756425.0717 MB/s
total_size 67108864K134217728K rsz 1024 crg 816 thr 3264 write 41123926.3349 MB/s 816 x 514245.4573 = 41153931.6073 MB/s read 25246484.3825 MB/s 816 x 315405.8487 = 25266493.7099 MB/s
total_size 67108864K134217728K rsz 1024 crg 816 thr 64128 write 40923923.8997 MB/s 816 x 512245.2973 = 40983931.2873 MB/s read 25906868.9897 MB/s 816 x 324430.4302 = 25956880.4434 MB/s
total_size 67108864K134217728K rsz 1024 crg 816 thr 128256 write 41703924.5445 MB/s 816 x 521245.8946 = 41753927.1131 MB/s read 21906246.2878 MB/s 816 x 274390.0881 = 21926252.6190 MB/s
total_size 67108864K134217728K rsz 1024 crg 1632 thr 1632 write 42114027.7976 MB/s 1632 x 263126.8915 = 42224036.2687 MB/s read 28055552.2395 MB/s 1632 x 175173.3972 = 28065559.2408 MB/s
total_size 67108864K134217728K rsz 1024 crg 1632 thr 3264 write 42854018.5852 MB/s 1632 x 268125.0889 = 42894028.2563 MB/s read 28996020.5097 MB/s 1632 x 181188.4627 = 29036024.2978 MB/s
total_size 67108864K134217728K rsz 1024 crg 1632 thr 64128 write 43034018.1645 MB/s 1632 x 269125.6663 = 43144020.5808 MB/s read 28225561.2538 MB/s 1632 x 176174.5002 = 28235568.9454 MB/s
total_size 67108864K134217728K rsz 1024 crg 1632 thr 128256 write 41994023.5955 MB/s 1632 x 263125.2379 = 42114025.7327 MB/s read 27495737.9170 MB/s 1632 x 172179.0941 = 27535741.4527 MB/s
total_size 67108864K134217728K rsz 1024 crg 1632 thr 256512 write 42454006.41 MB/s 1632 x 266125.0832 = 42574010.2031 MB/s read 27145912.6617 MB/s 1632 x 169184.8583 = 27175914.5961 MB/s
total_size 67108864K134217728K rsz 1024 crg 3264 thr 3264 write 41864017.1939 MB/s 3264 x 13062.9480 = 41904019.0617 MB/s read 39717774.8234 MB/s 3264 x 124121.3781 = 39797796.8002 MB/s
total_size 67108864K134217728K rsz 1024 crg 3264 thr 64128 write 43274040.9762 MB/s 3264 x 13563.3728 = 43314049.9768 MB/s read 40677571.7059 MB/s 3264 x 127118.2252 = 40717585.0445 MB/s
total_size 67108864K134217728K rsz 1024 crg 3264 thr 128256 write 42974025.2283 MB/s 3264 x 13463.4107 = 43014036.1525 MB/s read 40967595.3823 MB/s 3264 x 128118.1391 = 41007610.0447 MB/s
total_size 67108864K134217728K rsz 1024 crg 3264 thr 256512 write 43464033.1538 MB/s 3264 x 13663.1814 = 43574041.6014 MB/s read 41247665.9684 MB/s 3264 x 128120.9712 = 41277687.2099 MB/s
total_size 67108864K134217728K rsz 1024 crg 3264 thr 5121024 write 43234056.2554 MB/s 3264 x 13563.6047 = 43394061.2989 MB/s read 38627349.3084 MB/s 3264 x 120115.9307 = 38697364.6350 MB/s
total_size 67108864K134217728K rsz 1024 crg 64128 thr 64128 write 42514020.3523 MB/s 64128 x 6631.7949 = 42744030.2976 MB/s read 54699940.9438 MB/s 64128 x 8578.7500 = 54889984.2813 MB/s
total_size 67108864K134217728K rsz 1024 crg 64128 thr 128256 write 41814068.7099 MB/s 64128 x 6531.5686 = 41954078.5637 MB/s read 54879738.4483 MB/s 64128 x 8676.0417 = 55069749.5976 MB/s
total_size 67108864K134217728K rsz 1024 crg 64128 thr 256512 write 41844090.9243 MB/s 64128 x 6532.6305 = 42004102.4478 MB/s read 549810002.9737 MB/s 64128 x 8678.2951 = 552210048.4683 MB/s
total_size 67108864K134217728K rsz 1024 crg 64128 thr 5121024 write 41964067.4912 MB/s 64128 x 6531.9383 = 42194074.3671 MB/s read 54129940.1619 MB/s 64128 x 8478.8500 = 54309984.3013 MB/s
total_size 67108864K134217728K rsz 1024 crg 64128 thr 10242048 write 42034056.8836 MB/s 64128 x 6531.9483 = 42194074.9771 MB/s read 527810046.9120 MB/s 64128 x 8279.6711 = 529110125.1473 MB/s
total_size 67108864K134217728K rsz 1024 crg 128256 thr 128256 write 41684106.3279 MB/s 128256 x 3216.8410 = 42024121.8809 MB/s read 45799031.7065 MB/s 128256 x 35.9248 = 45979082.1703 MB/s
total_size 67108864K134217728K rsz 1024 crg 128256 thr 256512 write 40804123.6567 MB/s 128256 x 3116.9317 = 40864140.9162 MB/s read 45908832.1780 MB/s 128256 x 3634.0070 = 46088884.1528 MB/s
total_size 67108864K134217728K rsz 1024 crg 128256 thr 5121024 write 40904056.7903 MB/s 128256 x 3215.0186 = 40964060.6806 MB/s read 46408789.9341 MB/s 128256 x 3634.3955 = 46588845.2021 MB/s
total_size 67108864K134217728K rsz 1024 crg 128256 thr 10242048 write 41344083.3265 MB/s 128256 x 3215.3596 = 41404086.6291 MB/s read 45408863.9450 MB/s 128256 x 3534.5985 = 45558920.6690 MB/s
total_size 67108864K134217728K rsz 1024 crg 128256 thr 20484096 write 41534066.9379 MB/s 128256 x 3215.4990 = 41584069.9482 MB/s read 46258812.9239 MB/s 128256 x 3634.2965 = 46448869.7863 MB/s
total_size 67108864K134217728K rsz 1024 crg 256512 thr 256512 write 42514014.2648 MB/s 256512 x 167.7488 = 42844033.6720 MB/s read 44658436.3008 MB/s 256512 x 1716.5360 = 44878500.3098 MB/s
total_size 67108864K134217728K rsz 1024 crg 256512 thr 5121024 write 42364074.4240 MB/s 256512 x 167.5899 = 42454091.6180 MB/s read 44608229.7917 MB/s 256512 x 1716.5022 = 44798305.9866 MB/s
total_size 67108864K134217728K rsz 1024 crg 256512 thr 10242048 write 42384071.5895 MB/s 256512 x 167.6998 = 42724086.4691 MB/s read 45388453.0005 MB/s 256512 x 1716.8467 = 45678535.8716 MB/s
total_size 67108864K134217728K rsz 1024 crg 256512 thr 20484096 write 41894094.1875 MB/s 256512 x 168.6609 = 42654140.1462 MB/s read 43998294.9824 MB/s 256512 x 1716.2437 = 44148378.0691 MB/s
total_size 67108864K134217728K rsz 1024 crg 5121024 thr 5121024 write 38764085.7938 MB/s 5121024 x 74.6902 = 39354121.5509 MB/s read 52679709.2457 MB/s 5121024 x 109.4179 = 533210029.0330 MB/s
total_size 67108864K134217728K rsz 1024 crg 5121024 thr 10242048 write 40844019.0582 MB/s 5121024 x 74.9901 = 40914111.8033 MB/s read 51799815.5434 MB/s 5121024 x 109.2587 = 524910107.0242 MB/s
total_size 67108864K134217728K rsz 1024 crg 5121024 thr 20484096 write 39803844.9114 MB/s 5121024 x 73.8885 = 40333945.2031 MB/s read 49489757.0280 MB/s 5121024 x 9.9178 = 507310019.2453 MB/s
total_size 67108864K134217728K rsz 1024 crg 10242048 thr 10242048 write 30803412.9225 MB/s 10242048 x 31.0173 = 30853535.9416 MB/s read 43667988.6000 MB/s 10242048 x 4.3917 = 44928535.1916 MB/s
total_size 67108864K134217728K rsz 1024 crg 10242048 thr 20484096 write 31523212.3893 MB/s 10242048 x 31.1967 = 32713417.4897 MB/s read 44467873.4992 MB/s 10242048 x 4.4110 = 45118398.7244 MB/s
total_size 67108864K134217728K rsz 1024 crg 20484096 thr 20484096 write 22962423.7229 MB/s 20484096 x 10.1463 = 23432578.7513 MB/s read 38836793.9141 MB/s 20484096 x 1.9891 = 40627812.50 MB/s
...