The Dell MD3460 controller array can support up to 120 drives with an MD3060e expansion tray attached. Single drive capacities have been steadily increasing, but transfer speeds have remained fairly constant. As a result, RAID6 rebuild times can exceed 24hrs when using 4TB drives. Dynamic Disk Pools (DDP) provide lower rebuild times compared to RAID6 because the hotspare space is spread across all drives in the pool. For the latest Fermi storage configuration, we are benchmarking eight 15-disk DDP volumes on each 120-drive MD3460+MD3060e storage config. Both arrays are dual-attached to a failover pair of servers (fermi-gpfs01,2). In a failover scenario, one machine could serve all sixteen DDP volumes.
The sgpdd-survey benchmark performs parallel I/O using raw unformatted LUNs. The tests spawn an increasing number of threads (thr) that write and read to an increasing number of concurrent regions (crg). As the tests progress, I/O becomes more random. The transfer operations use 1MB records/chunks (rsz). This is a local test that does not require any networking or formatted filesystem. The purpose is to understand the performance limits of the storage hardware before we create filesystems (GPFS, Lustre, etc.). The benchmark results show the MD3460 is capable of sustaining throughput well above 2GB/sec. In order to avoid networking bottlenecks, we may use 4x10Gb ethernet link aggregation instead of the usual 2x10Gb. We also have a limited number of 40Gb ports on the SLAC network.
Max write ~1GB/sec , Max read ~1GB/sec
Mon Oct 6 10:26:06 PDT 2014 sgpdd-survey on /dev/raw/raw1 from fermi-gpfs01
total_size 8388608K rsz 1024 crg 1 thr 1 write 438.64 MB/s 1 x 438.94 = 438.94 MB/s read 893.60 MB/s 1 x 894.41 = 894.41 MB/s
total_size 8388608K rsz 1024 crg 1 thr 2 write 482.91 MB/s 1 x 483.15 = 483.15 MB/s read 882.93 MB/s 1 x 883.72 = 883.72 MB/s
total_size 8388608K rsz 1024 crg 1 thr 4 write 469.27 MB/s 1 x 469.49 = 469.49 MB/s read 871.93 MB/s 1 x 872.71 = 872.71 MB/s
total_size 8388608K rsz 1024 crg 1 thr 8 write 476.82 MB/s 1 x 477.05 = 477.05 MB/s read 876.47 MB/s 1 x 877.22 = 877.22 MB/s
total_size 8388608K rsz 1024 crg 1 thr 16 write 473.50 MB/s 1 x 474.09 = 474.09 MB/s read 869.43 MB/s 1 x 870.23 = 870.23 MB/s
total_size 8388608K rsz 1024 crg 2 thr 2 write 813.07 MB/s 2 x 406.87 = 813.73 MB/s read 489.03 MB/s 2 x 244.65 = 489.29 MB/s
total_size 8388608K rsz 1024 crg 2 thr 4 write 855.20 MB/s 2 x 427.98 = 855.96 MB/s read 482.71 MB/s 2 x 241.48 = 482.96 MB/s
total_size 8388608K rsz 1024 crg 2 thr 8 write 848.02 MB/s 2 x 424.39 = 848.77 MB/s read 493.07 MB/s 2 x 246.66 = 493.32 MB/s
total_size 8388608K rsz 1024 crg 2 thr 16 write 835.96 MB/s 2 x 418.38 = 836.75 MB/s read 480.13 MB/s 2 x 240.18 = 480.37 MB/s
total_size 8388608K rsz 1024 crg 2 thr 32 write 830.18 MB/s 2 x 415.48 = 830.96 MB/s read 487.15 MB/s 2 x 243.70 = 487.40 MB/s
total_size 8388608K rsz 1024 crg 4 thr 4 write 1042.60 MB/s 4 x 260.93 = 1043.70 MB/s read 669.98 MB/s 4 x 167.62 = 670.47 MB/s
total_size 8388608K rsz 1024 crg 4 thr 8 write 972.11 MB/s 4 x 243.27 = 973.09 MB/s read 668.05 MB/s 4 x 167.13 = 668.53 MB/s
total_size 8388608K rsz 1024 crg 4 thr 16 write 1019.04 MB/s 4 x 255.04 = 1020.16 MB/s read 640.78 MB/s 4 x 160.30 = 641.21 MB/s
total_size 8388608K rsz 1024 crg 4 thr 32 write 1034.45 MB/s 4 x 258.92 = 1035.69 MB/s read 656.30 MB/s 4 x 164.19 = 656.78 MB/s
total_size 8388608K rsz 1024 crg 4 thr 64 write 1016.87 MB/s 4 x 254.52 = 1018.07 MB/s read 644.36 MB/s 4 x 161.21 = 644.84 MB/s
total_size 8388608K rsz 1024 crg 8 thr 8 write 823.00 MB/s 8 x 102.97 = 823.75 MB/s read 866.24 MB/s 8 x 108.39 = 867.08 MB/s
total_size 8388608K rsz 1024 crg 8 thr 16 write 788.51 MB/s 8 x 98.66 = 789.26 MB/s read 862.38 MB/s 8 x 107.90 = 863.19 MB/s
total_size 8388608K rsz 1024 crg 8 thr 32 write 781.63 MB/s 8 x 97.79 = 782.32 MB/s read 863.46 MB/s 8 x 108.03 = 864.26 MB/s
total_size 8388608K rsz 1024 crg 8 thr 64 write 790.44 MB/s 8 x 98.90 = 791.17 MB/s read 868.12 MB/s 8 x 108.63 = 869.06 MB/s
total_size 8388608K rsz 1024 crg 8 thr 128 write 787.10 MB/s 8 x 98.49 = 787.89 MB/s read 847.31 MB/s 8 x 106.02 = 848.16 MB/s
total_size 8388608K rsz 1024 crg 16 thr 16 write 609.99 MB/s 16 x 38.16 = 610.50 MB/s read 1028.24 MB/s 16 x 64.33 = 1029.36 MB/s
total_size 8388608K rsz 1024 crg 16 thr 32 write 585.41 MB/s 16 x 36.61 = 585.78 MB/s read 1018.23 MB/s 16 x 63.72 = 1019.59 MB/s
total_size 8388608K rsz 1024 crg 16 thr 64 write 589.83 MB/s 16 x 36.89 = 590.21 MB/s read 1009.79 MB/s 16 x 63.18 = 1010.89 MB/s
total_size 8388608K rsz 1024 crg 16 thr 128 write 585.71 MB/s 16 x 36.64 = 586.24 MB/s read 1024.85 MB/s 16 x 64.13 = 1026.00 MB/s
total_size 8388608K rsz 1024 crg 16 thr 256 write 596.42 MB/s 16 x 37.31 = 596.92 MB/s read 1008.59 MB/s 16 x 63.11 = 1009.83 MB/s
total_size 8388608K rsz 1024 crg 32 thr 32 write 667.16 MB/s 32 x 20.88 = 668.03 MB/s read 946.89 MB/s 32 x 29.62 = 947.88 MB/s
total_size 8388608K rsz 1024 crg 32 thr 64 write 695.57 MB/s 32 x 21.76 = 696.41 MB/s read 960.61 MB/s 32 x 30.06 = 961.91 MB/s
total_size 8388608K rsz 1024 crg 32 thr 128 write 694.31 MB/s 32 x 21.72 = 695.19 MB/s read 956.88 MB/s 32 x 29.94 = 957.95 MB/s
total_size 8388608K rsz 1024 crg 32 thr 256 write 682.67 MB/s 32 x 21.36 = 683.59 MB/s read 951.72 MB/s 32 x 29.79 = 953.37 MB/s
total_size 8388608K rsz 1024 crg 64 thr 64 write 659.29 MB/s 64 x 10.32 = 660.40 MB/s read 757.79 MB/s 64 x 11.85 = 758.67 MB/s
total_size 8388608K rsz 1024 crg 64 thr 128 write 656.74 MB/s 64 x 10.28 = 657.96 MB/s read 757.13 MB/s 64 x 11.84 = 758.06 MB/s
total_size 8388608K rsz 1024 crg 64 thr 256 write 644.79 MB/s 64 x 10.10 = 646.36 MB/s read 756.03 MB/s 64 x 11.84 = 757.45 MB/s
total_size 8388608K rsz 1024 crg 128 thr 128 write 602.88 MB/s 128 x 4.73 = 605.47 MB/s read 638.55 MB/s 128 x 5.01 = 640.87 MB/s
total_size 8388608K rsz 1024 crg 128 thr 256 write 588.55 MB/s 128 x 4.61 = 589.60 MB/s read 623.18 MB/s 128 x 4.88 = 625.00 MB/s
total_size 8388608K rsz 1024 crg 256 thr 256 write 470.98 MB/s 256 x 1.84 = 471.19 MB/s read 564.40 MB/s 256 x 2.22 = 568.85 MB/s
Max write ~2GB/sec , Max read ~3.8GB/sec
Mon Oct 6 11:06:36 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 from fermi-gpfs01
total_size 33554432K rsz 1024 crg 4 thr 4 write 1328.92 MB/s 4 x 332.39 = 1329.57 MB/s read 2911.00 MB/s 4 x 728.76 = 2915.04 MB/s
total_size 33554432K rsz 1024 crg 4 thr 8 write 1454.79 MB/s 4 x 364.14 = 1456.57 MB/s read 2895.55 MB/s 4 x 724.64 = 2898.56 MB/s
total_size 33554432K rsz 1024 crg 4 thr 16 write 1456.28 MB/s 4 x 364.27 = 1457.10 MB/s read 2458.23 MB/s 4 x 615.12 = 2460.48 MB/s
total_size 33554432K rsz 1024 crg 4 thr 32 write 1445.62 MB/s 4 x 361.72 = 1446.88 MB/s read 2994.64 MB/s 4 x 749.94 = 2999.76 MB/s
total_size 33554432K rsz 1024 crg 4 thr 64 write 1434.84 MB/s 4 x 358.97 = 1435.89 MB/s read 2740.28 MB/s 4 x 685.58 = 2742.31 MB/s
total_size 33554432K rsz 1024 crg 8 thr 8 write 2001.29 MB/s 8 x 250.36 = 2002.87 MB/s read 1639.01 MB/s 8 x 204.96 = 1639.71 MB/s
total_size 33554432K rsz 1024 crg 8 thr 16 write 1980.60 MB/s 8 x 247.88 = 1983.03 MB/s read 1440.08 MB/s 8 x 180.07 = 1440.58 MB/s
total_size 33554432K rsz 1024 crg 8 thr 32 write 1947.07 MB/s 8 x 243.69 = 1949.54 MB/s read 1661.00 MB/s 8 x 207.72 = 1661.76 MB/s
total_size 33554432K rsz 1024 crg 8 thr 64 write 2016.98 MB/s 8 x 252.47 = 2019.73 MB/s read 1715.79 MB/s 8 x 214.57 = 1716.54 MB/s
total_size 33554432K rsz 1024 crg 8 thr 128 write 1961.61 MB/s 8 x 245.53 = 1964.26 MB/s read 1682.21 MB/s 8 x 210.47 = 1683.73 MB/s
total_size 33554432K rsz 1024 crg 16 thr 16 write 2097.46 MB/s 16 x 131.28 = 2100.52 MB/s read 2379.03 MB/s 16 x 148.78 = 2380.52 MB/s
total_size 33554432K rsz 1024 crg 16 thr 32 write 2078.31 MB/s 16 x 130.00 = 2080.08 MB/s read 2389.99 MB/s 16 x 149.53 = 2392.43 MB/s
total_size 33554432K rsz 1024 crg 16 thr 64 write 2075.24 MB/s 16 x 129.81 = 2077.03 MB/s read 2193.77 MB/s 16 x 137.20 = 2195.13 MB/s
total_size 33554432K rsz 1024 crg 16 thr 128 write 2088.21 MB/s 16 x 130.62 = 2090.00 MB/s read 2206.04 MB/s 16 x 137.96 = 2207.34 MB/s
total_size 33554432K rsz 1024 crg 16 thr 256 write 2085.91 MB/s 16 x 130.49 = 2087.86 MB/s read 2393.09 MB/s 16 x 149.68 = 2394.87 MB/s
total_size 33554432K rsz 1024 crg 32 thr 32 write 2008.13 MB/s 32 x 62.81 = 2009.89 MB/s read 3197.92 MB/s 32 x 100.09 = 3202.82 MB/s
total_size 33554432K rsz 1024 crg 32 thr 64 write 1979.69 MB/s 32 x 61.92 = 1981.51 MB/s read 3190.03 MB/s 32 x 99.79 = 3193.36 MB/s
total_size 33554432K rsz 1024 crg 32 thr 128 write 1997.90 MB/s 32 x 62.49 = 1999.82 MB/s read 3237.11 MB/s 32 x 101.26 = 3240.36 MB/s
total_size 33554432K rsz 1024 crg 32 thr 256 write 2005.33 MB/s 32 x 62.71 = 2006.84 MB/s read 3213.86 MB/s 32 x 100.64 = 3220.52 MB/s
total_size 33554432K rsz 1024 crg 32 thr 512 write 1981.90 MB/s 32 x 62.00 = 1983.95 MB/s read 3277.99 MB/s 32 x 102.53 = 3280.94 MB/s
total_size 33554432K rsz 1024 crg 64 thr 64 write 1986.19 MB/s 64 x 31.06 = 1987.92 MB/s read 3835.45 MB/s 64 x 60.01 = 3840.33 MB/s
total_size 33554432K rsz 1024 crg 64 thr 128 write 1985.35 MB/s 64 x 31.06 = 1987.92 MB/s read 3804.40 MB/s 64 x 59.53 = 3809.81 MB/s
total_size 33554432K rsz 1024 crg 64 thr 256 write 2009.90 MB/s 64 x 31.47 = 2014.16 MB/s read 3827.19 MB/s 64 x 59.99 = 3839.11 MB/s
total_size 33554432K rsz 1024 crg 64 thr 512 write 1986.98 MB/s 64 x 31.09 = 1989.75 MB/s read 3876.90 MB/s 64 x 60.76 = 3888.55 MB/s
total_size 33554432K rsz 1024 crg 64 thr 1024 write 1978.71 MB/s 64 x 30.96 = 1981.20 MB/s read 3807.05 MB/s 64 x 59.57 = 3812.26 MB/s
total_size 33554432K rsz 1024 crg 128 thr 128 write 1999.93 MB/s 128 x 15.65 = 2003.17 MB/s read 3608.74 MB/s 128 x 28.24 = 3614.50 MB/s
total_size 33554432K rsz 1024 crg 128 thr 256 write 2016.86 MB/s 128 x 15.80 = 2022.71 MB/s read 3600.97 MB/s 128 x 28.17 = 3605.96 MB/s
total_size 33554432K rsz 1024 crg 128 thr 512 write 2012.85 MB/s 128 x 15.75 = 2016.60 MB/s read 3541.17 MB/s 128 x 27.71 = 3547.36 MB/s
total_size 33554432K rsz 1024 crg 128 thr 1024 write 2012.09 MB/s 128 x 15.75 = 2016.60 MB/s read 3506.41 MB/s 128 x 27.54 = 3525.39 MB/s
total_size 33554432K rsz 1024 crg 256 thr 256 write 1868.84 MB/s 256 x 7.61 = 1948.24 MB/s read 2809.95 MB/s 256 x 11.05 = 2829.59 MB/s
total_size 33554432K rsz 1024 crg 256 thr 512 write 1944.84 MB/s 256 x 7.65 = 1958.01 MB/s read 2729.52 MB/s 256 x 10.76 = 2753.91 MB/s
total_size 33554432K rsz 1024 crg 256 thr 1024 write 1960.35 MB/s 256 x 7.69 = 1967.77 MB/s read 2814.37 MB/s 256 x 11.07 = 2834.47 MB/s
total_size 33554432K rsz 1024 crg 512 thr 512 write 1671.90 MB/s 512 x 3.29 = 1684.57 MB/s read 2378.45 MB/s 512 x 4.67 = 2392.58 MB/s
total_size 33554432K rsz 1024 crg 512 thr 1024 write 1637.35 MB/s 512 x 3.22 = 1650.39 MB/s read 2373.27 MB/s 512 x 4.68 = 2397.46 MB/s
total_size 33554432K rsz 1024 crg 1024 thr 1024 write 1231.95 MB/s 1024 x 1.20 = 1230.47 MB/s read 2094.72 MB/s 1024 x 2.06 = 2109.38 MB/s
Max write ~2GB/sec , Max read ~5.5GB/sec
Mon Oct 6 11:27:15 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 from fermi-gpfs01total_size 67108864K rsz 1024 crg 8 thr 8 write 1922.79 MB/s 8 x 240.58 = 1924.67 MB/s read 3002.11 MB/s 8 x 375.48 = 3003.85 MB/s
total_size 67108864K rsz 1024 crg 8 thr 16 write 1954.89 MB/s 8 x 244.51 = 1956.10 MB/s read 3067.43 MB/s 8 x 383.64 = 3069.15 MB/s
total_size 67108864K rsz 1024 crg 8 thr 32 write 1945.17 MB/s 8 x 243.23 = 1945.88 MB/s read 3554.62 MB/s 8 x 444.55 = 3556.37 MB/s
total_size 67108864K rsz 1024 crg 8 thr 64 write 1963.69 MB/s 8 x 245.61 = 1964.87 MB/s read 3241.54 MB/s 8 x 405.60 = 3244.78 MB/s
total_size 67108864K rsz 1024 crg 8 thr 128 write 1952.31 MB/s 8 x 244.34 = 1954.73 MB/s read 3438.20 MB/s 8 x 430.24 = 3441.93 MB/s
total_size 67108864K rsz 1024 crg 16 thr 16 write 2012.85 MB/s 16 x 125.86 = 2013.70 MB/s read 3145.18 MB/s 16 x 196.70 = 3147.13 MB/s
total_size 67108864K rsz 1024 crg 16 thr 32 write 1994.44 MB/s 16 x 124.70 = 1995.24 MB/s read 3025.56 MB/s 16 x 189.17 = 3026.73 MB/s
total_size 67108864K rsz 1024 crg 16 thr 64 write 2004.65 MB/s 16 x 125.36 = 2005.77 MB/s read 3004.32 MB/s 16 x 187.85 = 3005.52 MB/s
total_size 67108864K rsz 1024 crg 16 thr 128 write 2007.39 MB/s 16 x 125.61 = 2009.74 MB/s read 3039.40 MB/s 16 x 190.05 = 3040.77 MB/s
total_size 67108864K rsz 1024 crg 16 thr 256 write 1997.99 MB/s 16 x 125.04 = 2000.58 MB/s read 3112.63 MB/s 16 x 194.82 = 3117.07 MB/s
total_size 67108864K rsz 1024 crg 32 thr 32 write 2013.20 MB/s 32 x 62.95 = 2014.47 MB/s read 4255.10 MB/s 32 x 133.25 = 4263.92 MB/s
total_size 67108864K rsz 1024 crg 32 thr 64 write 2045.06 MB/s 32 x 64.00 = 2048.03 MB/s read 3722.31 MB/s 32 x 116.44 = 3726.20 MB/s
total_size 67108864K rsz 1024 crg 32 thr 128 write 2047.41 MB/s 32 x 64.08 = 2050.48 MB/s read 3978.66 MB/s 32 x 124.47 = 3983.15 MB/s
total_size 67108864K rsz 1024 crg 32 thr 256 write 2041.97 MB/s 32 x 63.89 = 2044.37 MB/s read 4020.13 MB/s 32 x 125.73 = 4023.44 MB/s
total_size 67108864K rsz 1024 crg 32 thr 512 write 2047.79 MB/s 32 x 64.11 = 2051.39 MB/s read 3841.14 MB/s 32 x 120.11 = 3843.38 MB/s
total_size 67108864K rsz 1024 crg 64 thr 64 write 2037.39 MB/s 64 x 31.90 = 2041.63 MB/s read 5316.71 MB/s 64 x 83.33 = 5333.25 MB/s
total_size 67108864K rsz 1024 crg 64 thr 128 write 2051.62 MB/s 64 x 32.08 = 2053.22 MB/s read 5470.01 MB/s 64 x 85.74 = 5487.67 MB/s
total_size 67108864K rsz 1024 crg 64 thr 256 write 2057.33 MB/s 64 x 32.18 = 2059.33 MB/s read 5499.66 MB/s 64 x 86.01 = 5504.76 MB/s
total_size 67108864K rsz 1024 crg 64 thr 512 write 2054.44 MB/s 64 x 32.13 = 2056.27 MB/s read 5356.79 MB/s 64 x 83.97 = 5374.15 MB/s
total_size 67108864K rsz 1024 crg 64 thr 1024 write 2062.91 MB/s 64 x 32.26 = 2064.82 MB/s read 5516.21 MB/s 64 x 86.26 = 5520.63 MB/s
total_size 67108864K rsz 1024 crg 128 thr 128 write 2067.42 MB/s 128 x 16.17 = 2070.31 MB/s read 4584.08 MB/s 128 x 35.94 = 4600.83 MB/s
total_size 67108864K rsz 1024 crg 128 thr 256 write 2060.90 MB/s 128 x 16.11 = 2061.77 MB/s read 4556.67 MB/s 128 x 35.69 = 4567.87 MB/s
total_size 67108864K rsz 1024 crg 128 thr 512 write 2065.59 MB/s 128 x 16.16 = 2067.87 MB/s read 4671.71 MB/s 128 x 36.60 = 4685.06 MB/s
total_size 67108864K rsz 1024 crg 128 thr 1024 write 2067.70 MB/s 128 x 16.16 = 2069.09 MB/s read 4517.05 MB/s 128 x 35.47 = 4539.79 MB/s
total_size 67108864K rsz 1024 crg 128 thr 2048 write 2070.00 MB/s 128 x 16.21 = 2075.20 MB/s read 4738.98 MB/s 128 x 37.17 = 4758.30 MB/s
total_size 67108864K rsz 1024 crg 256 thr 256 write 2073.01 MB/s 256 x 8.12 = 2077.64 MB/s read 4380.17 MB/s 256 x 17.19 = 4401.86 MB/s
total_size 67108864K rsz 1024 crg 256 thr 512 write 2071.51 MB/s 256 x 8.10 = 2072.75 MB/s read 4316.17 MB/s 256 x 16.96 = 4340.82 MB/s
total_size 67108864K rsz 1024 crg 256 thr 1024 write 2064.41 MB/s 256 x 8.09 = 2070.31 MB/s read 4915.45 MB/s 256 x 19.34 = 4951.17 MB/s
total_size 67108864K rsz 1024 crg 256 thr 2048 write 2064.69 MB/s 256 x 8.08 = 2067.87 MB/s read 4284.50 MB/s 256 x 16.87 = 4318.85 MB/s
total_size 67108864K rsz 1024 crg 512 thr 512 write 2027.22 MB/s 512 x 3.96 = 2026.37 MB/s read 5133.79 MB/s 512 x 10.04 = 5141.60 MB/s
total_size 67108864K rsz 1024 crg 512 thr 1024 write 2033.61 MB/s 512 x 3.98 = 2036.13 MB/s read 5157.11 MB/s 512 x 10.09 = 5166.02 MB/s
total_size 67108864K rsz 1024 crg 512 thr 2048 write 2049.99 MB/s 512 x 4.01 = 2050.78 MB/s read 5086.87 MB/s 512 x 10.05 = 5146.48 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 1024 write 1813.79 MB/s 1024 x 1.77 = 1816.41 MB/s read 4298.05 MB/s 1024 x 4.28 = 4384.77 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 2048 write 1718.45 MB/s 1024 x 1.68 = 1718.75 MB/s read 4311.54 MB/s 1024 x 4.29 = 4394.53 MB/s
total_size 67108864K rsz 1024 crg 2048 thr 2048 write 1247.56 MB/s 2048 x 0.63 = 1289.06 MB/s read 3850.62 MB/s 2048 x 1.92 = 3925.78 MB/s
Max write ~4.2GB/sec , Max read ~5.5GB/sec
Mon Oct 6 12:07:30 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 from fermi-gpfs01
total_size 67108864K rsz 1024 crg 8 thr 8 write 4117.34 MB/s 8 x 515.06 = 4120.48 MB/s read 2449.74 MB/s 8 x 306.57 = 2452.55 MB/s
total_size 67108864K rsz 1024 crg 8 thr 16 write 4094.32 MB/s 8 x 512.19 = 4097.52 MB/s read 2568.02 MB/s 8 x 321.88 = 2575.07 MB/s
total_size 67108864K rsz 1024 crg 8 thr 32 write 4112.33 MB/s 8 x 514.45 = 4115.60 MB/s read 2524.38 MB/s 8 x 315.84 = 2526.70 MB/s
total_size 67108864K rsz 1024 crg 8 thr 64 write 4092.89 MB/s 8 x 512.29 = 4098.28 MB/s read 2590.98 MB/s 8 x 324.43 = 2595.44 MB/s
total_size 67108864K rsz 1024 crg 8 thr 128 write 4170.54 MB/s 8 x 521.89 = 4175.11 MB/s read 2190.28 MB/s 8 x 274.08 = 2192.61 MB/s
total_size 67108864K rsz 1024 crg 16 thr 16 write 4211.79 MB/s 16 x 263.89 = 4222.26 MB/s read 2805.23 MB/s 16 x 175.39 = 2806.24 MB/s
total_size 67108864K rsz 1024 crg 16 thr 32 write 4285.58 MB/s 16 x 268.08 = 4289.25 MB/s read 2899.50 MB/s 16 x 181.46 = 2903.29 MB/s
total_size 67108864K rsz 1024 crg 16 thr 64 write 4303.16 MB/s 16 x 269.66 = 4314.58 MB/s read 2822.25 MB/s 16 x 176.50 = 2823.94 MB/s
total_size 67108864K rsz 1024 crg 16 thr 128 write 4199.59 MB/s 16 x 263.23 = 4211.73 MB/s read 2749.91 MB/s 16 x 172.09 = 2753.45 MB/s
total_size 67108864K rsz 1024 crg 16 thr 256 write 4245.41 MB/s 16 x 266.08 = 4257.20 MB/s read 2714.66 MB/s 16 x 169.85 = 2717.59 MB/s
total_size 67108864K rsz 1024 crg 32 thr 32 write 4186.19 MB/s 32 x 130.94 = 4190.06 MB/s read 3971.82 MB/s 32 x 124.37 = 3979.80 MB/s
total_size 67108864K rsz 1024 crg 32 thr 64 write 4327.97 MB/s 32 x 135.37 = 4331.97 MB/s read 4067.70 MB/s 32 x 127.22 = 4071.04 MB/s
total_size 67108864K rsz 1024 crg 32 thr 128 write 4297.22 MB/s 32 x 134.41 = 4301.15 MB/s read 4096.38 MB/s 32 x 128.13 = 4100.04 MB/s
total_size 67108864K rsz 1024 crg 32 thr 256 write 4346.15 MB/s 32 x 136.18 = 4357.60 MB/s read 4124.96 MB/s 32 x 128.97 = 4127.20 MB/s
total_size 67108864K rsz 1024 crg 32 thr 512 write 4323.25 MB/s 32 x 135.60 = 4339.29 MB/s read 3862.30 MB/s 32 x 120.93 = 3869.63 MB/s
total_size 67108864K rsz 1024 crg 64 thr 64 write 4251.35 MB/s 64 x 66.79 = 4274.29 MB/s read 5469.94 MB/s 64 x 85.75 = 5488.28 MB/s
total_size 67108864K rsz 1024 crg 64 thr 128 write 4181.70 MB/s 64 x 65.56 = 4195.56 MB/s read 5487.44 MB/s 64 x 86.04 = 5506.59 MB/s
total_size 67108864K rsz 1024 crg 64 thr 256 write 4184.92 MB/s 64 x 65.63 = 4200.44 MB/s read 5498.97 MB/s 64 x 86.29 = 5522.46 MB/s
total_size 67108864K rsz 1024 crg 64 thr 512 write 4196.49 MB/s 64 x 65.93 = 4219.36 MB/s read 5412.16 MB/s 64 x 84.85 = 5430.30 MB/s
total_size 67108864K rsz 1024 crg 64 thr 1024 write 4203.88 MB/s 64 x 65.94 = 4219.97 MB/s read 5278.91 MB/s 64 x 82.67 = 5291.14 MB/s
total_size 67108864K rsz 1024 crg 128 thr 128 write 4168.32 MB/s 128 x 32.84 = 4202.88 MB/s read 4579.70 MB/s 128 x 35.92 = 4597.17 MB/s
total_size 67108864K rsz 1024 crg 128 thr 256 write 4080.65 MB/s 128 x 31.93 = 4086.91 MB/s read 4590.17 MB/s 128 x 36.00 = 4608.15 MB/s
total_size 67108864K rsz 1024 crg 128 thr 512 write 4090.79 MB/s 128 x 32.01 = 4096.68 MB/s read 4640.93 MB/s 128 x 36.39 = 4658.20 MB/s
total_size 67108864K rsz 1024 crg 128 thr 1024 write 4134.32 MB/s 128 x 32.35 = 4140.62 MB/s read 4540.94 MB/s 128 x 35.59 = 4555.66 MB/s
total_size 67108864K rsz 1024 crg 128 thr 2048 write 4153.93 MB/s 128 x 32.49 = 4158.94 MB/s read 4625.92 MB/s 128 x 36.29 = 4644.78 MB/s
total_size 67108864K rsz 1024 crg 256 thr 256 write 4251.26 MB/s 256 x 16.74 = 4284.67 MB/s read 4465.30 MB/s 256 x 17.53 = 4487.30 MB/s
total_size 67108864K rsz 1024 crg 256 thr 512 write 4236.42 MB/s 256 x 16.58 = 4245.61 MB/s read 4460.79 MB/s 256 x 17.50 = 4479.98 MB/s
total_size 67108864K rsz 1024 crg 256 thr 1024 write 4238.58 MB/s 256 x 16.69 = 4272.46 MB/s read 4538.00 MB/s 256 x 17.84 = 4567.87 MB/s
total_size 67108864K rsz 1024 crg 256 thr 2048 write 4189.18 MB/s 256 x 16.66 = 4265.14 MB/s read 4399.98 MB/s 256 x 17.24 = 4414.06 MB/s
total_size 67108864K rsz 1024 crg 512 thr 512 write 3876.79 MB/s 512 x 7.69 = 3935.55 MB/s read 5267.24 MB/s 512 x 10.41 = 5332.03 MB/s
total_size 67108864K rsz 1024 crg 512 thr 1024 write 4084.05 MB/s 512 x 7.99 = 4091.80 MB/s read 5179.54 MB/s 512 x 10.25 = 5249.02 MB/s
total_size 67108864K rsz 1024 crg 512 thr 2048 write 3980.91 MB/s 512 x 7.88 = 4033.20 MB/s read 4948.02 MB/s 512 x 9.91 = 5073.24 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 1024 write 3080.92 MB/s 1024 x 3.01 = 3085.94 MB/s read 4366.60 MB/s 1024 x 4.39 = 4492.19 MB/s
total_size 67108864K rsz 1024 crg 1024 thr 2048 write 3152.38 MB/s 1024 x 3.19 = 3271.48 MB/s read 4446.49 MB/s 1024 x 4.41 = 4511.72 MB/s
total_size 67108864K rsz 1024 crg 2048 thr 2048 write 2296.72 MB/s 2048 x 1.14 = 2343.75 MB/s read 3883.91 MB/s 2048 x 1.98 = 4062.50 MB/s
Max write ~4GB/sec , Max read ~10GB/sec
Mon Oct 6 13:32:20 PDT 2014 sgpdd-survey on /dev/raw/raw1 /dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5 /dev/raw/raw6 /dev/raw/raw7 /dev/raw/raw8 /dev/raw/raw9 /dev/raw/raw10 /dev/raw/raw11 /dev/raw/raw12 /dev/raw/raw13 /dev/raw/raw14 /dev/raw/raw15 /dev/raw/raw16 from fermi-gpfs01
total_size 134217728K rsz 1024 crg 16 thr 16 write 3855.02 MB/s 16 x 241.03 = 3856.51 MB/s read 5777.80 MB/s 16 x 361.66 = 5786.59 MB/s
total_size 134217728K rsz 1024 crg 16 thr 32 write 3928.20 MB/s 16 x 245.61 = 3929.75 MB/s read 6414.31 MB/s 16 x 401.57 = 6425.17 MB/s
total_size 134217728K rsz 1024 crg 16 thr 64 write 3926.49 MB/s 16 x 245.73 = 3931.73 MB/s read 6484.25 MB/s 16 x 405.87 = 6493.99 MB/s
total_size 134217728K rsz 1024 crg 16 thr 128 write 3923.97 MB/s 16 x 245.73 = 3931.73 MB/s read 6868.97 MB/s 16 x 430.02 = 6880.34 MB/s
total_size 134217728K rsz 1024 crg 16 thr 256 write 3924.45 MB/s 16 x 245.46 = 3927.31 MB/s read 6246.78 MB/s 16 x 390.81 = 6252.90 MB/s
total_size 134217728K rsz 1024 crg 32 thr 32 write 4027.76 MB/s 32 x 126.15 = 4036.87 MB/s read 5552.95 MB/s 32 x 173.72 = 5559.08 MB/s
total_size 134217728K rsz 1024 crg 32 thr 64 write 4018.52 MB/s 32 x 125.89 = 4028.63 MB/s read 6020.97 MB/s 32 x 188.27 = 6024.78 MB/s
total_size 134217728K rsz 1024 crg 32 thr 128 write 4018.45 MB/s 32 x 125.63 = 4020.08 MB/s read 5561.38 MB/s 32 x 174.02 = 5568.54 MB/s
total_size 134217728K rsz 1024 crg 32 thr 256 write 4023.55 MB/s 32 x 125.79 = 4025.27 MB/s read 5737.70 MB/s 32 x 179.41 = 5741.27 MB/s
total_size 134217728K rsz 1024 crg 32 thr 512 write 4006.41 MB/s 32 x 125.32 = 4010.31 MB/s read 5912.17 MB/s 32 x 184.83 = 5914.61 MB/s
total_size 134217728K rsz 1024 crg 64 thr 64 write 4017.39 MB/s 64 x 62.80 = 4019.17 MB/s read 7774.34 MB/s 64 x 121.81 = 7796.02 MB/s
total_size 134217728K rsz 1024 crg 64 thr 128 write 4040.62 MB/s 64 x 63.28 = 4049.68 MB/s read 7571.59 MB/s 64 x 118.52 = 7585.45 MB/s
total_size 134217728K rsz 1024 crg 64 thr 256 write 4025.83 MB/s 64 x 63.07 = 4036.25 MB/s read 7595.23 MB/s 64 x 118.91 = 7610.47 MB/s
total_size 134217728K rsz 1024 crg 64 thr 512 write 4033.38 MB/s 64 x 63.14 = 4041.14 MB/s read 7665.84 MB/s 64 x 120.12 = 7687.99 MB/s
total_size 134217728K rsz 1024 crg 64 thr 1024 write 4056.54 MB/s 64 x 63.47 = 4061.89 MB/s read 7349.84 MB/s 64 x 115.07 = 7364.50 MB/s
total_size 134217728K rsz 1024 crg 128 thr 128 write 4020.23 MB/s 128 x 31.49 = 4030.76 MB/s read 9940.38 MB/s 128 x 78.00 = 9984.13 MB/s
total_size 134217728K rsz 1024 crg 128 thr 256 write 4068.99 MB/s 128 x 31.86 = 4078.37 MB/s read 9738.83 MB/s 128 x 76.17 = 9749.76 MB/s
total_size 134217728K rsz 1024 crg 128 thr 512 write 4090.43 MB/s 128 x 32.05 = 4102.78 MB/s read 10002.37 MB/s 128 x 78.51 = 10048.83 MB/s
total_size 134217728K rsz 1024 crg 128 thr 1024 write 4067.12 MB/s 128 x 31.83 = 4074.71 MB/s read 9940.19 MB/s 128 x 78.00 = 9984.13 MB/s
total_size 134217728K rsz 1024 crg 128 thr 2048 write 4056.36 MB/s 128 x 31.83 = 4074.71 MB/s read 10046.20 MB/s 128 x 79.11 = 10125.73 MB/s
total_size 134217728K rsz 1024 crg 256 thr 256 write 4106.79 MB/s 256 x 16.10 = 4121.09 MB/s read 9031.65 MB/s 256 x 35.48 = 9082.03 MB/s
total_size 134217728K rsz 1024 crg 256 thr 512 write 4123.67 MB/s 256 x 16.17 = 4140.62 MB/s read 8832.80 MB/s 256 x 34.70 = 8884.28 MB/s
total_size 134217728K rsz 1024 crg 256 thr 1024 write 4056.03 MB/s 256 x 15.86 = 4060.06 MB/s read 8789.41 MB/s 256 x 34.55 = 8845.21 MB/s
total_size 134217728K rsz 1024 crg 256 thr 2048 write 4083.65 MB/s 256 x 15.96 = 4086.91 MB/s read 8863.50 MB/s 256 x 34.85 = 8920.90 MB/s
total_size 134217728K rsz 1024 crg 256 thr 4096 write 4066.79 MB/s 256 x 15.90 = 4069.82 MB/s read 8812.39 MB/s 256 x 34.65 = 8869.63 MB/s
total_size 134217728K rsz 1024 crg 512 thr 512 write 4014.48 MB/s 512 x 7.88 = 4033.20 MB/s read 8436.08 MB/s 512 x 16.60 = 8500.98 MB/s
total_size 134217728K rsz 1024 crg 512 thr 1024 write 4074.40 MB/s 512 x 7.99 = 4091.80 MB/s read 8229.17 MB/s 512 x 16.22 = 8305.66 MB/s
total_size 134217728K rsz 1024 crg 512 thr 2048 write 4071.95 MB/s 512 x 7.98 = 4086.91 MB/s read 8453.05 MB/s 512 x 16.67 = 8535.16 MB/s
total_size 134217728K rsz 1024 crg 512 thr 4096 write 4094.75 MB/s 512 x 8.09 = 4140.62 MB/s read 8294.24 MB/s 512 x 16.37 = 8378.91 MB/s
total_size 134217728K rsz 1024 crg 1024 thr 1024 write 4085.38 MB/s 1024 x 4.02 = 4121.09 MB/s read 9709.57 MB/s 1024 x 9.79 = 10029.30 MB/s
total_size 134217728K rsz 1024 crg 1024 thr 2048 write 4019.82 MB/s 1024 x 4.01 = 4111.33 MB/s read 9815.34 MB/s 1024 x 9.87 = 10107.42 MB/s
total_size 134217728K rsz 1024 crg 1024 thr 4096 write 3844.14 MB/s 1024 x 3.85 = 3945.31 MB/s read 9757.80 MB/s 1024 x 9.78 = 10019.53 MB/s
total_size 134217728K rsz 1024 crg 2048 thr 2048 write 3412.25 MB/s 2048 x 1.73 = 3535.16 MB/s read 7988.00 MB/s 2048 x 4.17 = 8535.16 MB/s
total_size 134217728K rsz 1024 crg 2048 thr 4096 write 3212.93 MB/s 2048 x 1.67 = 3417.97 MB/s read 7873.92 MB/s 2048 x 4.10 = 8398.44 MB/s
total_size 134217728K rsz 1024 crg 4096 thr 4096 write 2423.29 MB/s 4096 x 0.63 = 2578.13 MB/s read 6793.41 MB/s 4096 x 1.91 = 7812.50 MB/s