Linux IO性能測試

　　開發環境系統配置如下:

　　Intel SR1625 server <http://hi.baidu.com/fatux/blog/item/:;>, 2 CPU, 32GB內存, 用主板自帶卡做了raid1 0,8個7200轉SATA硬碟

　　操作系統是RHEL 5.3 64位

　　因為物理內存是32GB,因此整個過程都選用了60GB 的數據量來測試,以避免cache的影響

　　1. 用自帶的dd命令先測一下, 塊大小為8k

　　dd只能提供一個大概的測試結果,是連續IO而不是隨機IO

　　讀測試

　　# time dd if=/dev/sda2 f=/dev/null bs=8k count=8388608

　　8388608 0 records in

　　8388608 0 records out

　　68719476736 bytes (69 GB) copied, 516.547 seconds, 133 MB/s

　　real 8m36.926s

　　user 0m0.117s

　　sys 0m55.216s

　　寫測試

　　# time dd if=/dev/zero f=/opt/iotest bs=8k count=8388608

　　8388608 0 records in

　　8388608 0 records out

　　68719476736 bytes (69 GB) copied, 888.398 seconds, 77.4 MB/s

　　real 14m48.743s

　　user 0m3.678s

　　sys 2m47.158s

　　讀寫測試

　　# time dd if=/dev/sda2 f=/opt/iotest bs=8k count=8388608

　　8388608 0 records in

　　8388608 0 records out

　　68719476736 bytes (69 GB) copied, 1869.89 seconds, 36.8 MB/s

　　real 31m10.343s

　　user 0m2.613s

　　sys 3m25.548s

　　2. 接下來用Oracle <http://hi.baidu.com/fatux/blog/item/:;>的orion <http://www.oracle.com/technology/software/tech/orion/index.html>工具來測一下

　　解壓即可使用

　　# gzip orion_linux_x86-64.gz

　　測非同步IO時需要libaio庫

　　# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64

　　# echo $LD_LIBRARY_PATH

　　:/opt/oracle/product/10.2.0/lib:/usr/lib64

　　創建配置文件mytest.lun,列出要測試的分區即可. 注意文件名前綴要跟下面的 testname一致

　　# vi mytest.lun

　　查看mytest.jun內容

　　# cat mytest.lun

　　/dev/sda2

　　先來個simple test

　　# ./orion_linux_x86-64 -run simple -testname mytest -num_disks 8

　　查看測試結果

　　# cat mytest_20081111_1431_summary.txt

　　ORION VERSION 11.1.0.7.0

　　Commandline:

　　-run simple -testname mytest -num_disks 8

　　This maps to this test:

　　Test: mytest

　　Small IO size: 8 KB

　　Large IO size: 1024 KB

　　IO Types: Small Random IOs, Large Random IOs

　　Simulated Array Type: CONCAT

　　Write: 0%

　　Cache Size: Not Entered

　　Duration for each Data Point: 60 seconds

　　Small Columns:, 0

　　Large Columns:, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

　　Total Data Points: 38

　　Name: /dev/sda2 Size: 629143441920

　　1 FILEs found.

　　Maximum Large MBPS=56.97 @ Small=0 and Large=7

　　Maximum Small IOPS=442 @ Small=40 and Large=0

　　Minimum Small Latency=14.62 @ Small=1 and Large=0

　　最大MBPS為56.97,最大IOPS為442

　　再測一下8k隨機讀操作

　　# ./orion_linux_x86-64 -run advanced -testname mytest -num_disks 8 -size_small 8 -size_large 8 -type rand &

　　看看結果

　　# cat mytest_20081111_1519_summary.txt

　　ORION VERSION 11.1.0.7.0

　　Commandline:

　　-run advanced -testname mytest -num_disks 8 -size_small 8 -size_large 8 -type rand

　　This maps to this test:

　　Test: mytest

　　Small IO size: 8 KB

　　Large IO size: 8 KB

　　IO Types: Small Random IOs, Large Random IOs

　　Simulated Array Type: CONCAT

　　Write: 0%

　　Cache Size: Not Entered

　　Duration for each Data Point: 60 seconds

　　Small Columns:, 0

　　Large Columns:, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

　　Total Data Points: 38

　　Name: /dev/sda2 Size: 629143441920

　　1 FILEs found.

　　Maximum Large MBPS=3.21 @ Small=0 and Large=13

　　Maximum Small IOPS=448 @ Small=38 and Large=0

　　Minimum Small Latency=15.16 @ Small=1 and Large=0

　　最大MBPS為3.21(這麼低？？),最大IOPS為448

　　再測一下1M順序讀操作, 失敗了, 原因不明...

　　# ./orion_linux_x86-64 -run advanced -testname mytest -num_disks 8 -size_small 1024 -size_large 1024 -type seq

　　ORION: ORacle IO Numbers -- Version 11.1.0.7.0

　　mytest_20081114_1349

　　Test will take approximately 73 minutes

　　Larger caches may take longer

　　rwbase_run_test: rwbase_reap_req failed

　　rwbase_run_process: rwbase_run_test failed

　　rwbase_rwluns: rwbase_run_process failed

　　orion_warm_cache: Warming cache failed. Continuing

　　看看結果

　　# cat mytest_20081111_1620_summary.txt

　　ORION VERSION 11.1.0.7.0

　　Commandline:

　　-run advanced -testname mytest -num_disks 8 -size_small 1024 -size_large 1024 -type seq

　　This maps to this test:

　　Test: mytest

　　Small IO size: 1024 KB

　　Large IO size: 1024 KB

　　IO Types: Small Random IOs, Large Sequential Streams

　　Number of Concurrent IOs Per Stream: 4

　　Force streams to separate disks: No

　　Simulated Array Type: CONCAT

　　Write: 0%

　　Cache Size: Not Entered

　　Duration for each Data Point: 60 seconds

　　沒結果,失敗

　　3. 用iozone <http://www.iozone.org/>來測一下

　　安裝 <http://hi.baidu.com/fatux/blog/item/:;>

　　# tar -xvf iozone3_345.tar

　　# make linux-AMD64

　　指定64G的文件,只測read/reread和write/rewrite,記錄大小從4k-16k.同時生成一個excel文件iozone.wks

　　# ./iozone -Rab iozone.wks -s64G -i 0 -i 1 -y 4k -q 16k

　　Iozone: Performance Test of File I/O

　　Version $Revision: 3.345 $

　　Compiled for 64 bit mode.

　　Build: linux-AMD64

　　Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins

　　Al Slater, Scott Rhine, Mike Wisner, Ken Goss

　　Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,

　　Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,

　　Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,

　　Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,

　　Fabrice Bacchella, Zhenghua Xue, Qin Li.

　　Run began: Tue Nov 11 10:23:25 2008

　　Excel chart generation enabled

　　Auto Mode

　　File size set to 67108864 KB

　　Using Minimum Record Size 4 KB

　　Using Maximum Record Size 16 KB

　　Command line used: ./iozone -Rab iozone.wks -s64G -i 0 -i 1 -y 4k -q 16k

　　Output is in Kbytes/sec

　　Time Resolution = 0.000001 seconds.

　　Processor cache size set to 1024 Kbytes.

　　Processor cache line size set to 32 bytes.

　　File stride size set to 17 * record size.

　　random random bkwd record stride

　　KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread

　　67108864 4 72882 69470 104898 125512

　　67108864 8 72083 69256 133689 109061

　　67108864 16 73375 69155 142019 116034

　　iozone test complete.

　　Excel output is below:

　　"Writer report"

　　"4" "8" "16"

　　"67108864" 72882 72083 73375

　　"Re-writer report"

　　"4" "8" "16"

　　"67108864" 69470 69256 69155

　　"Reader report"

　　"4" "8" "16"

　　"67108864" 104898 133689 142019

　　"Re-Reader report"

　　"4" "8" "16"

　　"67108864" 125512 109061 116034

　　可以看到,8k的寫是72M/s左右,讀是133M/s左右,跟dd的結果比較接近

　　測一下64G文件8k隨機讀寫

　　# ./iozone -Rab iozone.wks -s64G -i 2 -y 8k -q 8k

　　Iozone: Performance Test of File I/O

　　Version $Revision: 3.345 $

　　Compiled for 64 bit mode.

　　Build: linux-AMD64

　　Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins

　　Al Slater, Scott Rhine, Mike Wisner, Ken Goss

　　Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,

　　Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,

　　Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,

　　Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,

　　Fabrice Bacchella, Zhenghua Xue, Qin Li.

　　Run began: Fri Nov 14 15:52:01 2008

　　Excel chart generation enabled

　　Auto Mode

　　File size set to 67108864 KB

　　Using Minimum Record Size 8 KB

　　Using Maximum Record Size 8 KB

　　Command line used: ./iozone -Rab iozone.wks -s64G -i 2 -y 8k -q 8k

　　Output is in Kbytes/sec

　　Time Resolution = 0.000001 seconds.

　　Processor cache size set to 1024 Kbytes.

　　Processor cache line size set to 32 bytes.

　　File stride size set to 17 * record size.

　　random random bkwd record stride

　　KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread

　　67108864 8

　　Error reading block at 6501007360

　　read: Success

　　出錯了(??)

　　4. 用bonnie <http://www.coker.com.au/bonnie />測一下

　　安裝

　　# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64

　　# ./configure

　　# make

　　# make install

　　開始測試,默認文件大小是內存的2倍

　　# bonnie -d /opt/IOTest/ -m sva17 -u root

　　Using uid:0, gid:0.

　　Writing with putc()...done

　　Writing intelligently...done

　　Rewriting...done

　　Reading with getc()...done

　　Reading intelligently...done

　　start 'em...done...done...done...

　　Create files in sequential order...done.

　　Stat files in sequential order...done.

　　Delete files in sequential order...done.

　　Create files in random order...done.

　　Stat files in random order...done.

　　Delete files in random order...done.

　　Version 1.03e ------Sequential Output------ --Sequential Input- --Random-

　　-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

　　Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

　　sva17 63G 52391 84 35222 7 34323 6 56362 88 131568 10 176.7 0

　　------Sequential Create------ --------Random Create--------

　　-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

　　files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

　　16

　　sva17,63G,52391,84,35222,7,34323,6,56362,88,131568,10,176.7,0,16, , , , , , , , , , , ,

　　順序寫: 按字元是52391KB/s,CPU佔用率84%；按塊是35222KB/s,CPU佔用率7%

　　順序讀: 按字元是56362KB/s, CPU佔用率88%；按塊是131568KB/s,CPU佔用率10%

　　隨機讀寫: 176.7次/s,CPU佔用率0%

　　后兩項全是 (沒結果?)

　　結論:不同測試工具構建出來的測試環境不同,側重點也不一樣,得到的結果可能相差比較大.

　　MBPS:

　　dd和iozone比較接近,讀寫分別是130 和70 .

　　orion讀57左右,寫沒測(會刪掉分區內所有文件!)

　　bonnie 按塊讀是130左右,寫是35左右；按字元讀是56左右,寫是52左右

　　IOPS:

　　dd 無結果

　　orion 440左右(只讀)

　　iozone 出錯

　　bonnie 176.7 (讀寫)

Tags: linux system 系統