Unsolved
This post is more than 5 years old
6 Posts
0
8394
April 16th, 2012 10:00
Performance issues with linux/multipath
Hi, let me first describe my setup.
---GbE x 4----sw1---GbE x 2----server1---GbE x 2 ---- sw2 --- admin lan
In the link between the iscsi switch and the server I'm using LACP, the switch its an HP 1910.
I have found great difference in the performance between VMWARE ESXi and other linux. So far I tried ubuntu server, centos and proxmox ve. My intention it's to setup a virtualization enviroment with KVM.
To meassure performance, I'm using fio, with the following test on a 45Gb lun. raid 6 over 12x 1TB 7.2k :
[global]
rw=randread
size=256m
directory=/opt/data
ioengine=libaio
iodepth=4
invalidate=1
direct=1
[bgwriter]
rw=randwrite
iodepth=32
[queryA]
iodepth=1
ioengine=mmap
direct=0
thinktime=3
[queryB]
iodepth=1
ioengine=mmap
direct=0
thinktime=5
[bgupdater]
rw=randrw
iodepth=16
thinktime=40
size=32m
[seqrw]
iodepth=1
rw=rw
size=1g
This is the output from a vm on vmware:
bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
seqrw: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
Starting 5 processes
seqrw: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [_r___] [100.0% done] [8241K/0K /s] [2012/0 iops] [eta 00m:00s]
bgwriter: (groupid=0, jobs=1): err= 0: pid=1652
write: io=262144KB, bw=5495KB/s, iops=1373, runt= 47710msec
slat (usec): min=2, max=24094K, avg=374.85, stdev=94117.27
clat (usec): min=213, max=29255K, avg=22919.10, stdev=676064.97
bw (KB/s) : min= 0, max=53888, per=536.69%, avg=29212.25, stdev=21405.17
cpu : usr=0.23%, sys=1.05%, ctx=15756, majf=0, minf=26
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued r/w: total=0/65536, short=0/0
lat (usec): 250=0.01%, 1000=0.04%
lat (msec): 2=7.09%, 4=77.75%, 10=14.50%, 20=0.20%, 50=0.07%
lat (msec): 100=0.04%, 500=0.20%, >=2000=0.10%
queryA: (groupid=0, jobs=1): err= 0: pid=1653
read : io=262144KB, bw=1723KB/s, iops=430, runt=152138msec
clat (usec): min=234, max=24112K, avg=2315.23, stdev=104795.59
bw (KB/s) : min= 0, max= 8000, per=32.90%, avg=2305.57, stdev=1196.23
cpu : usr=0.42%, sys=0.75%, ctx=65600, majf=65536, minf=52
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=65536/0, short=0/0
lat (usec): 250=0.01%, 500=55.83%, 750=26.56%, 1000=0.73%
lat (msec): 2=1.29%, 4=3.44%, 10=8.69%, 20=3.07%, 50=0.30%
lat (msec): 100=0.06%, 250=0.01%, 500=0.01%, >=2000=0.01%
queryB: (groupid=0, jobs=1): err= 0: pid=1654
read : io=262144KB, bw=509017KB/s, iops=127254, runt= 515msec
clat (usec): min=1, max=85, avg= 2.11, stdev= 0.70
bw (KB/s) : min=516816, max=516816, per=7375.71%, avg=516816.00, stdev= 0.00
cpu : usr=75.88%, sys=23.35%, ctx=3, majf=65536, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=65536/0, short=0/0
lat (usec): 2=1.24%, 4=98.02%, 10=0.68%, 20=0.04%, 50=0.02%
lat (usec): 100=0.01%
bgupdater: (groupid=0, jobs=1): err= 0: pid=1655
read : io=16652KB, bw=384687B/s, iops=93, runt= 44326msec
slat (usec): min=2, max=608, avg= 4.98, stdev=10.04
clat (usec): min=679, max=29214K, avg=59092.40, stdev=948086.69
bw (KB/s) : min= 0, max= 6674, per=36.42%, avg=2552.10, stdev=2142.30
write: io=16116KB, bw=372304B/s, iops=90, runt= 44326msec
slat (usec): min=2, max=1953, avg= 6.92, stdev=31.04
clat (msec): min=1, max=29225, avg=114.87, stdev=1692.76
bw (KB/s) : min= 13, max= 6019, per=50.36%, avg=2741.11, stdev=1953.27
cpu : usr=0.72%, sys=0.20%, ctx=3520, majf=0, minf=26
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=4163/4029, short=0/0
lat (usec): 750=0.01%
lat (msec): 2=0.51%, 4=46.39%, 10=45.81%, 20=2.84%, 50=2.97%
lat (msec): 100=0.22%, 250=0.05%, 500=0.78%, 2000=0.02%, >=2000=0.39%
seqrw: (groupid=0, jobs=1): err= 0: pid=1656
read : io=525120KB, bw=3566KB/s, iops=891, runt=147277msec
slat (usec): min=2, max=1456, avg=15.09, stdev= 6.86
clat (usec): min=1, max=10509K, avg=436.72, stdev=29037.71
bw (KB/s) : min= 1, max= 6080, per=70.50%, avg=4940.17, stdev=1434.18
write: io=523456KB, bw=3554KB/s, iops=888, runt=147277msec
slat (usec): min=3, max=190, avg=17.75, stdev= 4.63
clat (usec): min=196, max=29177K, avg=649.62, stdev=80759.46
bw (KB/s) : min= 1, max= 5928, per=90.14%, avg=4906.30, stdev=1465.57
cpu : usr=0.81%, sys=3.20%, ctx=262619, majf=0, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=131280/130864, short=0/0
lat (usec): 2=0.01%, 4=0.01%, 250=0.40%, 500=96.81%, 750=1.36%
lat (usec): 1000=0.18%
lat (msec): 2=0.43%, 4=0.65%, 10=0.13%, 20=0.01%, 50=0.01%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 2000=0.01%, >=2000=0.01%
Run status group 0 (all jobs):
READ: io=1041MB, aggrb=7007KB/s, minb=384KB/s, maxb=521233KB/s, mint=515msec, maxt=152138msec
WRITE: io=801716KB, aggrb=5443KB/s, minb=372KB/s, maxb=5626KB/s, mint=44326msec, maxt=147277msec
Disk stats (read/write):
sdb: ios=200904/202511, merge=14/260173, ticks=453720/34716300, in_queue=35169810, util=99.77%
And this is the output from centos (ubutnu and proxmox had very similar outputs):
bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
seqrw: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio 1.50
Starting 5 processes
bgwriter: (groupid=0, jobs=1): err= 0: pid=4075
write: io=262144KB, bw=42674KB/s, iops=10668 , runt= 6143msec
slat (usec): min=3 , max=290 , avg=11.01, stdev=10.80
clat (usec): min=374 , max=191014 , avg=2986.16, stdev=6824.67
lat (usec): min=402 , max=191022 , avg=2997.48, stdev=6824.77
bw (KB/s) : min= 4800, max=69120, per=798.93%, avg=42695.08, stdev=25075.13
cpu : usr=1.99%, sys=13.01%, ctx=14183, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued r/w/d: total=0/65536/0, short=0/0/0
lat (usec): 500=0.11%, 750=1.15%, 1000=3.65%
lat (msec): 2=40.75%, 4=47.13%, 10=5.11%, 20=0.70%, 50=0.81%
lat (msec): 100=0.48%, 250=0.11%
queryA: (groupid=0, jobs=1): err= 0: pid=4076
read : io=262144KB, bw=1093.3KB/s, iops=273 , runt=239776msec
clat (usec): min=203 , max=91801 , avg=3594.85, stdev=3493.15
lat (usec): min=203 , max=91801 , avg=3595.08, stdev=3493.15
bw (KB/s) : min= 81, max= 3184, per=24.60%, avg=1093.80, stdev=207.64
cpu : usr=0.20%, sys=0.66%, ctx=131171, majf=65536, minf=110
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=65536/0/0, short=0/0/0
lat (usec): 250=0.02%, 500=15.54%, 750=6.77%, 1000=21.48%
lat (msec): 2=4.34%, 4=12.80%, 10=37.07%, 20=1.83%, 50=0.13%
lat (msec): 100=0.02%
queryB: (groupid=0, jobs=1): err= 0: pid=4077
read : io=262144KB, bw=1102.8KB/s, iops=275 , runt=237713msec
clat (usec): min=208 , max=88477 , avg=3561.44, stdev=3484.17
lat (usec): min=208 , max=88477 , avg=3561.67, stdev=3484.17
bw (KB/s) : min= 88, max= 2286, per=24.81%, avg=1103.18, stdev=181.68
cpu : usr=0.24%, sys=0.63%, ctx=131182, majf=65536, minf=85
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=65536/0/0, short=0/0/0
lat (usec): 250=0.02%, 500=15.79%, 750=6.85%, 1000=21.46%
lat (msec): 2=4.53%, 4=12.78%, 10=36.59%, 20=1.85%, 50=0.12%
lat (msec): 100=0.02%
bgupdater: (groupid=0, jobs=1): err= 0: pid=4078
read : io=16372KB, bw=5352.8KB/s, iops=1338 , runt= 3059msec
slat (usec): min=4 , max=187 , avg=20.94, stdev=17.59
clat (usec): min=181 , max=34167 , avg=3395.42, stdev=2714.34
lat (usec): min=226 , max=34184 , avg=3416.86, stdev=2714.11
bw (KB/s) : min= 1478, max=10360, per=121.50%, avg=5403.00, stdev=3324.94
write: io=16396KB, bw=5359.1KB/s, iops=1339 , runt= 3059msec
slat (usec): min=4 , max=268 , avg=21.98, stdev=17.47
clat (usec): min=896 , max=199607 , avg=8286.59, stdev=17293.02
lat (usec): min=918 , max=199624 , avg=8308.98, stdev=17292.56
bw (KB/s) : min= 1538, max=11104, per=100.89%, avg=5391.67, stdev=3538.76
cpu : usr=1.93%, sys=6.64%, ctx=10500, majf=0, minf=27
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=4093/4099/0, short=0/0/0
lat (usec): 250=0.04%, 500=0.34%, 750=0.51%, 1000=0.57%
lat (msec): 2=12.87%, 4=58.80%, 10=19.69%, 20=2.76%, 50=2.45%
lat (msec): 100=1.57%, 250=0.39%
seqrw: (groupid=0, jobs=1): err= 0: pid=4079
read : io=525836KB, bw=3507.4KB/s, iops=876 , runt=149924msec
slat (usec): min=4 , max=269 , avg=18.24, stdev= 4.92
clat (usec): min=138 , max=31222 , avg=564.14, stdev=514.73
lat (usec): min=164 , max=31234 , avg=582.82, stdev=514.70
bw (KB/s) : min= 107, max= 4112, per=79.50%, avg=3535.30, stdev=629.70
write: io=522740KB, bw=3486.8KB/s, iops=871 , runt=149924msec
slat (usec): min=6 , max=1986 , avg=19.96, stdev= 7.48
clat (usec): min=1 , max=1340.6K, avg=535.53, stdev=3810.97
lat (usec): min=198 , max=1340.6K, avg=555.92, stdev=3810.93
bw (KB/s) : min= 173, max= 4304, per=65.73%, avg=3512.70, stdev=632.68
cpu : usr=0.97%, sys=3.63%, ctx=262286, majf=0, minf=35
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=131459/130685/0, short=0/0/0
lat (usec): 2=0.01%, 250=17.31%, 500=25.33%, 750=51.15%, 1000=4.43%
lat (msec): 2=0.97%, 4=0.47%, 10=0.32%, 20=0.02%, 50=0.01%
lat (msec): 100=0.01%, 250=0.01%, 2000=0.01%
Run status group 0 (all jobs):
READ: io=1041.6MB, aggrb=4447KB/s, minb=1119KB/s, maxb=5480KB/s, mint=3059msec, maxt=239776msec
WRITE: io=801280KB, aggrb=5344KB/s, minb=3570KB/s, maxb=43697KB/s, mint=3059msec, maxt=149924msec
Disk stats (read/write):
dm-4: ios=266493/200326, merge=0/0, ticks=547712/291404, in_queue=838851, util=99.80%, aggrios=66656/50081, aggrmerge=0/0, aggrticks=136765/72574, aggrin_queue=209340, aggrutil=57.76%
sdf: ios=68152/48586, merge=0/0, ticks=147766/75317, in_queue=223083, util=57.76%
sdh: ios=66294/50444, merge=0/0, ticks=135576/74153, in_queue=209729, util=55.41%
sdg: ios=66649/50088, merge=0/0, ticks=133185/67635, in_queue=200821, util=54.20%
sdi: ios=65529/51208, merge=0/0, ticks=130534/73194, in_queue=203727, util=53.77%
I do understand that I'm comparing a certified setup vs a non-certified one, but going from saturating two NICs to not reaching to sature one?
Does anybody has any experience in a setup like this one?
Please excuse my bad English and ask away any other info you may consider relevant.
Thanks in advance.
0 events found