Unsolved
This post is more than 5 years old
16 Posts
0
8850
VMware iSCSI TOE using hardware HBA's Broadcom NICs with MD3200i - Performance Problem with 64KB Reads
When i first heard that with vSphere 4 the Broadcom TOE NICs were available for hardware iSCSI configuration i thought great. After initial trial i first learned that you can not use Jumbo frames and HBAs together. If your used to using Jumbo Frames with your VMware software iSCSI initiator that will not work with Boradcom HBAs as the MTU on them is 1500 as can be seen in the service console using esxcli and can not be changed (not that i know of anyways). This however is not a problem as performance was still far batter with HBAs as per VMware iSCSI SAN Configuration Guide. This I found interesting having results that i have seen.
After setting up a cluster i run SQLIO ( and then IOMeter to confirm results) to find out something so strange that we thought had to be a problem with the SAN! All tests showed great trhoughput except one! 64KB Sequential Reads showed 1.3MB/sec with 1895ms latency!! Mind you this is a test suppose to show BEST results being not only Reads and large block size of 64KB but also Sequential! Other same build show 128MB/sec with 3ms latency on same test! Whats even more crazy is that 64KB writes Sequential were doing normal (100MB/sec) and also 64KB RANDOM reads were doing normal.
Out of all the tests ONLY 64KB Sequential Reads showed this result. I tried everything including swaping hosts, VMs, multiple tests, and the results remained. because the same size and frame tests (64K write) worked fine we assumed it had to be the SAN and nothing to do with transport as it would affect the writes! Even esxtop showed the latency to be in the DEV queue (also in kernel but nowhere as much as DEV which is normal once DEV is very high).
WRONG! only after we replaced the Controllers and even the backplane we realized its the TOE! As soon as i switched back to the iSCSI software everything was fine. I realized this after i had the 64 Read test running and ran another from different VM which was reporting 1000s IOPs with 16K writes while supposedly the latency was 1893ms on the other VM. I realized that if SAN was overloaded that test wouldn't be showing off the charts performance! To prove it more i vMotioned the VM running the 64KB Read test over to the host running the secondary test to and as soon as i did the secondary test started dropping. Here i realized its in the transport!!! Why would it affect it from same host but not another if on same LUN!? here is the full test with problem test cut and pasted at very top:
6 threads reading for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 25.90
MBs/sec: 1.61
latency metrics:
Min_Latency(ms): 7
Avg_Latency(ms): 1848
Max_Latency(ms): 2434
--------------- FULL TEST BELOW----------------------
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 3616.12
MBs/sec: 28.25
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 12
Max_Latency(ms): 557
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 1 2 1 0 0 3 11 22 24 18 9 4 1 1 0 0 0 0 0 0
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1768.47
MBs/sec: 110.52
latency metrics:
Min_Latency(ms): 7
Avg_Latency(ms): 26
Max_Latency(ms): 101
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 10 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB random IOs
enabling multiple I/Os per thread with 32 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 4297.82
MBs/sec: 33.57
latency metrics:
Min_Latency(ms): 5
Avg_Latency(ms): 44
Max_Latency(ms): 156
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 96
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB random IOs
enabling multiple I/Os per thread with 32 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1771.39
MBs/sec: 110.71
latency metrics:
Min_Latency(ms): 12
Avg_Latency(ms): 107
Max_Latency(ms): 262
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 10 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB random IOs
enabling multiple I/Os per thread with 40 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 4292.32
MBs/sec: 33.53
latency metrics:
Min_Latency(ms): 3
Avg_Latency(ms): 55
Max_Latency(ms): 117
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB random IOs
enabling multiple I/Os per thread with 40 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1728.11
MBs/sec: 108.00
latency metrics:
Min_Latency(ms): 9
Avg_Latency(ms): 138
Max_Latency(ms): 2416
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 7094.55
MBs/sec: 55.42
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 6
Max_Latency(ms): 23
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 2 18 42 31 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1772.75
MBs/sec: 110.79
latency metrics:
Min_Latency(ms): 5
Avg_Latency(ms): 26
Max_Latency(ms): 46
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 10 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 32 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 7042.03
MBs/sec: 55.01
latency metrics:
Min_Latency(ms): 5
Avg_Latency(ms): 26
Max_Latency(ms): 42
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 95
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 32 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1773.47
MBs/sec: 110.84
latency metrics:
Min_Latency(ms): 15
Avg_Latency(ms): 107
Max_Latency(ms): 179
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 10 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 40 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 7077.37
MBs/sec: 55.29
latency metrics:
Min_Latency(ms): 4
Avg_Latency(ms): 33
Max_Latency(ms): 50
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads writing for 120 secs to files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 40 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1772.35
MBs/sec: 110.77
latency metrics:
Min_Latency(ms): 10
Avg_Latency(ms): 134
Max_Latency(ms): 201
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 2497.36
MBs/sec: 19.51
latency metrics:
Min_Latency(ms): 3
Avg_Latency(ms): 18
Max_Latency(ms): 90
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 1 1 3 5 6 8 9 9 9 8 7 5 5 4 3 17
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 999.95
MBs/sec: 62.49
latency metrics:
Min_Latency(ms): 10
Avg_Latency(ms): 47
Max_Latency(ms): 1114
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB random IOs
enabling multiple I/Os per thread with 32 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 492.86
MBs/sec: 30.80
latency metrics:
Min_Latency(ms): 12
Avg_Latency(ms): 388
Max_Latency(ms): 3517
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 10 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB random IOs
enabling multiple I/Os per thread with 40 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 2534.43
MBs/sec: 19.80
latency metrics:
Min_Latency(ms): 4
Avg_Latency(ms): 93
Max_Latency(ms): 161
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB random IOs
enabling multiple I/Os per thread with 40 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 769.28
MBs/sec: 48.08
latency metrics:
Min_Latency(ms): 19
Avg_Latency(ms): 311
Max_Latency(ms): 1964
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 8633.18
MBs/sec: 67.44
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 5
Max_Latency(ms): 26
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 1 20 52 26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 25.90
MBs/sec: 1.61
latency metrics:
Min_Latency(ms): 7
Avg_Latency(ms): 1848
Max_Latency(ms): 2434
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 10 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 32 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 8671.03
MBs/sec: 67.74
latency metrics:
Min_Latency(ms): 4
Avg_Latency(ms): 21
Max_Latency(ms): 31
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 4 9 26 39 16 4
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 32 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 26.22
MBs/sec: 1.63
latency metrics:
Min_Latency(ms): 9
Avg_Latency(ms): 7235
Max_Latency(ms): 8570
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 99
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 10 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 40 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 8641.11
MBs/sec: 67.50
latency metrics:
Min_Latency(ms): 2
Avg_Latency(ms): 27
Max_Latency(ms): 43
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 98
sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file D:\SQLIO_testfile0.dat with 2 threads (0-1) using mask 0x0 (0)
file E:\SQLIO_testfile0.dat with 2 threads (2-3) using mask 0x0 (0)
file L:\SQLIO_testfile0.dat with 2 threads (4-5) using mask 0x0 (0)
6 threads reading for 120 secs from files D:\SQLIO_testfile0.dat, E:\SQLIO_testfile0.dat and L:\SQLIO_testfile0.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 40 outstanding
using specified size: 10240 MB for file: D:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: E:\SQLIO_testfile0.dat
using specified size: 10240 MB for file: L:\SQLIO_testfile0.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 26.29
MBs/sec: 1.64
latency metrics:
Min_Latency(ms): 13
Avg_Latency(ms): 8949
Max_Latency(ms): 13804
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
jabuzz
1 Rookie
1 Rookie
•
16 Posts
0
April 8th, 2011 05:00
Been there have the tee shirt. There is a fundamental problem with the MD3200i series storage arrays that leads to poor read performance. After considerable effort and trying everything under the sun from different network cards, full hardware iSCSI HBA's, full software iSCSI, hardware accelerated iSCSI, and different switches we finally identified the problem.
Basically the storage array is overwhelming your NIC's. This never manifests itself under write as the NIC's on the host only send out what they can. However on read, under a range of configurations including the Dell recommended configuration you get more than one port on the MD3200 overwhelming a single port on the host. At this point either you have flowcontrol on your switch and depending on how good your switch is the performance falls back horribly. In this regard the PowerConnect 5548 is a load of rubbish. If you don't have flow control then the TCP stack backs off and performance drops again. Either way you appear to be screwed.
The only way out of this for us was to have one dedicated ethernet port on the host for each iscsi ethernet port on the controller. The other possible solution is to use 10Gb on the hosts, though this might require the use of QOS to stop the host flooding the controllers and creating the problem in reverse. Having used LSI Engenio storage for many years in the guise of IBM's DS3000/DS4000 series arrays using Fibre Channel we have never seen anything like this and it puts us right off iscsi. If only IBM had not been trying to sell us XIV and told us about their DS3500 (basically a MD3200 in different clothes) we would have saved ourselves months of hassle.
ArmaniSeal
16 Posts
0
April 8th, 2011 07:00
All my clients follow just such design using single nic per vSwitch connected to different VLAN on different switch. Its the recommended configuration for the MD3200i however i still see the issue with HBAs but NOT with software iSCSI.
As for your statement "..overwhelming a single port on the host" i think your right there definitely right there. Looking at my test having 2 VMs on 2 hosts on same LUN running IOmeter and one is showing 64KB Reads Seq at 1MB/sec with latency of 1983ms and then I kick of the second test on other VM it shows 1000s of IOPs with 3ms latency! Well if SAN is overwhelmed with 1983ms latency from test one the second test running simultaneously would show even worse results not perfect.
Then then I vMotion the VM to the same host as the one running test one (64KB read) all of the sudden the second test just drops to same high latency! Meaning it is on the NIC. Especially since even when I have them on separate hosts they are still pointing not only to same LUN but same port and controller on the MD3200i. And network traffic shows nothing at this time mind you.
JOHNADCO
847 Posts
0
April 8th, 2011 12:00
We did testing with the Broadcom "iSCSI offload" and it stunk at every turn for us. Disabled it, went back to the Vmware iSCSI initiators and never had a problem since. This was / is with MD3000i's though, which do not have the read oddiies being experienced by some of the newer PV arrays.