Very slow READ performance with MD3200i

I'm reposting a thread here from another forum regarding extremely slow read performance with MD3200i in hope for some insights. I apologize for the length, but I'm hoping to be thorough.

Network Info:

In our network, there are two SANs, an MD3000i (15 disk 15k SAS) and a MD3200i (12 disk 10k SAS). They're both connected to the same two switches dedicated just to the SANs.

We have 4 physical servers, each with 2 dedicated Broadcom iSCSI TOE NICs. MPIO is enabled.

We're running Windows 2008 R2 with Hyper-V on the physical servers, and we have a bunch of VMs on the SANs.

The MD3200i was installed some months ago, and a few VMs were created on it - but the bulk of them still remain on the MD3000i. We are not using jumbo frames at this point.

The problem:

A few days ago I noticed very slow I/O performance on the VMs on the MD3200i. Read speeds were as slow as 2MB/s while copying files off the VMs stored on the MD3200i.

I started investigating so I made a new 1GB LUN on the array and attached it to one of the physical servers. Same ~2MB/s read speed, but the write speed seemed ok.

I took the following benchmark at that point:

Note how the read speed drops with 128KB block sizes but seems fine with 32KB blocks.

 

After reading further forums, I found that some people disabled MPIO and that fixed the problem for them. So I proceeded by disabling MPIO and reran the test:

The read speed is slightly improved without MPIO, but now the slowdown occurs at 32KB block size

 

Here is the exact same test in the same conditions using our other MD3000i array:

So I started from scratch:

I started by reconfiguring my iSCSI host ports from scratch in the controller:

 

Controller 0

Port 0: 192.168.133.110 (Switch 1)

Port 1: 192.168.131.110 (Switch 1)

Port 2: 192.168.132.110 (Switch 2)

Port 3: 192.168.130.110 (Switch 2)

 

Controller 1:

Port 0: 192.168.133.111 (Switch 1)

Port 1: 192.168.131.111 (Switch 1)

Port 2: 192.168.132.111 (Switch 2)

Port 3: 192.168.130.111 (Switch 2)

 

After reconfiguring the IPs however, even though I could ping them, the iSCSI initiator would throw errors and was unable to connect.

I don't have physical access to the SAN (it's colocated) to reboot it. So I had to use the SMCLI to reset both controllers on the SAN:

 

smcli.exe -n san2 -c "reset controller [0];"

smcli.exe -n san2 -c "reset controller [1];"

 

After both controllers were reset, the iSCSI Initiators were able to connect again fine.

Strangely, the read speed jumped up a bit:

However, now I can't seem to get it past 25 MB/s or so sequential read speed. The write speed has never been an issue.

 

I also ran an IOMeter test on LUNs from both arrays with the following results: (Max Throughput - 100% Read)

MD3200i - 699 IOPS / 21 MB/s

MD3000i - 5519 IOPS / 172 MB/s

 

Here are a few other resources showing more or less the same problem:

http://en.community.dell.com/support-forums/storage/f/1216/t/19351578.aspx

http://www.experts-exchange.com/Hardware/Servers/Q_26719712.html

http://www.passmark.com/forum/showthread.php?t=2773

And the thread that this post is based from: http://communities.vmware.com/thread/304340?start=0&tstart=0

Any ideas on how to get this fixed? We'd open a support ticket but from our experience, that won't help us much at all, so I'd rather ask here and see if someone has any ideas.