VNX 5300 Block iSCSI Slow??

Hi,

I have a VNX 5300 which has the following:

8 x 2TB SATA disks

16 x 15K 600GB disks

Vault Drives (Not being used for anything other than Vault)

The 8 x 2TB and 16 x 600GB are all grouped together in a RAID 6 Storage Pool.

I have 3 x VMware hosts which share 3 x 2TB Thin LUN's on this pool (All set to autotier) and connected via Fibre Channel.

I now have a requirement for a single 2008 R2 SP1 Hyper-V host. At the moment I have no spare Fibre cards available so decided to try out 1Gb iSCSI instead. I have a Dell R710 with 128GB of RAM, dual quad core CPU's, using 2 x Broadcom 5709C with the Microsoft iSCSI initiator and EMC PowerPath with MPIO and a 1TB Thin LUN on the VNX 5300. This is currently connected by a single HP Procurve 2910al. This was setup following the attachment guide from EMC. From Unisphere I can see four connections from the host, and the host is set to use Failover Mode 4 ALUA. I can remove a NIC and it all seems to work fine. The MPIO type is Vendor Specific (PowerPath).

The problem with this setup is that the performance is really bad...

I did a very basic test of creating an empty VM and installing Microsoft Server 2008 R2 onto it. Now the R710 itself has 4 local 15K disks in a RAID 10, this can do this task in around 8 minutes. If I do the same test using one of my VMware hosts on a LUN, it can also do this in around 7 minutes. If I use the iSCSI LUN with Hyper-V, it takes 25 minutes plus, and the performance degrades significantly if I start a second one etc...

Now as this is my first iSCSI connection I am trying to understand if this is normal or not?! I can not believe that it is? Looking in Task Manager each NIC never seems to go above 25% utilization. However the strange thing is if I perform a large copy from a server on the network to the VNX LUN (Say 3 x 3GB ISO files), the File Progress window shows a copy speed of 60-70-80 MB a second and the Task Manager NIC utilizations ramp up to 80-90%??

It does seem that the SAN can do more (Hence the VMware test, although I appreciate it is not a fair comparison due to the use of Fibre), but for some reason the connection is stopping me especially for lots of small IO? When looking at the PowerPath MMC I can see the IOPS ramp up to multiple hundreds too...

Things I have attempted so far...

Pushed the iSCSI LUN to the highest tier of storage

Enabled Jumbo Frames (9000) on both NIC's, the switch and set the iSCSI NIC's on the VNX to 9000 MTU

Enabled Flow Control on the switch on all ports used for this (4 for the VNX,and 2 for the server)

Disabled various functions on the NIC's such as TCP Offloading, Receive Side Scaling etc.

Creating the VM with a thick disk

Changed the Jumbo Frames to 4000 everywhere to see if there was fragmentation anywhere etc

Setting the TcpDelAckFrequency=1 registry value on both the NIC's and the EMC location as referenced in a primus article...

Setting the DisableNagle registry value

Disabling one of the iSCSI NIC's (This naturally doubles the utilization in task manager, but still never goes above 50%)...

I think the problem may be that I have nearly all the right things here to try, but perhaps keep getting bad combinations of them?!

Can anyone comment on this (Even if its just to say it does not sound right!!), in particular which settings should I be using/not using etc!