I forgot to include the Benchmark Images from the Test VM.

Here is the standard HDTunePro benchmark:

Here is the file writing benchmark:

Here is the Random Access Benchmark:

As you can see there is a high amount of variation in the benchmarks. The speeds are far from consistent and the access time fluctuates a great deal.

A few items to check first:

On the Switchs, ensure you have the latest version of the firmware installed, this should be v4.1.0.15 or later.  Try various combinations of the following: disable/enable the iSCSI optimization setting, enable/disable jumbo, enable/disable flow-control.  If I were to choose a starting point, it would be to disable the iSCSI optimization, and jumbo first, then test.

Broadcom NIC’s, you didn’t specify what model, so please list you model, we may have a setting to tweek (if not done already).

If we don't get anywhere here, and you still have bad performance, I would suggest you open a support case with EqualLogic so one of the performance engineers can help you isolated the cause.

Joe

The Broadcom NICs are 5709C NetXtreme IIs. I have disabled RSS, and TCP Chimney as recommended in another thread, but if those features should be re-enabled please let me know.

I've now done the following:

1. Turned off flow control - Same results

2. Turned off iSCSI Optomizations - Same results

3. Turned off flow control and iSCSI Optomiziations - Same results

The switch firmware is at 4.1.0.19, and I will be upgraded it to 4.1.1.9 tonight.

I'm in the midst of trying to find a physical server to connect to the SAN, but this may take a few days as I don't have anything set up at the moment.

I am also in the midst of benchmarking a directly attached LUN with IOMeter and SQLIO. This disk is being passed through to a Hyper-V VM bypassing the Host Opertating System's storage stack.

I will report back when I have more testing done.

After some more testing with directly attached LUNs, I consulted this document again http://www.equallogic.com/WorkArea/DownloadAsset.aspx?id=10771 and I think I may have resolved my problems.

As recommended:

• I have disabled autotune on the interfaces in my R710 with:

netsh int tcp set blobal autotuninglevel=disabled

• On the switches I've re-enabled iSCSI optomizations, flow control, and Port Fast on all interfaces.
• I've also renabled Jumbo Frame support on the Broadcom NICs, Switches, and SAN after disabling it for testing.

Before making the switch changes all three LUN types were performing horribly.

After making these changes; Disk-to-disk file copies, HDTunePro, and IOMeter synthetic benchmarks seem to be much faster on a Hyper-V VHD in the CSV, a Hyper-V VM with a pass-through LUN, and a Server with a Directly attached LUN. The directly attached LUNs are benchmarking faster than the Hyper-V VHDs, but I believe this difference is Hyper-V overhead. 

Based on these results I am inclined to believe that the switches were misconfigured. I'm assuming I missed a setting mentioned in the document the first time I configured everything. I'm still not certain if all of my settings are 100% correct, though. I want to ensure that this cluster has optimal performance so I can duplicate the settings on the next cluster I build. Should I be disabling anything in the Broadcom Control Suite? Are their any other global interface options that should be disabled via netsh? 

Lastly, I also have to do the firmware upgrade on the switches, so that may increase performance as well.

Thanks again for your help.

I've manually set Jumbo Frames on the interfaces, and manually assigned a group of 24 ports to a VLAN dedicated to iSCSI traffic. I've also forced the subnet in use for iSCSI to use only the VLAN assigned.

We also have a document on some recommended setting for the Broadcom NIC as well that you may want to review. I'll send it to you via the forum send mail feature.

Joe

Just some quick thoughts...

Does performing the same test from the Hyper-V host against the same CSV produce the same result?

Running the test on non-CSV - same result?

Are the VM's disks VHDs?  Or, are they direct-connect (using VM's iSCSI initiator to connect to Storage)?

If VHDs, have you tried the same 'test' with 'Fixed'-type VHDs?

I'd also be interested in the Broadcom NIC article. We've got NetXtreme II BCM5709s connected to a PS6100XS.

Thanks!

I have sent you the document via the forum "send mail".  I'll see if I can get this posted to a public location so that in the future I can just add the link to this forum thread.

- joe

Is there any chance I could get that document as well, I have been having some issue with out iSCSI

thanks

Blackduke77,

All set, I sent you the information via the forum email.

-joe

I would be interested if you are willing to perform some tests beginning at zero. which means a lot of work:

- put it ALL in default vlan, NO extra vlans

- disable jumbo frames at initiator AND at switch level, re-adjust the switch mtu to standard (very important when operating in default vlan)

- disable MPIO at initiator level, in fact, don´t use HIT or OS MPIO at all (just for this test)

- disable iscsi optimization on the switches

- enable flow control on the switches

- disable ANY broadcast storm control or arp spoofing protection on the switches

- do NOT use broadcom hardware acceleration or dedicated hardware iscsi initiator

- just use software iSCSI initiator

- check your EQL at advanced settings to see if somehow DCB was turned on - if yes, turn it off!

- give it LOAD. And as you do, check SANHQ for disk latency and also at disk level the queue depth

- also, in SANHQ check for NIC errors during load.

- you don´t have by any chance some Intel NICs at hand which you could swap against the broadcoms (just for this test)?

Best regards,

Joerg