Skip to main content
Start a Conversation

Unsolved

This post is more than 5 years old

huberw

4 Posts

73555

March 17th, 2011 07:00

vSphere 4.1 and Broadcom 57711 cards attached to EQL PS6010XV

Hi,

We are deploying 3x Equallogic PS6010XV arrays with 13x M610 blade servers, each configured with 2x dual port 57711 10GbE mezz cards.

As I understand it, VMware's driver for the Broadcom 57711 cards does not support jumbo frames when utilizing the hardware iSCSI offload function in ESX 4.1. They DO however support jumbo frames for those cards if you choose to use the software iSCSI initiator.

My question is this. Which is better? Hardware iSCSI offload and standard 1500 MTU, or the stock software iSCSI initiator with MTU 9000?

My experience in the past is that jumbo frames with the software initiator didn't really do much for you in terms of performance, and that it was typically not worth the trouble of enabling it, since you can get very good throughput with standard size frames. That was specifically with EMC storage, I'm not sure if Equallogic is different.

Thanks,

Will

JOHNADCO

847 Posts

March 17th, 2011 09:00

No direct experience, although looking at 10gbe ourselves. As I understand it Off-load is really important on 10GBE as to sustain that sort of throughput, the proc would be slammed. I can't imagine jumbo would be so needed at those tX / RX speeds.

Donald_Williams

72 Posts

March 17th, 2011 10:00

There's allot of discussion about HW or not HW. Here's a good article from the VMware forum. in this customers testing Jumbo Frames made a bigger difference.

http://www.vmadmin.co.uk/vmware/35-esxserver/252-esxihwswiscsijumbo

Also with the HW ISCSI HBA you are limited to the number of iSCSI targets you can connect to, 64. (2 are reserved for Boot-From-SAN)

if you do decide to use HW iSCSI offload the minimum FW for the broadcoms is 5.2.7. Without that you will have problems with Dell/EQL arrays. AFAIK, there's no way within ESX to upgrade that FW. Just updating the drivers is not sufficient.

You'll also need to be at the latest ESX revision. That has updated Broadcom drivers for 1GbE and 10GbE NICs.

-don

JOHNADCO

847 Posts

March 17th, 2011 10:00

My point was? On 10GBE it should be so fast with standard 1500 sized frames that it (jumbo frames) shouldn't matter.
And 10GBE suposedly really hits the proc using SW initiators.

This testing linked to seems only to have been done on 1GB connections.

huberw

4 Posts

March 17th, 2011 10:00

I was thinking along the same lines. I'm going to set it up with the HW offload and run some iometer tests to validate that we are getting the desired throughput. Time permitting, I'll setup a different host using the sw initiator and run the same tests to see which returns the better result. I'll post my findings here.

thanks for the reply.

huberw

4 Posts

March 17th, 2011 12:00

donald,

very interesting read. thanks so much for linking me to that... i was wondering if anyone had done the test.

i agree i want to do the same test on 10GbE. I also believe that the high latencies he was experiencing was possibly more due to a driver issue with the broadcom NICs in ESX than jumbo vs. standard sized frames. I would have liked to see the same test with the software initiator and standard sized packets - I bet that the results would have been close to the software initiator with jumbo frames.

at vmworld last year chad sakac and vaughn stewart did an iSCSI best practices session, and in that session they said most times enabling jumbo frames isn't worth the small increase in performance that you MIGHT get by doing it. the iSCSI stack is pretty darn efficient with standard size frames.

I'll try to do my own testing and will write up the results here when i'm done.

thanks again (to you both) for your feedback.

Donald_Williams

72 Posts

March 17th, 2011 12:00

You would think so, that Jumbo vs. standard. However, that does mean that the processor has to deal with allot more packets. Also, JF helps with network efficiency when doing things like iSCSI or FCoE. Lot's of data to more at one time. Typically non storage loads won't benefit as much. As long as you can fill the larger packet JF works well.

What I can tell is what customers report back. On switches that can handle Jumbo Frames and Flowcontrol with 10GbE, JF helped with performance on iSCSI. As that one forum poster discovered.

-don

JOHNADCO

847 Posts

March 17th, 2011 14:00

Hard to analyze... Many people also find the best frame size is something in between 1500 and 9000 too.

Donald_Williams

72 Posts

March 17th, 2011 15:00

On GbE, I totally agree that the difference can be small. Typically only a few percent. 10GbE has been different so far in my experience. Not to say that when updates to the stack come along that might change. it's one of those things you can try for yourself w/o too much harm. Especially with ESX. Change just one node to JF and monitor the results.

Regards,

-don

JOHNADCO

847 Posts

March 18th, 2011 08:00

I ask that we keep this thread open and updated. I am seriously considering some 10GBE. We are going to 100% thin clienting and I would love to get all my Vmware hosts talking to each other over 10GBE.

enderwa

1 Message

March 30th, 2011 12:00

very much interested in feedback from those with 10GbE gear.

in our case: R910s with single dual port 57711 10GbE cards thru two PowerConnect8024F onwards to a PS6510x, flowcontrol on & Jumbo Frames enabled across the board on all bits, and using the ESXi iSCSI s/w initiator. iSCSI a totally separate private network.

initial config and testing all fine and dandy, once we stuck real machines on there (P2V), performance turned to sh!t - lots of head scratching and frustration.
symptoms: command aborts errors, shocking disk latency, poor VM performance (fast then slow cycling).

Once JF were disabled on the ESX servers (i.e. remove and reinstall the iSCSI vswitch (w/ mtu as 1500) using the EQL vSphere CLI scripts) performance was usable. More testing still to do...
Techie comments before and during design lead us to believe that the 10GbE JF configuration was the best optimal config to use with our high-end hardware.

I have to question if using the 57711 HBAs directly instead of via the ESXi s/w initiator will give even better performance seeing as JumboFrames are off the table. Would one expect significantly less CPU load on the Host as a result and far greater performance ? How do people best measure the ESXi hypervisor load on the physical Host?

shame I hadn't poked around this site sooner. it's very informative.

huberw

4 Posts

April 11th, 2011 11:00

Hi all -

sorry for the lack of updates.

I tested the following setup:

vSphere 4.1 server with standard frames and the broadcom offload - with both NMP round robin AND the MEM multipathing module
vSphere 4.1 server with jumbo frames and NO hardware offload (sw iSCSI initiator) - also with both native NMP and the MEM module

in our results the clear consistent winner was esx 4.1 using standard sized frames, the broadcom hardware offload, and the MEM module. It beat out the others by on average 1000 IOPS, lower latency by about 1%, and about 20 MBps higher throughput.

hopefully this helps others out that are trying to decide which way to go!

chimeranzl

3 Posts

June 7th, 2011 04:00

Not exactly part of the initial question, but I see a few have mentioned poor performance with software iSCSI initiator and jumbo frames - ensure you're running the latest Broadcom driver for this NIC under ESX4.1. There is a latency issue with the factory 4.1 driver. See this post: http://communities.vmware.com/thread/284628?start=45

paul_adam123

7 Posts

September 26th, 2011 16:00

Just to add my two cents :) (not hijacking the thread, just sharing my experiences) - We had some pretty major fun and games with these broadcom cards going through to a PS6010XVS (our VDI Environment) and also a PS6510XV (our server environment) - where retransmits were through the roof and performance was ok one minute then the iSCSI connections would randomly drop (as in totally reset and re-establish - funnily enough our VMs didnt like this too much!) ended up being an issue with Delayed Ack's which we had to disable on the ESX hosts before things stabalised.

We had both VMware and EQL support engaged and ended up with each other blaming each other for their interpretations of TCP/IP and iSCSI RFC's being slightly different, but they did both acknowledge it as an issue? - What got us was that so few people seem to have seen the issue before and why did we suffer so badly, anyway....

We are running with jumbo frames enabled currently, but our VDI SAN is being crippled once more with poor throughput, so will probably open up another thread on it, I think we are running to many linked clones on one datastore though, so we may need to split things up into smaller chunks....just thought I would mention it

networksolution

13 Posts

August 21st, 2012 09:00

Hi guys I've been revisiting this issue myself for a while now and I have heard that the limit for the MTU is 1500 when using TOE with this Broadcom NIC. Is there any documentation from Broadcom that specifically states this? I'm just curious.

Anonymous

5 Practitioner

 • 

274.2K Posts

August 22nd, 2012 13:00

It's in the ESX documentation I believe. Either SAN Guide or release notes.   Since it's Vmware's driver.   The first version of ESX that I've seen with Jumbo Frame support with ISCSI offload is in ESXi v5.0 July build.   It has been an issue with the broadcoms for many years under Windows and Linux as well.   To get this support you have to have the current broadcom FW and drivers from VMware.  AFAIK, that driver is not available for ESX v4.1.

 However, it's been my experience that in actual use, there's not much difference.  Since most iSCSI SANs don't max out the bandwidth.  Especially when you have multiple servers going to a single SAN.   If all of the server NICs max out you would swamp the SAN device.

 It's about IOs/sec and latency.  Jumbo Frames reduce load on CPU, if you are already offloading iSCSI and network functions in the card,  JF there won't provide much improvement.   If you monitor you network usage now, that will probably be very clear.   You're not running at hundreds of MBs/sec.

With the SW iSCSI initiator, what I have found most important, is disabling Delayed ACK and Large Receive Offload.   That will improve performance very significantly.  Delayed ACK should also apply to the iSCSI offload cards, though I've not tried that myself.  

Also, make sure you have SANHQ installed.  That will provide excellent monitoring on the EQL SAN.

View More

View All

No Events found!

Top