iSCSI TCP Tuning - what are your best practices?

Question

Google and these forums are full of tcp optimizing tips (i'm talking mainly windows here) for iSCSI connections. It seems to be more of importance on 10Gbit networks these days. However, i'm also reading alot of contradicting information, mainly because people mix and match networking terms like nagle, delayed acks, window scaling and some statements describing these settings are not necesary on some firmware levels etc. etc.

I was wondering which settings you use and have experience with. Which settings do good to your setup?

I use the only following:

- disabling delayed ack (TCPAckFrequency = 1) setting for iSCSI adapters in windows 2003 and up using generic nics for iSCSI traffic.

I'm reading up about a new setting in Windows 2008 for the iSCSI initiator called "iSCSIDisableNagle". It's in the EMC kb docs currently.

I cant figure out if this replaces the disabling of the delayed acks or if it's additional. I have not been able to test it either in my setup.

Anyone else got that one tested?

Looking forward to read about your tuning best practices. Please describe your experiences in performance gain if you can.

kelleg · Answer

There are a couple of places where we attempt to describe EMC's best practces for iSCSI - Support Solution emc245445 contains all the support articles and the following document on PowerLink is useful when using the 10Gb iSCSI interface.

"Clariion iSCSI Data and Control Traffic in a routed IP Storage Netwrok Environment.pdf"

glen

Clustor · Answer

ok thanks Glen. I still hope some people here can share their thoughts on the settings and their experiences with them.

kelleg · Answer

FC is newer and was designed for what we use it for - storage devices. iSCSI was glommed onto existing TCP/IP that's been around for decades and was never thought be used in this manner. Delayed ack was used for old terminals to minimize the overhead sending each and every charcater to save precious resources (bandwidth). Now everyone wants high bandwidth for storage performance and those old hacks to make a slow connection run more efficiently are a roadblock (IMHO). Until very recently you couldn't even disable delayed ack on ESX as it was running the BSD TCP stack from 1994 and it's still a pain on some versions of Linux.

glen

(Not the viewpoint of anyone but me)

Clustor · Answer

i think iSCSI is great, but i read up on more and more tunables on the host side that seem to make quite a difference in performance. This makes it a little more complex, although it should be a simpler in my opinion! FC on a host is more like a set-and-forget weapon in my experience. iSCSI needs more tuning to give reasonable results in alot of scenarios.

I'll try to repost this in the Clariion forums.

kelleg · Answer

I hope so also, there are some very knowledgeable people on these forums (also the CLARiioN forum) that I've seen alot of discussions about using iSCSI. The tricky part that seems to come up a lot are the different switches and how best to configure them. Seems if when using 10Gb that you need to consider flow control on the switches, routing issues, separating iSCSI traffic from LAN traffic, blades and their switches, etc.

Have you looked in the CLARiiON forum? There might be more people watching that than the VNX or maybe repost you question in that forum?

glen

Clustor · Answer

i agree.

so we're looking for a way to make the TCP/IP stack as SCSI friendly as possible.

I don't know if that means (as an example) disabling delayed ack on a host per definition.

What host-based optimizations actually have a positive effect on iSCSI? Window scaling, RSS, auto tuning features of Windows 2008... you name it.

EMC KB has an article about the delayed ack performance problem, which suddenly is "fixed" in the latest flare 30 patch .509 and later. (emc275481)

i'm still sceptical ....

kelleg · Answer

That article was for a very specil issue found on the 10Gb iSCSI on VNX, it does not affect the 1Gb or the 10Gb on the CX4. From what I can gather about this issuel, even after the fix, you should still dsable delayed ack, but I will check with engineering on this particular article.

glen

gprdawson · Answer

Caveat - we have a FC set-up; I've jused TCP/IP for other stuff over the years but not iSCSI.

Delayed ack (http://en.wikipedia.org/wiki/TCP_delayed_acknowledgment) and Nagle (http://en.wikipedia.org/wiki/Nagle%27s_algorithm) are two similar but different TCP/IP optomisations which, when working 'together' can have an unfortunate performance impact (http://www.stuartcheshire.org/papers/NagleDelayedAck/).

Delayed ack involves the TCP/IP stack trying to double guess that the application (iSCSI) is doing and having the receiver delay the sending of an ack in the hope that sender will send more data and one ack can be sent to acknowledge all the data. In a low bandwidth network this saves on the bandwidth as fewer ack packets are sent. In 1Gb or 10Gb ethernet networks there's usually enough bandwidth to render the saving worthless. The delays on the other hand...

Nagle's algorithm involves the sender grouping 'small' amounts of data smaller than a packet and sending it all in a packet. This delays the sending of the data but uses less packets with less overhead. Again, in 1Gb or 10Gb ethernet networks we can afford the over head.

With delayed ack and Nagle turned on we have the receiver delaying acks in the hope that more data will be sent and the sender delaying sending in the hope that a packet can be filled. Both ends are waiting for the other to do something!

Personally, I'd turn off both delayed ack and Nagle. I'd have the systems send the data and accept the small overhead of extra ack packets which, given jumbo frames would not be an issue.

It's possible that jumbo frames, Nagle and delayed ack could interact extra badly too. The jumbo frame is much larger than a standard ethernet frame so there might be occasions when Nagle causes jumbo frames to be held back more often that standard (1492 byte) frames.

VNX

iSCSI TCP Tuning - what are your best practices?

Was this post helpful?