Info - CX4/VNX Delayed ACK (ISCSI) Performance Issues

Hi Everyone,

If you are an EMC/VMware customer running ISCSI I would love to hear from you!  Please email me.  ben.dunning@emc.com

I wanted to bring everyones attention to situation a customer will see when deploying ISCSI on a CX4/VNX array.  This applies to all Host OS's (MS/Vmware/Other)

Below is an explanation of the behavior as well as relative VMware and EMC support articles.

Symptom: With delayed Ack enabled customers may experience a noticeable  difference in ISCSI performance.

So whats the root cause?

                We use Broadcom hardware for our ISCSI SLICS (ISCSI Target.)  There is an incompatability between the driver we use in VNX OE and the Broadcom hardware.

In vSphere 4.0 and above can disable delayed ack on a target by target or vmnic basis.  What this means is that vMotion, FT or VM networks will not be affected.  In the 3.x days this was an ESX level setting and affected all interfaces which is lets just say bad. 

How do you disable Delayed ACK?

            Disabling Delayed Ack in ESX 4.0

1. Log in to the vSphere Client and select the host.
2. Navigate to the the Configuration tab.
3. Select Storage Adapters.
4. Select the iSCSI vmhba to be modified.
5. Click Properties.
6. Modify the delayed Ack setting using the option that best matches your site's needs, as follows:

• Modify the delayed Ack setting on a discovery address (recommended).

1. On a discovery address, select the Dynamic Discovery tab.
2. Select the Server Address tab.
3. Click Settings.
4. Click Advanced.

• Modify the delayed Ack setting on a specific target.

1. Select the Static Discovery tab.
2. Select the target.
3. Click Settings.
4. Click Advanced.

• Modify the delayed Ack setting globally.

1. Select the General tab.
2. Click Advanced.
3. In the Advanced Settings dialog box, scroll down to the delayed Ack setting.
4. Uncheck Inherit From parent.
5. Uncheck DelayedAck.
6. Reboot the ESX host.

What is EMC doing to remediate the issues.?

1. EMC will be issuing an ETA (EMC      Technical Advisory) which will go to all EMC customers and Partners      explaining the behavior and the work around.
2. Engineering will be working on a      hotfix which to fix this issue. (Stay tuned for timing.)
3. Until the hotfix is available      customer should disable delayed ACK.

Restoring EMC VNX iSCSI Performance by Disabling DelayedACK

Problem:

The EMC VNX array performed significantly below par in simple task of creating a VM. The test showed an unusual stuttering performance pattern at 100ms gaps that appeared to be holding the array in check.

Resolution:

Disabling the DelayedACK parameter on the ESX hosts removed any stops in communication and VNX performed significantly above par, completing the VM creation comparatively instantly.

Observed TCP Traffic Flow Control Bottleneck:

The root of optimizing TCP networks to minimizecongestion due to TCP/IP overhead rests on Nagle’s Algorithm (http://en.wikipedia.org/wiki/Nagle's_algorithm) where small packets (first observed with Telnet sessions) could have 1 byte of data (key-press)wrapped with 40 byte header. To avoid congestion, small packets are aggregated until 1) TCP segment is full, or 2) host times out – 100ms.

Today’s IP Storage Arrays are much faster and 10GbE provides a fast network path. At the same time, iSCSI storage 1) requires fastest response time possible, and 2) iSCSI payloads can be very small (especially control commands).

If iSCSI is triggered into Congestion Mode, forexample from high network traffic or a burst of commands from the storage array, a delayed ACK could wait indefinitely to receive a full TCP segment or at least until the host times out and issues a delayed ACK. This degraded condition would restore to normal when negotiating devices restore normal traffic mode.

Microsoft observed this degraded iSCSI performance (Article ID: 2020559), even with no network congestion, and recommends shrinking delayed ACK to restore performance:

“In situations where there is no network congestion, the default 200 millisecond delay on the acknowledgement can significantly impact SCSI command latency in an iSCSI environment.  Use of multipathing solutions which load balancing read requests across multiple array ports, increases the likelihood that those SCSI OpCode read requests (Read Lun, ReadLunCapacity) completions from multiple ports will result in network congestion.”

Consequently, a dedicated storage network may have no need for delayed acknowledgement and an iSCSI storage target may shrink or disable the congestion delay window, paradoxically increasing performance by minimizing traffic delays introduced by congestion control.

Recommendation:

?         VMware KB Article 1002598 advises that beginning with ESX 4.0 the recommended method for configuring DelayedACK setting is on a per individual target basis. Targeted control, of course, no longer impacts the entire TCP/IP stack of the network and therefore permits isolation and better performance management of iSCSI and Applications on that network.

?         An EMC Primus article is being drafted documenting optimal performance with dedicated NIC to SP-port block storage paths (no degradation irrespective of DelayedACK setting), and recommend disabling DelayedACK when not dedicated (multiple ports on same subnet, multipath to same target).

These configuration settings do not gate system communications and enable EMC VNX array to run at their native performance.