EqualLogic PS6100XV SAN Compatibility with Cisco 3650X

Nothing in the Cisco 3000 line is really suitable for iSCSI as I understand it. The issue is small output buffers that cause frames to get dropped during microbursts, causing TCP retransmits which really hurt performance. The 3000 series are really access switches. To stay with Cisco, you need o go to the 4000 series such as the 4900M or 4948E. Dell says they have some small Force 10 switches that should work well.

That's not a tested switch.  The 3750E/X has some history of working fine in smaller environments.  The 3650X has 1/2 memory of 3750.   For very small environments I suspect it will work OK.  If I had to choose I would use 3750X.  

I would make sure you have latest firmware for switch (and EQL).  That's very important.  There are some fixes for flowcontrol in later firmware builds.

Make sure that flowcontrol receive desired and spanning tree pvst are set on all server and EQL ports.   Stack the switches offers better performance. Don't use the default VLAN and set jumbo frames.

This solution will help better align the buffers and help reduce retransmissions.

Solution Title PERF: Optimizing Cisco 3750x buffers for iSCSI performance

Solution Details Important: Cisco 3750X switches should not be used for a SAN solution that will expand beyond 3-4 arrays, or involves large block or sequential data IO. If there are more than 4 arrays consider the Cisco 4948 or a Nexus 5548 instead.

In order to provide optimal iSCSI performance, configure QoS and optimize buffers for EQL iSCSI use. A "queue-set" configuration is applied on all the switches and in a stack, there is no additional fine tuning required. Generally, queue-set 1 should ONLY be dedicated to iSCSI traffic.

Here is the config we currently recommend:

switch(config)#mls qos queue-set output 1 buffers 4 88 4 4

switch(config)#mls qos queue-set output 1 threshold 1 100 100 100 400

switch(config)#mls qos queue-set output 1 threshold 2 3200 100 10 3200

switch(config)#mls qos queue-set output 1 threshold 3 100 100 100 400

switch(config)#mls qos queue-set output 1 threshold 4 100 100 100 400

Configuring a 3200 threshold value will allow the maximum memory allocation of 3200% for iscsi packets.

After tuning the buffers as recommended above, packet drops may continue to be seen under certain traffic conditions.

The changes made to ‘queue-set 1’ may result in a blockage of TX queues on some ports and halt communication on them especially if Jumbo Frames are in use. This becomes a platform restriction; the 3750x offers line rate performance and hence it does not offer a good amount of buffer space to handle bursty traffic. iSCSI traffic is usually bursty in nature and may demand large buffer resources to be used to prevent drops on the switch interfaces. If available buffers run out, we may continue to see packet drops as a result.

It is also important to understand that pause frames cause the interface to hold traffic in their transmit queue, which eventually chokes them and an error is thrown that you received in the logs while making qos queue-set configuration changes.

Consideration should be made that the initial queue-set configuration needs to be fine tuned and there are revised buffer tweaks that can be implemented. Instead of using a Threshold value of 3200, keep it under 2000 as noted below:

mls qos queue-set output 1 threshold 1 2000 2000 1 2000

mls qos queue-set output 1 threshold 2 2000 2000 100 2000

mls qos queue-set output 1 threshold 3 2000 2000 1 2000

mls qos queue-set output 1 threshold 4 2000 2000 1 2000

mls qos queue-set output 1 buffers 10 70 10 10

If dropped packets continue to be seen with the new tuning recommendation and there is no voice/video traffic or any traffic passing through this switch stack that needs special treatment, reconsider disabling QoS on the switch completely or try to equally load-share each switch in the stack. Keep in mind that the switch’s backplane capacity won’t be more than 64Gbps at any given point. More switches in the stack means more interfaces in use and possibility of stack-ring to over-subscribe.

Regards,

Don

I just ran into the same question but unfortunately the switch was already purchased.

However, the mls settings you recommend only work on the 3750x. The 3650 seems to utilize auto qos instead. I am planning to run the "auto qos trust" but don't know whether I should select cos or dscp. Any knowledge?

Hello smeek,

Sorry no. Have you spoken with Cisco support about it?

 In a small environment you won't likely run into trouble.  Make sure that flowcontrol is set to desired, all the ports used in the SAN, (array and servers) are set to either portfast or PVST.  On all switches I prefer to create a new VLAN for iSCSI and not use the default.  On some switches the default VLAN won't properly pass jumbo frames.  To prevent worry about does this or that switch support it, I make that part of my SOP.   I would enable jumbo frames as well.  

Lastly making sure that you have the latest firmware is important also.

Regards,

Don