Start a Conversation


This post is more than 5 years old


April 14th, 2016 19:00

flow control with iSCSI + Nexus switches

We used to configure "flowcontrol receive on" on the nexus ports connecting our arrays and hosts.... The latest version of the Host Configuration guide mentions that flow control should be disabled....

Just wanted to confirm what are the latest best practices.... Also, it looks like support for DCB is not yet in place?

Thanks - Didier.

Below is an example of one of the host port as seen on the Nexus 5596 side

!Command: show running-config interface Ethernet186/1/23

!Time: Thu Apr 14 22:35:31 2016

version 6.0(2)N2(3)

interface Ethernet186/1/23

  description coe-xen607g_iscsi-a

  switchport access vlan 3897

  flowcontrol receive on

vlab-k10-n5k-3-top# show int eth186/1/23

Ethernet186/1/23 is up

  Hardware: 1000/10000 Ethernet, address: 3c5e.c3b1.0958 (bia 3c5e.c3b1.0958)

  Description: coe-xen607g_iscsi-a

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is access

  full-duplex, 10 Gb/s

  Beacon is turned off

  Input flow-control is on, output flow-control is on

  Switchport monitor is off

  EtherType is 0x8100

  Last link flapped 00:06:34

  Last clearing of "show interface" counters 32w1d

  40 interface resets

  30 seconds input rate 1068008 bits/sec, 140 packets/sec

  30 seconds output rate 534112 bits/sec, 95 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 1.75 Mbps, 330 pps; output rate 10.17 Mbps, 912 pps


    25432589144 unicast packets  75 multicast packets  16 broadcast packets

    25432589238 input packets  18728312514391 bytes

    0 jumbo packets  0 storm suppression bytes

    3 runts  0 giants  0 CRC  0 no buffer

    3 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause


    69662597238 unicast packets  22800361 multicast packets  3537905 broadcast packets

    69688936113 output packets  99317457830451 bytes

    54 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble 0 output discard

    20180816 Tx pause

727 Posts

April 19th, 2016 16:00

The reason we advise against flow control is that enabling flow control serializes the flow for iSCSI port; hence causing major performance impact. It does not make sense to buy XtremIO and then enable flow control which limits the performance.

29 Posts

April 20th, 2016 20:00

Thanks… Just wanted to double-check is that this is a departure from an older (early 2014 / code 2.x) recommendation ->

Enable flow control on all network switch ports(so on Cisco switches, this would be link level flow control) that handle iSCSI traffic as well as any host based adapters. If the host is using a software iSCSI initiator and NIC combination or HBA to handle iSCSI traffic, flow control must be enabled on the NICs and HBAs to obtain the performance benefit. Flow control ensures a receiver can make the sender pace its speed and is important in avoiding data loss.

Explanation: Typically in iSCSI SAN configurations, many servers(initiators) are communicating with the 10G storage ports on the XtremIO nodes. This can sometimes lead to an imbalance in the network traffic between the initiators that send network traffic and the ports that receive the traffic. If senders transmit data simultaneously, they may exceed the throughput capacity of the receiver. When this occurs, the receiver may drop packets, forcing senders to retransmit the data after a delay. Although this will not result in any loss of data, latency will increase because of the retransmissions of dropped packets, and I/O performance will degrade. Flow control allows the receiver to instruct the sender to pause transmission of additional data when the receiver senses that it is being overwhelmed. The receiver does this by sending pause frames to the sender, which causes the sender to stop packet transmission for a short period of time. This pause allows the receiver to process its backlog so it can later resume accepting input. According to Cisco, while latency is introduced into the SAN by flow control, it is much smaller when using flow control than when flow control is disabled and packets must be retransmitted.

No Events found!