Poort443,

Hi, I'm Joe with Dell EqualLogic. Flow control is a method by which a receiver of Gigabit Ethernet traffic can briefly "throttle back" the sender of the traffic to keep from get ting overwhelmed. To do this, the receiver sends a special Ethernet frame, called a "pause frame", back to the sender; the sender then stops sending data for a short period of time. This allows the receiver to process the packets it has buffered and to prepare for the next group.

If flow control is not available, and the receiving network device is not able to handle the rate of incoming traffic, then the only alternative it has is to start discarding incoming packets that it cannot process. These packets will not be permanently lost, because TCP/IP has mechanisms for detecting discarded or lost packets and retransmitting them. However, the detection and retransmission cycle takes much longer than flow control (several seconds as opposed to several milliseconds), so you are likely to see performance degradation if this happens frequently.

For this reason, EqualLogic strongly recommends that flow control be enabled on all switches, NICs and HBAs that carry iSCSI traffic.

Hope this answers your question.

Regards,
Joe

Hi Joe,

Thanks very much for your response. I think you just explained quite clearly what flow control should do, and it's what I always assumed myself. You do not however go into the comments made in the URL's I posted. I'll quote from the first:

" TCP has it’s own ability to manage flow and is completely unaware of of any layer 2 mechanisms. I have been told that TCP is basically designed to assume that there is no other flow control in place other than itself. This means that PAUSE frames on layer 2 could cause problems with TCP. TCP is designed to only slow down when *it* detects an overloaded situation. Since the layer 2 PAUSE frames are invisible to TCP it will continue to send more and more data assuming that the path is clear.

Also, Flow Control at Layer 2 doesn’t have built in priority control either, you are allowing it to pause ALL traffic instead of specific traffic that is flooding for an end point that is overloaded. (the next paragraph explains why).

Another bad/good thing (depending on your environment) is that Ethernet Flow Control is switch to switch only. Even though the mechanism uses mutlicast the frame is not forwarded on from the receiving switch. This means logically TCP is an end-to-end (Server to Client) Flow Control rather than a switch to switch(virtual or not). TCP is more granular and scalable which is a prereq to cloud environments. "

Could you look at these points (made by some very knowledgeable people) and respond to them? I'm definitely not a networking specialist myself, but they seem to have some good points.The point of their argument being that flow control does thing unknown to higher layers (TCP) and therefore has the potential to decrease overall performance.

I look forward to your thoughts.

Regards,
Poort.

Joe,

Thanks for your elaborate answer, you've at least reassured me that I'm still doing the right thing when enabling flow-control. I will test the next time I have the opportunity. Unfortunately no Equallogic in the lab :-) Thanks.

Poort.

Poort,

I think the information in the posts you referenced is very informative; however I don’t think they address the specific needs of the PS Array and iSCSI traffic. If a server is using a software iSCSI initiator and NIC combination to handle iSCSI traffic, you must also enable Flow Control on the NICs and switch ports connected to the array, to obtain any performance benefit. The EqualLogic PS Series storage arrays will correctly respond to Flow Control.

In a typical iSCSI SAN network there can be unbalanced network traffic between the devices that send network traffic and the devices that receive the traffic. For example, you have several servers (iSCSI Initiators) that are communicating with the Array storage devices (targets), and the senders transmit data simultaneously, they may exceed the throughput capacity of the receiver. When this occurs, the receiver could drop packets, forcing the senders to retransmit the data after a delay. Although this will not result in any loss of data, latency will increase because of the retransmissions, and I/O performance will degrade. Again, the pause frame takes less time then re-transmits.

In the statement flow control is switch to switch only, this is only true if the uplink has the capacity for flow control. The EqualLogic array has this capacity, as do most NIC’s and iSCSI HBA’s. Additionally, iSCSI uses unicast and the pause frame is passed from the target to the sender (the next uplink port, i.e., array to switch), conversely, if the switch gets flooded from the server it would tell the server to pause.

I guess if you have doubts if your environment is being degraded by flow control would be to test with flow control disable/enabled. You can use our free SANHQ tool to monitor your array to see how the two different setup’s effect the I/O and Latency in your SAN setup.

Please, let me know if you have any other questions.

Regards,
Joe

Hello,

Just to add some info to what Joe has already said. TCP congestion control is used on the array. The array will change window size or scaling to maintain the best possible performance for all initiators. The array won't send pause frames favoring the TCP congestion controls instead. However, if the array receives a pause frame it will respond appropriately.

One thing about flowcontrol is that it's only activated for a specific period of time on each frame. It does not have to wait for a resume frame. So the impact on TCP is not significant.

The typical scenario is that the server can't buffer the incoming data fast enough. Either because their aren't enough ring buffers or it's too busy to process the packets. In that case it will issue a pause frame to that switch port and data will stop for a very brief period. That gives the server a chance to catch up and be ready when data resumes. If flowcontrol is not enabled, packets will be dropped in large quantities and those will have to be retransmitted. That causes reduction in performance and if severe enough, dropped iSCSI connections.

Regards,

-don

This question has come up a lot recently in my group. What we've found is that the right answer is to enable flow control, but also provide a switch with enough buffering per port to keep from falling over. When you see lots of pause frames, then you know your switch isn't keeping up. Of course, this assumes your switch is using store and forward vs. cut through.