This article summaries the Chinese ATE activity: " SAN Switch Fill Word Concept and Configuration ". The original thread is https://community.emc.com/thread/202202.

Question 1: What is fill word?

Answer:

To discuss SAN fill word, we should talk about the underlying FC protocol first. We know that in FC network, the data is divided into frames for transmission. When there is not enough frames on the link, the fill words will be transmitted for link activation and time synchronization. The fill word will be used as the link initialization signal, after the port negotiation, the switch will set the port status to “up” after receiving the first fill word.

In 2G and 4G FC protocols, the idle signal is IDLE, 0x3EAA2AAAAA, while in 8G FC protocol, T11 organization FC-PI-4 and FC-FS-3 standards defined the fill word should be ARB(FF), 0x3E8B2A654E.

So, to accommodate the new specifications and different vendor implementations, Brocade developed a user-selectable method to set the fill words to either IDLEs or ARB (FF). There are four modes:

Mode 0 - Use IDLEs in link initialization and IDLEs as fill word (default mode).

Mode 1 - Use ARB (FF) in link initialization and ARB (FF) as fill words.

Mode 2 - Use IDLEs in link initialization and ARB (FF) as fill words.

Mode 3 - Try Mode 1 first; if it fails, then try Mode 2.

Question 2: How to verify and change fill word?

Answer:

For Brocade FOS system, we can use portcfgshow to view fill word settings:

New_DS_5100B:root> portcfgshow 2

Area Number:              2

Speed Level:              AUTO(HW)

Fill Word(On Active)      0(Idle-Idle)

Fill Word(Current)        0(Idle-Idle)

......

And use portcfgfillword to change the setting:

New_DS_5100B:root> portcfgfillword 2 3

New_DS_5100B:root> portcfgshow 2

Area Number:              2

Speed Level:              AUTO(HW)

Fill Word(On Active)      3(A-A then SW I-A)

Fill Word(Current)        3(A-A then SW I-A)

......

Please be noted that, the portcfgfillword command will make the port to be “down” status for about 3 seconds. You might see “disk operation error” on the host, and it is normal.

For Brocade 16G products (DCX-8510, DS6500), the fill word issue has been fixed on the code level. Users don’t have to care about its setting anymore.

There is no fill word setting in Cisco SAN switches, the issue had been fixed in code level as well.

Question 3: What problem would be caused by the wrong configuration in fill word?

Answer:

If the port runs at 8Gbps speed and the fill word is set as the default IDLE mode, the er_bad_os counter will increase very quickly, as well as a lot of enc_out. For example:

New_DS_5100B:root> portstatsclear 2

New_DS_5100B:root> portstatsshow 2

stat_wtx                1734        4-byte words transmitted

stat_wrx                4912        4-byte words received

......

er_bad_os               852032805   Invalid ordered set

And if the or_bad_os count reaches some threshold, the switch might set the port status as faulty due to the high invalid word. Some HBA cards might also have ports “up and down” issue.

Question 4: Changing SFP, HBA then FC cables are three common solutions to SAN network troubleshooting. If a Brocade switch shows “disk operation error”, how can we troubleshoot this issue? Using porterrshow or errshow commands?

Answer:

Usually we check the output of switchshow command first, then the counters in porterrshow output. We should also take a look at the up and down recorders in fabriclog –show command.

There are several major error types in porterrshow command you should pay attention to:

      enc_in: Encoding errors inside frames.

      crc_err: The port received CRC checksum errors. It might be SFP or HBA hardware issues.

      crc_g_eof: The end of the CRC frame is correct, the CRC is caused by the port itself. The SFP module might have hardware issue.

      enc_out: Similar to enc_in this is the same encoding error however this error was outside normal frame boundaries. The individual enc_out error means the fiber optic transmission is not good or has the interference. Reseating the fiber cable will lead this counter increasing very quickly.

      disc_c3: There are discarded frames. Need to use portstatsshow command to check the port counters.

      c3timeout (tx/rx): These are counters which indicate that a port is not able to forward a frame in time to its destination. These either show a problem downstream of this port (tx) or a problem on this port where it has received a frame meant to be forwarded to another port inside the same switch. (rx). Frames are always discarded at the RX side (since that's where the buffers hold the frame).

If the c3timeout_tx occurs on F-port, it means the connected device is a slow drain device. If it occurs on E-port, the device connected to ISL is a slow drain device.

We can pay attention to the C2 and C3 error sin errshow or errdump, such as C2-1010：Internal monitoring has identified suspect hardware, blade may need to be reset or replaced which means there might be a hardware issue. We can use errdump -r  to get reversed output, and add | more to display one screen of output at a time.

Question 5: By default, shall we pay attention to the configuration of fill word when initializing the switch?

Answer:

Brocade switches default fill word is mode 0 (IDLE). The detailed configuration depends on the network design. The goal is to eliminate the er_bad_os errors. The setting should be applied to 8Gbps switches only. You don’t need to change the settings on 2Gbps or 4Gbps switches.

For DS6500 and DCX-8510, they are based on cordor3 platform so users don’t need to change the fill word. We should only pay attention to DS300B/5100/5300, MP7500/7600/7800 and DCX/DCX-4S which are based on condor2 or golden eye2 platforms.

Translator: Roger