This article summaries the Chinese ATE activity: " SAN Switch Fill Word Concept and Configuration ". The original thread is https://community.emc.com/thread/202202.
Question 1: What is fill word?
To discuss SAN fill word, we should talk about the underlying FC protocol first. We know that in FC network, the data is divided into frames for transmission. When there is not enough frames on the link, the fill words will be transmitted for link activation and time synchronization. The fill word will be used as the link initialization signal, after the port negotiation, the switch will set the port status to “up” after receiving the first fill word.
In 2G and 4G FC protocols, the idle signal is IDLE, 0x3EAA2AAAAA, while in 8G FC protocol, T11 organization FC-PI-4 and FC-FS-3 standards defined the fill word should be ARB(FF), 0x3E8B2A654E.
So, to accommodate the new specifications and different vendor implementations, Brocade developed a user-selectable method to set the fill words to either IDLEs or ARB (FF). There are four modes:
Mode 0 - Use IDLEs in link initialization and IDLEs as fill word (default mode).
Mode 1 - Use ARB (FF) in link initialization and ARB (FF) as fill words.
Mode 2 - Use IDLEs in link initialization and ARB (FF) as fill words.
Mode 3 - Try Mode 1 first; if it fails, then try Mode 2.
Question 2: How to verify and change fill word?
For Brocade FOS system, we can use portcfgshow to view fill word settings:
New_DS_5100B:root> portcfgshow 2
Area Number: 2
Speed Level: AUTO(HW)
Fill Word(On Active) 0(Idle-Idle)
Fill Word(Current) 0(Idle-Idle)
And use portcfgfillword to change the setting:
New_DS_5100B:root> portcfgfillword 2 3
New_DS_5100B:root> portcfgshow 2
Area Number: 2
Speed Level: AUTO(HW)
Fill Word(On Active) 3(A-A then SW I-A)
Fill Word(Current) 3(A-A then SW I-A)
Please be noted that, the portcfgfillword command will make the port to be “down” status for about 3 seconds. You might see “disk operation error” on the host, and it is normal.
For Brocade 16G products (DCX-8510, DS6500), the fill word issue has been fixed on the code level. Users don’t have to care about its setting anymore.
There is no fill word setting in Cisco SAN switches, the issue had been fixed in code level as well.
Question 3: What problem would be caused by the wrong configuration in fill word?
If the port runs at 8Gbps speed and the fill word is set as the default IDLE mode, the er_bad_os counter will increase very quickly, as well as a lot of enc_out. For example:
New_DS_5100B:root> portstatsclear 2
New_DS_5100B:root> portstatsshow 2
stat_wtx 1734 4-byte words transmitted
stat_wrx 4912 4-byte words received
er_bad_os 852032805 Invalid ordered set
And if the or_bad_os count reaches some threshold, the switch might set the port status as faulty due to the high invalid word. Some HBA cards might also have ports “up and down” issue.
Question 4: Changing SFP, HBA then FC cables are three common solutions to SAN network troubleshooting. If a Brocade switch shows “disk operation error”, how can we troubleshoot this issue? Using porterrshow or errshow commands?
Usually we check the output of switchshow command first, then the counters in porterrshow output. We should also take a look at the up and down recorders in fabriclog –show command.
There are several major error types in porterrshow command you should pay attention to:
enc_in: Encoding errors inside frames.
crc_err: The port received CRC checksum errors. It might be SFP or HBA hardware issues.
crc_g_eof: The end of the CRC frame is correct, the CRC is caused by the port itself. The SFP module might have hardware issue.
enc_out: Similar to enc_in this is the same encoding error however this error was outside normal frame boundaries. The individual enc_out error means the fiber optic transmission is not good or has the interference. Reseating the fiber cable will lead this counter increasing very quickly.
disc_c3: There are discarded frames. Need to use portstatsshow command to check the port counters.
c3timeout (tx/rx): These are counters which indicate that a port is not able to forward a frame in time to its destination. These either show a problem downstream of this port (tx) or a problem on this port where it has received a frame meant to be forwarded to another port inside the same switch. (rx). Frames are always discarded at the RX side (since that's where the buffers hold the frame).
If the c3timeout_tx occurs on F-port, it means the connected device is a slow drain device. If it occurs on E-port, the device connected to ISL is a slow drain device.
We can pay attention to the C2 and C3 error sin errshow or errdump, such as C2-1010：Internal monitoring has identified suspect hardware, blade may need to be reset or replaced which means there might be a hardware issue. We can use errdump -r to get reversed output, and add | more to display one screen of output at a time.
Question 5: By default, shall we pay attention to the configuration of fill word when initializing the switch?
Brocade switches default fill word is mode 0 (IDLE). The detailed configuration depends on the network design. The goal is to eliminate the er_bad_os errors. The setting should be applied to 8Gbps switches only. You don’t need to change the settings on 2Gbps or 4Gbps switches.
For DS6500 and DCX-8510, they are based on cordor3 platform so users don’t need to change the fill word. We should only pay attention to DS300B/5100/5300, MP7500/7600/7800 and DCX/DCX-4S which are based on condor2 or golden eye2 platforms.
Need help in understanding what does the fillword actually do during link initialization. How is the link initialization done with IDLE and ARB(FF) values? What is the actual difference in link initialization with the two different values IDLE amd ARB(FF)? Does ARB stand for arbitration? Why is FF written along with ARB?
Some 8G devices are not capable of properly establishing links with Brocade 8G Fibre Channel switches when ARB/ARB or IDLE/ARB primitives are used. These 8G devices require the legacy IDLE/IDLE sequence to achieve successful link initialization. To address this issue, Brocade has provided the ability to configure any of the three possible combinations (IDLE/IDLE, ARB/ARB, or IDLE/ARB) for link initialization and fill words. Any of these modes can be configured on an individual port basis using the CLI “portcfgfillword”; hence some newer 8 GB devices may experience problems with the use of Idle/Idle sequence at 8 GB. These devices may require the ARB (FF) / ARB (FF) sequence to have a successful link initialization.
The “portcfgfillword” command is applicable only to FC ports on Condor 2 ASIC and the setting is only enforced when the port is running at 8 Gbps. For Brocade 8G platforms running FOS v6.3.1 or higher, configuring the fill word to mode 3 typically enables the port to use the right combination of Fibre Channel primitives to fully inter-operate with the attached end device. Once set to mode 3, user intervention is no longer needed to configure the right primitives to enable the link between the Brocade 8G switch port and the end device.
Note: The user configurable fillword parameter has been removed on Condor 3 platforms as the Condor 3 automatically negotiates to the correct fillword. There is no need to configure the fillword anymore. However, the FC8-64 blade (8G), based on Condor 2 ASICs is supported for installation into a 8510 chassis (16G), based on Condor 3 ASICs. As such the FC8-64 blade can be configured with “portcfgfillword” command, even when installed in a Condor 3 based chassis.
Fillword should be manually set against ports negotiating at 8Gbps and cannot be set globally on the switch. Changing the fillword setting is disruptive to the port.
Thanks for the above information.
Need to know about the below:
1.) How is the link initialization done with IDLE and ARB(FF) values? What is the actual difference in link initialization process with the two different values IDLE and ARB(FF)?
2.) Does ARB stand for arbitration? Why is FF written along with ARB?
IDLE and ARB(xx) are Fibre Channel Primitives.
The EMC Networked Storage Concepts and Protocols techbook (refer to figure 34 on page 104) describes the link initialization protocol and how IDLEs are used within it.
With regards to ARB(ff), the Fibre Channel standard (FC-FS-4) contains some helpful information:
3.1.39 Emission Lowering Protocol
option in the 8B/10B transmission code that uses the ARBff Primitive Signal, in place of the Idle Primitive
Signal, as a Fill Word for maintaining link synchronization in the absence of other Transmission Words (see
and then later on...
11.3.5 Emission Lowering Protocol
An FC-0 standard (e.g., FC-PI-3) may specify the use of Emission Lowering Protocol when using the 8B/
10B transmission code.
When Emission Lowering Protocol is used, the Fill Word shall be the ARBff Ordered Set.
When Emission Lowering Protocol is not used, the Fill Word shall be the Idle Ordered Set.
I hope this helps.