Connectrix B-Series: C3 TX on ISL’s, when ISL R_RDY Mode turned on and HP 3PAR storage in the environment with IOCTL messages turned on
Summary: Connectrix B-Series: C3 TX discards logging on ISL’s, when ISL R_RDY Mode turned on and HP 3PAR storage in the environment with IOCTL messages turned on
Symptoms
Impact:
C3 TX discards, listed on the ISL ports in porterrshow and the errdump.
Environment:
EMC Hardware: Connectrix DS-6505B
EMC Hardware: Connectrix DS-6510B
EMC Hardware: Connectrix DS-6520B
EMC Hardware: Connectrix MP-7840B
EMC Hardware: Connectrix ED-8510-4B
EMC Hardware: Connectrix ED-8510-8B
Problem:
Class 3 TX discards reported on ISL s when ISL_R_RDY is turned on and HP 3PAR storage is in the environment with the IOCTL switch turned to default. (0) When VC Link Init is in use, no C3 discards are observed.
Porterrshow show C3 TX discards. No other TX discards reported in the fabric, except on the ISL s.
Errdump:
[MAPS-1002], 9666, SLOT 6 | FID 128, ERROR, ED-8510, E-Port 1/0, Condition=ALL_E_PORTS(C3TXTO/min>10)
Cause
I. When ISL_R_RDY is turned on, on the ISL s (Can be seen in the portcfgshow command.) All traffic is collapsed into one channel and no VC (Virtual Channels) on the ISL are used. This means that all traffic, including internal switch management traffic (F_class traffic) is mixed up with normal data traffic and also using the same credits as normal data traffic.
When VC Link Init is used, it uses Brocade proprietary method of separating Switch management traffic (F_class traffic) from the data traffic. In VC Link Init mode the data VCs are separated in VC s 2 -5 (no QoS), and the switch management traffic uses a separate VC, (VC 0) not interfering with the Data VC s were the discards were being observed.
II. HP 3PAR storage has been seen flooding the management server (FFFFFA) with large numbers of queries that require those request to be transmitted across the ISL s. When ISL_R_RDY is turned on, these thousands of requests will use the collapsed path as described above, using the same bb credits and causing credit shortage on the link, causing the frames to be discarded.
Resolution
If proven by the using the workarounds mentioned below, customer needs to open a SR with HP, and investigate the problem.
Workaround:
There are 2 option for the workaround.
I. Use VC Link Init mode on the ISL s instead of the ISL_R_RDY . Note that this is not always possible, if DWDM is in use and the DWDM vendor is not able to use VC Link Init mode.
II. On the HP 3Par storage, HP needs to run some script, and one of the option in that script is to turn of fabric_ioctl, by changing the parameter of the setting skip_fabric_ioctls from Default: 0 to 1 ( 0 = off, 1= on)
Unknown of any other cofiguration settings in the script which change the configuration on the 3PAR storage to stop the excessive queries.
Customer needs to contact HP for further assistance.