Connectrix: Troubleshooting xbar Issues

When an xbar error is seen, and the module goes down, and often a mechanical replacement is performed.

To troubleshoot an xbar issue, isolate the issue (provided the customer allows) between the xbar, supervisor, any line cards, and the chassis.

Perform isolation tasks in dealing with an xbar failure. The process calls for performing the following steps to isolate: 

1. Replace xbar.
   If the replacement fails, the issue lies outside the xbar.
2. With the replacement xbar still inserted, perform a system switchover (pending a maintenance window) to swap active and standby supervisors, and check xbar status. As they can be hot swapped, you can attempt to reinsert the old xbar if the replacement still fails.

2016 Jul 17 14:42:39 MDSswitch Jul 27 14:42:38 %KERN-2-SYSTEM_MSG: [25315862.942013] Switchover started by redundancy driver - kernel

2016 Jul 17 14:42:39 MDSswitch %SYSMGR-2-HASWITCHOVER_PRE_START: This supervisor is becoming active (pre-start phase)

3. If the xbar is still down and the switchover has occurred, collect show tech-support details output and review the exception log. In many instances, the exception log shows errors on a line card from the CHICO ASIC.

exception information --- exception instance 7 ----
Module Slot Number: 7
Device Id         : 145
Device Name       : Sacramento Xbar ASIC
Device Errorcode  : 0xc9100200
Device ID         : 145 (0x91)
Device Instance   : 00 (0x00)
Dev Type (HW/SW)  : 02 (0x02)
ErrNum (devInfo)  : 00 (0x00)
System Errorcode  : 0x41ec006d SAC sync lost
Error Type        : Warning
PhyPortLayer      : Fibre Channel
Port(s) Affected  :
Error Description : Lost sync on slot:7 asic:0 chico:19-->connected to-->Fab-1 link-01
DSAP              : 0 (0x0)
UUID              : 0 (0x0)
Time              : Wed Jul 17 14:43:09 2016
                    (Ticks: 579900BD jiffies)

4. If this occurs, remove and replace that line card, and then attempt to reinsert the xbar after the line card removal.