Connectrix MDS 9700 series - Faulty module' warnings and port loopback failures

Initially, the issue is non impacting and manifests as errors on the active supervisor:



After some time, ports on some or all linecards begin to continuously fail diagnostic loopback tests. The following type of syslog messages are logged:

%DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:1 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 6 Error:Loopback test failed. Packets possibly lost on the switch SUP fabric
%MODULE-4-MOD_WARNING: Module 1 reported warning fc1/1 due to Loopback test failed. Packets possibly lost on the switch SUP fabric in device DEV_UNDEF (device error 0xc0000300)

Reloading the linecards does not resolve this issue.

This issue only occurs on MDS 9706 and MDS 9710 platforms with IPFC interfaces that receive broadcast IPFC frames (such as ARP requests). 

If the system was last cold booted with any version earlier than NX-OS 6.2(9), the issue is further limited to systems where a supervisor switchover (eg during ISSU/D or manually) has been done.

If the system was last cold booted with any version of NX-OS 6.2(9) through 6.2(15) (inclusive) or 7.3(0)D1(1) then the issue will not occur at all.

Cisco bug : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy06296

Workaround :
Perform a supervisor switchover with the system switchover command and reload any linecards that continue to fail testing after the switchover (this will disrupt traffic through any rebooted linecards). Alternatively, contact Cisco TAC for a proactive nondisruptive workaround.

This issue can be avoided by shutting down all IPFC interfaces. This removes the trigger.

Permanent fix : upgrade to NX-OS 6.2(17)

Further details : IPFC frame forwarding configuration was able to get out of synchronisation between the supervisor and linecards. This lead to credit loss on the inband interface which was detected by run-time diagnostic tests. The fix ensures that the supervisor and linecard configurations remain in synchronisation under all conditions.