Cisco MDS: Heartbeat between active and standby on the redundancy channel not working for last 1 minutes

• MDS 9718 began logging continuous errors "Heartbeat between active and standby on the redundancy channel not working for last 1 minutes" following a firmware upgrade to NXOS 9.4(2a)
• The upgrade completed successfully. In some instances, ISSU may fail
• No impact was reported

CSCwn90610 > Heartbeat between active and standby on the redundancy channel not working

Symptom:

show logging logfile contains many of the following messages.

%SYSMGR-STANDBY-2-REDUNDANCY_HEARTBEAT_FAILURE: Heartbeat between active and standby on the redundancy channel not working for last 1 minutes.

ISSU may fail with the error code 0x40930000. The show install all status command shows the following type of messages:

Continuing the installation in New Active Supervisor:
System is ready to continue. 
System in hot standby, ready to start installer.
Successfully started the installer:
Module 10: 
 Waiting for module online.
 -- SUCCESS 

 Operation didn't complete on time in state WAIT_FOR_SYSTEM_CONSISTENCY
.

 Remaining action:: 
"Module(s) 1, 2, 3, 4, 5, 6, 7, 8, 11, 12 still need to be upgraded".
 Install has failed. Return code 0x40930000 (Current operation failed to complete within specified time).
Please identify the cause of the failure, and try 'install all' again.
This is the log of last installation.

Conditions:

This issue affects only the following MDS platforms:

(a) MDS 9700 switch with Supervisor-1 or Supervisor-1E (also known as Supervisor-3) running NX-OS software versions 9.4(2), 9.4(2a) or 9.4(3)

(b) MDS 32 Gbps fabric switches MDS 9132T, MDS 9148T and MDS 9396T running NX-OS software versions 9.4(1), 9.4(2), 9.4(2a) or 9.4(3)

Workaround:

Proactive workaround:

Postpone NX-OS upgrades until a fixed version is available (the first fixed release will be NX-OS 9.4(3a), due for release in April 2025). 

Reactive workaround:

If an upgrade to the affected version has already been attempted and the switch is logging the heartbeat messages above, then

(a) For the MDS9700, perform two system switchovers and attempt the upgrade again. Repeat the reactive workaround if necessary.

(b) For a single supervisor switch, perform a non-disruptive reload and attempt the upgrade again. Repeat this step if the issue recurs.