PowerFlex 3.X: Upgrade Fails With MDM Cluster Version Mismatch

The PowerFlex Gateway upgrade operation fails to upgrade the MDM cluster because there is an MDM cluster version mismatch.

Primary MDM event logs showing that the command to start the upgrade failed.

2022-03-29 12:15:40.406 MDM_CLI_CONF_COMMAND_RECEIVED  INFO      Command start_upgrade received, User: 'admin'. [301312170] 
2022-03-29 12:15:40.406 CLI_COMMAND_FAILED  WARNING      Command start_upgrade was not successful. Error code: The command cannot be executed in the current upgrade state [301312170]

Primary MDM trace logs showing the newly upgraded MDM cannot communicate with the older version Primary MDM:

2022-03-29 12:32:18.079 MDM_CLUSTER_NODE_FAILURE  ERROR      This MDM cannot communicate with MDM cluster node, MDM2 (ID 03023c1c60dd2101), invalid response (The version of the MDM is too new)

Impact

MDM cluster will be in a degraded state. The upgrade is halted.

The PowerFlex Gateway instructs the MDM to start an upgrade. Even if the MDM fails to enter the upgrade state, the Gateway continues to perform the upgrade. The secondary MDMs is upgraded but will be unable to reconnect to the Primary MDM because the cluster is not in an upgrade state. 

The MDM cluster will first need to return to a healthy state.

1. From the PowerFlex Gateway, use the rollback button to revert the upgrade that was done on the secondary MDMs/TBs.

2. Verify that all secondary MDM/TBs are connected and the MDM cluster is in a healthy state.

3. Verify that the upgrade state is "No Upgrade." If the state is other than that, finalize the upgrade manually using SCLI:

scli --finilize_upgrade

4. Start the upgrade manually using SCLI:

scli --start_upgrade

5. From the PowerFlex Gateway, start the upgrade again now that the cluster is in an upgrade state.

Impacted Versions

3.x.x

Fixed In Version

PowerFlex 3.6.4

PFMP 4.5.3