SDS Fails To Exit Maintenance Mode After It Was Upgraded During PowerFlex NDU

MDM event logs show the SDS being upgraded followed by an exit maintenance mode command:

2023-07-15 03:12:47.469000:0012714:SDS_UPGRADED                     INFO     SDS: <sds_name> (ID  <sds_id> ) upgraded to version 3.6.x
2023-07-15 03:12:47.505000:0012715:SDS_RECONNECTED                  INFO     SDS: <sds_name> (ID  <sds_id>) reconnected. 
 
2023-07-15 03:13:21.618000:0012930:MDM_CLI_CONF_COMMAND_RECEIVED    INFO     Command exit_maintenance_mode received, User: ': <user>'. [50667] SDS: ID: <sds_id>;

The SDS attempts to exit maintenance mode but fails.

SDS trace logs may show the following:

2023/07/15 03:12:57.322606 7feaa54e0db0:contCmd_AddCombAndSetState:10712: Set comb state (reconstruct) failed 13598001806a new state SECONDARY, rc NOT_FOUND
2023/07/15 03:12:57.322806 7feaa54e0db0:contCmd_AddCombAndSetState:10712: Set comb state (reconstruct) failed 13598001808c new state SECONDARY, rc NOT_FOUND
...
 
2023/07/15 03:12:58.030449 7feaa57ffdb0:contDev_FlushDeviceDirtyDrl_Async:09708: Device 0xdfef6e6c001d0005 drl flush skipped due to device in error
2023/07/15 03:12:58.030459 7feaa5568db0:contDev_FlushDeviceDirtyDrl_Async:09708: Device 0xdfef6e70001d0009 drl flush skipped due to device in error
...
 
2023/07/15 03:13:21.650684 7fea9dc59db0:ioh_ResetCombDrl:05345: Reset Drl - No comb:13598001800b device: 0xdfef6e6c001d0005
2023/07/15 03:13:21.650911 7fea9dc15db0:ioh_ResetCombDrl:05345: Reset Drl - No comb:135980018079 device: 0xdfef6e70001d0009
...

The above output shows that after the upgrade the devices had to be reconstructed with the new metadata format which, happens when upgrading from 3.0.x to 3.5.x or newer.

The function contDev_FlushDeviceDirtyDrl_Async considers these devices as errored and then the devices start reporting that there are no combs.

The SDS trace logs stream the "Reset Drl - No comb" line until the issue is resolved.

During the device metadata reconstruction operation for the NDU, a race condition can occur in the MDM code that prohibits the operation from fully completing, and the devices get into a state where it does not think it owns any of the data.

Restarting the SDS service forces the SDS to reconnect to the MDM and the MDM educates the SDS on the combs it owns.

Workaround

To resolve the issue, restart the SDS service on the SDS that is failing to exit maintenance mode, to restart the SDS service, run a pkill sds command.
Wait for the SDS to exit maintenance mode and then retry the Installation Manager upgrade operation.

Impacted Versions

PowerFlex 3.x

Fixed In Version

PowerFlex 3.6.3
PowerFlex 4.5.2

Impacted Versions

PowerFlex 3.x

Fixed In Version

PowerFlex 3.6.3

PowerFlex 4.5.2