PowerFlex: SDS Unable to Exit Protected Maintenance Mode (PMM)

During planned maintenance on PFMC, the user placed the SDS in PMM via the Service Mode function in the Resource Group UI.

After the maintenance work is complete, the SDS nodes reconnect successfully, and the user exits the SDS from PMM. The PFMC workflow then attempts to bring the SDS out of PMM, but fails.

MDM events demonstrate the exact sequence

Enter PMM command goes through to the MDM via the MDS Gateway (GW) and is acted on by the MDM successfully, as expected, including the MDM-to-SDS flow (Decouple/Reconnect):

2025-07-01 00:52:08.518 MDM_CLI_CONF_COMMAND_RECEIVED INFO Command enter_protected_maintenance_mode received, User: 'block-legacy-gateway-mds'. [0000000c] SDS: ID: 0000000000000001
2025-07-01 00:59:40.723 SDS_MAINTENANCE_MODE_STARTED INFO SDS ID: 0000000000000001 entered maintenance mode
2025-07-01 00:59:57.927 SDS_DECOUPLED INFO SDS ID: 0000000000000001 decoupled
2025-07-01 01:16:11.255 SDS_RECONNECTED INFO SDS ID: 0000000000000001 reconnected

Clear device errors command goes through to the MDM via the MDS GW and is acted on by the MDM successfully, as expected:

2025-07-01 01:16:13.744 MDM_CLI_CONF_COMMAND_RECEIVED INFO Command clear_sds_device_error received, User: 'block-legacy-gateway-mds'. [00000005] SDS: ID: 0000000000000001. Clear on all devices.
2025-07-01 01:16:13.745 MDM_CLI_CONF_COMMAND_COMPLETED INFO Command clear_sds_device_error completed successfully. [00000005]

Exit PMM command goes through to the MDM via the MDS GW successfully, as expected, but fails with "wrong maintenance mode type was chosen":

2025-07-01 01:16:13.763 MDM_CLI_CONF_COMMAND_RECEIVED INFO Command exit_maintenance_mode received, User: 'block-legacy-gateway-mds'. [00000006] SDS: ID: 0000000000000001 2025-07-01 01:16:13.764 SDS_MAINTENANCE_MODE_EXIT_FAILED ERROR SDS ID: 0000000000000001 failed to exit maintenance mode. ResultCode: 3623, Message: "The wrong maintenance mode type was chosen for this SDS"

Exit PMM command goes through to the MDM by the Admin user (manual intervention) successfully, as expected:

2025-07-01 01:16:13.763 MDM_CLI_CONF_COMMAND_RECEIVED INFO Command exit_maintenance_mode received, User: 'block-legacy-gateway-mds'. [00000006] SDS: ID: 0000000000000001
2025-07-01 01:16:13.764 SDS_MAINTENANCE_MODE_EXIT_FAILED ERROR SDS ID: 0000000000000001 failed to exit maintenance mode. ResultCode: 3623, Message: "The wrong maintenance mode type was chosen for this SDS"

Key identifiers that confirm the source of the commands:

• User: block-legacy-gateway-mds in every failed exit_maintenance_mode entry.
• No external IPs or API calls were present; all actions originated from the PFMC management workflow.

Impact

The condition prevents the automated exit from Protected Maintenance Mode, extending the maintenance window and requiring manual intervention for each affected SDS node.

A software defect in the MDS GW component incorrectly clears the internal isProtectedMM flag, which defaults into a false value and then issues a generic exit_maintenance_mode request to the MDM. The MDM validates the requested mode against the SDS's current state, and, finding that the SDS is still in Protected Maintenance Mode, rejects the request with the error “The wrong maintenance mode type was chosen for this SDS”.

Manual execution of exit_protected_maintenance_mode by an administrator produces the expected SDS_MAINTENANCE_MODE_ENDED event, proving that the SDS remains in PMM and that the generic exit command is the sole failure point.

1.  Manual intervention is needed. Run the exit_protected_maintenance_mode command on the Primary MDM, by the administrator user (or any user with appropriate privileges):

   scli --login --username admin --password <admin_password> --management_system_ip <PFMP_UI_IP>

   scli --exit_protected_maintenance_mode --protection_domain_name <Protection Domain Name> --sds_name <SDS Name>

   
   
    
   Expected output:

   # scli --exit_protected_maintenance_mode --protection_domain_name pd1 --sds_name sds1
   Exit Protected Maintenance Mode Results: SDS sds1: Exiting maintenance-mode.

   
   
2. Confirm success by looking at the MDM events or PFMP UI. The SDS will be out of PMM and available for normal I/O.
3. If an upgrade cannot be performed immediately, continue using the manual exit_protected_maintenance_mode command after each automated recovery.

Impacted Versions

PFMP 4.6.xx

Fixed In Version