PowerFlex 3.X: Node removal fails with "Failed to scale-down MDM cluster"

In a 3-node MDM cluster mode, the end user attempts to remove node02 as a resource from a Service, but the operation fails. Node02 does not have the MDM RPM package installed and is not part of the MDM cluster.

The corresponding deployment.log reports that PFxM failed to scale down the MDM cluster.

WARN  [2024-08-27T15:55:28.431456] 3878: provider/elementmanager/scaleio.rb:582:in `vxos_sds_teardown!': scaleio-cluster11: Failed to scale-down MDM cluster for node02: RuntimeError: Requesting One Node mode is not currently allowed.  Failed to find standby for Three Node Cluster.
DEBUG [2024-08-27T15:55:28.431974] 3878: /opt/asm-deployer/rules/teardown/server_teardown_rule.rb:23:in `handle_scaleio_node_teardown': Failed to remove SDS: rackserver-535qmh3: RuntimeError: Failed to scale-down MDM cluster for node02: RuntimeError: Requesting One Node mode is not currently allowed.  Failed to find standby for Three Node Cluster.
WARN  [2024-08-27T15:55:28.432235] 3878: rule_engine/rule.rb:238:in `process_state': Rule #<ASM::RuleEngine::Rule:3960 priority: 50 name: server_teardown @ /opt/asm-deployer/rules/teardown/server_teardown_rule.rb> failed to run: Failed to scale-down MDM cluster for node02: RuntimeError: Requesting One Node mode is not currently allowed.  Failed to find standby for Three Node Cluster.
WARN  [2024-08-27T15:57:13.489492] 3822: rule_engine/rule.rb:238:in `process_state': Rule #<ASM::RuleEngine::Rule:3864 priority: 50 name: server_lane_teardown @ /opt/asm-deployer/rules/service/server_lane_teardown_rule.rb> failed to run: Failed to scale-down MDM cluster for node02: RuntimeError: Requesting One Node mode is not currently allowed.  Failed to find standby for Three Node Cluster.


The asmManager.log reports the deployment failed and that it failed to scale down:

2024-09-06 13:30:15 [JobMgrQuartzScheduler_Worker-70] (DeviceInventoryDAO.java:1328) [DEBUG] Update state to DEPLOYMENT_ERROR passed for device <node02_service_tag>
2024-09-06 13:30:15 [JobMgrQuartzScheduler_Worker-70] (ServiceDeploymentJob.java:850) [DEBUG] Updated device state for <node02_service_tag> to Deployment Error
2024-09-06 13:30:15 [JobMgrQuartzScheduler_Worker-70] (ServiceDeploymentJob.java:318) [WARN] Deployment 8afd098b917af927019195adbdfa12f0 final teardown status was Error
2024-09-06 13:30:15 [JobMgrQuartzScheduler_Worker-70] (ServiceDeploymentJob.java:435) [ERROR] ServiceDeploymentJob.execute error for deployment 8afd098b917af927019195adbdfa12f0
com.dell.asm.asmcore.asmmanager.exception.AsmManagerRuntimeException: Scale-down failed for Server or VM in PowerFlex Service 8afd098b917af927019195adbdfa12f0
    at com.dell.asm.asmcore.asmmanager.tasks.ServiceDeploymentJob.individualTeardownSuccessful(ServiceDeploymentJob.java:704) ~[classes!/:?]
    at com.dell.asm.asmcore.asmmanager.tasks.ServiceDeploymentJob.executeRazorDeployment(ServiceDeploymentJob.java:320) ~[classes!/:?]
    at com.dell.asm.asmcore.asmmanager.tasks.ServiceDeploymentJob.executeSafely(ServiceDeploymentJob.java:1109) ~[classes!/:?]
    at com.dell.asm.asmcore.asmmanager.tasks.AsmDefaultJob.execute(AsmDefaultJob.java:41) ~[classes!/:?]
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar!/:?]
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[quartz-2.3.2.jar!/:?]
Impact 
Unable to remove a node resource from a Service.

The asmManager.log reports the deployment failed and that it failed to scale down:

2024-09-06 13:30:15 [JobMgrQuartzScheduler_Worker-70] (DeviceInventoryDAO.java:1328) [DEBUG] Update state to DEPLOYMENT_ERROR passed for device <node02_service_tag>
2024-09-06 13:30:15 [JobMgrQuartzScheduler_Worker-70] (ServiceDeploymentJob.java:850) [DEBUG] Updated device state for <node02_service_tag> to Deployment Error
2024-09-06 13:30:15 [JobMgrQuartzScheduler_Worker-70] (ServiceDeploymentJob.java:318) [WARN] Deployment 8afd098b917af927019195adbdfa12f0 final teardown status was Error
2024-09-06 13:30:15 [JobMgrQuartzScheduler_Worker-70] (ServiceDeploymentJob.java:435) [ERROR] ServiceDeploymentJob.execute error for deployment 8afd098b917af927019195adbdfa12f0
com.dell.asm.asmcore.asmmanager.exception.AsmManagerRuntimeException: Scale-down failed for Server or VM in PowerFlex Service 8afd098b917af927019195adbdfa12f0
    at com.dell.asm.asmcore.asmmanager.tasks.ServiceDeploymentJob.individualTeardownSuccessful(ServiceDeploymentJob.java:704) ~[classes!/:?]
    at com.dell.asm.asmcore.asmmanager.tasks.ServiceDeploymentJob.executeRazorDeployment(ServiceDeploymentJob.java:320) ~[classes!/:?]
    at com.dell.asm.asmcore.asmmanager.tasks.ServiceDeploymentJob.executeSafely(ServiceDeploymentJob.java:1109) ~[classes!/:?]
    at com.dell.asm.asmcore.asmmanager.tasks.AsmDefaultJob.execute(AsmDefaultJob.java:41) ~[classes!/:?]
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar!/:?]
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[quartz-2.3.2.jar!/:?]

Impact 

Unable to remove a node resource from a Service.

When a node is removed from a Service, the code logic switches cluster modes based on the cluster’s size, even if the node being removed is not part of the MDM cluster. PFxM attempts to switch to a lower cluster mode depending on the current mode. If it is a 5-node cluster mode, it switches to a 3-node mode. If it is a 3-node mode, it attempts to switch to a 1-node mode, but this operation is not allowed.

Ensure that the MDM cluster is in 5-node mode before removing a node from a Service, and confirm that both the new Manager MDM and the TB MDM are added as Standby MDMs.

scli --switch_cluster_mode --cluster_mode 5_node --add_slave_mdm_ip <mdm_mgmt_ip> --add_tb_ip <tb_ip>

Impacted Version

PFxM 3.8.x and lower