VCF on VxRail: 4.3.1 failure to remove node from management domain

Summary: VCF on VxRail 4.3.1 failure to remove node from management domain on SDDC Manager UI.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

When removing the node from SDDC Manager UI for the management domain, there is failure on the task and it is stuck.
Rebooting VxRM, resetting password for management user, and recopying SSL certificates does not help out.

SDDC Manager log:

2022-04-07T10:58:17.570+0000 INFO  [vcf_dm,a22a8116cf0bdd37,d9d9] [c.v.e.s.c.v.VxRailManagerService,dm-exec-5]  Parsing VxRail Manager request /rest/vxm/v1/requests/ for response {"id":"36981dd7-d3fb-422a-ac3a-79ed93a2e009","owner":"HOST_REMOVAL","state":"FAILED","error":{"subErrorCode":1006,"detail":"Failed in do precheck action."},"progress":13,"start_time":1649328915334,"end_time":1649328990463,"step":"PrecheckAction","extension":[{"name":"MaintenanceModeValidationAction","status":true},{"name":"MinimalNodeChecker","status":true},{"name":"ResourceCapacityChecker","status":true},{"name":"ResourceCPUChecker","status":true},{"name":"ResourceMemoryChecker","status":false,"message":"Failed to check memory."},{"name":"VsanHealthyChecker","status":true}]}...


VxRail Manager log:

2022-04-07T10:54:14.161+0000 INFO [ForkJoinPool.commonPool-worker-3] com.emc.mystic.manager.web.service.nodes.prechecker.ResourceCapacityChecker ResourceCapacityChecker.check:31 - ResourceCapacityChecker: Storage Used = 55385026409
234, Storage Capacity = 637563693957120, CAPACITY_THRESHOLD = 0.8, result = -0.713130206543789
2022-04-07T10:54:14.161+0000 INFO [ForkJoinPool.commonPool-worker-3] com.emc.mystic.manager.web.service.nodes.prechecker.ResourceCPUChecker ResourceCPUChecker.check:28 - ResourceCPUChecker: Used CPU Capacity = 467060, CPU Capacity
= 1206912, Node CPU Capacity = 150864.0, CPU_THRESHOLD = 0.8, result = -0.35772843658621584
2022-04-07T10:54:14.162+0000 INFO [ForkJoinPool.commonPool-worker-3] com.emc.mystic.manager.web.service.nodes.prechecker.ResourceMemoryChecker ResourceMemoryChecker.check:29 - ResourceMemoryChecker: Used Memory Capacity = 587221434
3680, Memory Capacity = 8234508066816, Node Memory Capacity = 1.029313511424E12, MEM_THRESHOLD = 0.8, result = 0.014997332623799098
2022-04-07T10:54:14.228+0000 ERROR [ForkJoinPool.commonPool-worker-3] com.emc.mystic.manager.web.service.nodes.action.PrecheckAction PrecheckAction.perform:76 - Failed to do PrecheckAction. Name: ResourceMemoryChecker, Error: Failedto check memory.
2022-04-07T10:54:14.229+0000 ERROR [ForkJoinPool.commonPool-worker-3] com.emc.mystic.manager.web.service.nodes.NodeRemovalServiceImpl NodeRemovalServiceImpl.perform:167 - Failed to Remove node com.emc.mystic.manager.web.webutil.RestException: null

 

Cause

There is heavy load running on some VxRail nodes. VxRail Manager performs resource checking when running the node removal task in SDDC Manager. This is to ensure that there is enough resources to meet cluster operations.
When this is calculated by VxRail Manager and nodes are under heavy load, with DRS set to manual, the task fails. 
/usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/classes/com/emc/mystic/manager/web/config.properties
#Node removal resource threshold definition
mystic.node.removal.cpu.threshold=0.80
mystic.node.removal.capacity.threshold=0.80
mystic.node.removal.memory.threshold=0.80

Resolution

To resolve this issue:

  1. Reduce used memory by shutting down some unused VMs or enable automatic DRS setting.
  2. Modify /usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/classes/com/emc/mystic/manager/web/config.properties file
  3. Change mystic.node.removal.memory.threshold to a higher value than the current calculated one.
  4. Restart vmware-marvin service.
  5. Rerun the failing workflow under SDDC Manager UI to remove the node from the domain.

Affected Products

VxRail, VxRail Appliance Series, VxRail E560 VCF, VxRail E560F VCF, VxRail E560N VCF, VxRail G560 VCF, VxRail G560F VCF, VxRail P570 VCF, VxRail P570F VCF, VxRail P580N VCF, VxRail S570 VCF, VxRail V570 VCF, VxRail V570F VCF
Article Properties
Article Number: 000198326
Article Type: Solution
Last Modified: 09 May 2025
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.