VxRail: vCLS VMs Fail to Deploy Due to EAM Agency Being Disabled

Messages similar to the following are seen in vCenter logs:
 

/var/log/vmware/wcp/wcpsvc.log

2025-05-08T04:56:13.577Z info wcp [eamagency/util.go:214] [opID=vCLS] Getting all clusters in the inventory
2025-05-08T04:56:13.579Z info wcp [eamagency/util.go:246] [opID=vCLS] Returning the list of clusters: [ClusterComputeResource:domain-c8]
2025-05-08T04:56:13.592Z info wcp [eamagency/util.go:148] [opID=vCLS] Going to read service config
2025-05-08T04:56:13.592Z info wcp [eamagency/util.go:150] [opID=vCLS] Found OVF locator file:///storage/lifecycle/vmware-hdcs
2025-05-08T04:56:13.592Z info wcp [eamagency/util.go:161] [opID=vCLS] Found OVF directory  as /storage/lifecycle/vmware-hdcs
2025-05-08T04:56:13.595Z info wcp [clustersvc/monitor.go:190] [opID=vCLS] Agency &{0xc00163cc60 Agency:#######-#####-#####-#####-########### } status is yellow on cluster domain-c8
2025-05-08T04:56:13.595Z info wcp [clustersvc/monitor.go:140] [opID=vCLS] Agency (Agency:#######-#####-#####-#####-########### ) for cluster ClusterComputeResource:domain-c8 needs to be resolved, queuing resolve operation
2025-05-08T04:56:13.595Z info wcp [clustersvc/controller.go:162] [opID=vCLS] Received an operation from an emitter's channel: *eamagency.Resolve


/var/log/vmware/eam/eam.log

2025-05-08T20:48:33.080Z |  INFO | vim-monitor | IssueHandlerBase.java | 116 | Updating issues:
New issues:
 [
eam.issue.AgencyDisabled {
   time = 2025-05-08 20:48:33,049,
   description = <unset>,
   key = 0,
   agency = 'Agency:#######-#####-#####-#####-###########:null',
   solutionId = 'VSPHERE.LOCAL\vpxd-extension-#######-#####-#####-#####-###########',
   agencyName = 'vCLS',
   solutionName = ' ',

This happens when the cluster disabled agencies in ESX Agent Manager (EAM).

Before performing this workaround, make sure there are no certificate issues. Refer to vCert tool usage to check certificates status: vCert-Scripted vCenter expired certificate replacement 

Workaround:
Before starting, ensure the VMs on the vSAN cluster are running without accessibility issues.

To reenable the EAM agencies on the cluster:
 

1. Obtain the clusters MOID by selecting, in the vSphere vCenter Client, the cluster level and using the URL address bar find the domain-c# identifier.
2. From a browser, access the EAM-managed object browser (MOB)
   https://<VCENTER_IP>/eam/mob?vmodl=1
   Note: The default user and domain is Administrator@vsphere.local. If this is changed during configuration, change the domain to match your environment. When prompted, type in the Administrator@domain.local password
3. Click Enable at "com.vmware.vim.binding.vmodl.ManagedObjectReference" Method and it opens a new window.
4. In the value field, enter: <cluster type="ClusterComputeResource" serverGuid="Server GUID">MOID</cluster> 
   Replace MOID with the domain-c# value you collected in step 1.
5. Click Invoke Method.
   Give the cluster a few minutes after this to deploy the vCLS VMs.
   When you click invoke method, "Enable system VMs" task triggers on the vCenter cluster.
   
   * If the task fails, you must again follow the steps and remove the serverGUID part from step 4 and only add below line:
   <cluster type="ClusterComputeResource">MOID</cluster> replace MOID with the domain-c# value that you collected in step 1.
    
6. Once invoked, enable system VMs task shows as successful. After waiting for 10 minutes, if vCLS VMs are not deployed, you can put the cluster in retreat mode and disable retreat mode to take effect. See: Disable vCLS on a Cluster via Retreat Mode 

Broadcom article: vCLS VMs do not deploy due to the EAM Agency showing as "Disabled" following a vSAN Cluster Shutdown