VxRail: vSphere DRS Functionality is Impacted Due to Unhealthy State vSphere Cluster Services

Error on cluster: vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster services caused by the unavailability of vSphere Cluster Service VMs.

vCLS VMs are powered off or having the following HA failure: vSphere HA virtual machine failover failed.

DRS is not working.

vCLS VMs are powered off or having the following HA failure: vSphere HA virtual machine failover failed.

1. Take a snapshot of the vCenter virtual machine (VM).
2. Put the cluster in Retreat Mode: 

• 1. Log in to the vSphere Client
  2. Navigate to the cluster on which vCLS must be deactivated.
  3. Copy the cluster domain ID from the URL of the browser. It should be similar to domain-c(number).
     
     Note: Only copy the numbers to the left of the colon in the URL.
  4. Navigate to the vCenter Server Configure tab.
  5. Under Advanced Settings, click the Edit Settings button.
  6. Add a new entry 

     config.vcls.clusters.domain-c(number).enabled

     
      Use the domain ID copied in step 3.
  7. Set the Value to False.
  8. Click Save.
     • Note: Within 1 minute, all the vCLS VMs in the cluster are cleaned up and the Cluster Services health will be set to Degraded
• If the vCLS VMs were cleared up, remove the cluster from Retreat Mode and wait for 5 minutes for the vCLS VMs to be redeployed again.
  • To remove Retreat Mode from the cluster, change the value in to True.
    1. Log in to the vSphere Client
    2. Navigate to the cluster on which vCLS must be deactivated.
    3. Copy the cluster domain ID from the URL of the browser. It should be similar to domain-c(number).
       
       Note: Only copy the numbers to the left of the colon in the URL.
    4. Navigate to the vCenter Server Configure tab.
    5. Under Advanced Settings, click the Edit Settings button.
    6. Add a new entry

       config.vcls.clusters.domain-c(number).enabled

        Use the domain ID copied in step 3
    7. Set the Value to True.
    8. Click Save.
  • DRS should be working after the vCLS VMs were redeployed successfully, and the overall status is Healthy.

• If the vCLS VMs were not cleared up within couple of minutes, apply the following steps:
  
  1. Take a Snapshot of vCenter
  2. Run this command to retrieve the  vpxd-extension  solution user certificate and key:

     mkdir /certificate

     /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

     /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

  3. Run this command to update the extension's certificate with vCenter Server.

     python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <PNID/FQDN of vCenter Server> -u Administrator@vsphere.local

     Note: The default user and domain is Administrator@vsphere.local. If this was changed during configuration, change the domain to match your environment. When prompted, type in the  Ad ministrator@vsphere.local  password
  4. Restart the VMware ESX Manager service with these commands:

     • service-control --stop vmware-eam
       service-control --start vmware-eam

       
       ​​​​​​​​​​​​​​​​​​​​​​​​​​​
       Note: In certain situations, you might receive the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'". This error can be safely ignored if you are getting the error after the message "Successfully updated certificate for "com.vmware.vim.eam" extension" as this message confirms that Extension certificate updated successfully with vCenter Server.

       • root@hostname [ ~ ]# python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcsa1.test.com -u Administrator@vsphere.local
         Password to connect to VC server for user="Administrator@vsphere.local":
         2021-03-11T22:31:22.517Z  Updating certificate for "com.vmware.vim.eam" extension
         2021-03-11T22:31:22.649Z  Successfully updated certificate for "com.vmware.vim.eam" extension
         Traceback (most recent call last):
           File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 175, in <module>
             update_extension_cert_in_VC()
           File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 163, in update_extension_cert_in_VC
             sessionMgr = si.content.sessionManager
           File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 577, in __call__
             return self.f(*args, **kwargs)
           File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 382, in _InvokeAccessor
             return self._stub.InvokeAccessor(self, info)
           File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
             return self.InvokeMethod(mo, info, (prop, ))
           File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1525, in InvokeMethod
             conn.request('POST', self.path, req, headers)
           File "/usr/lib/python3.7/http/client.py", line 1277, in request
             self._send_request(method, url, body, headers, encode_chunked)
           File "/usr/lib/python3.7/http/client.py", line 1323, in _send_request
             self.endheaders(body, encode_chunked=encode_chunked)
           File "/usr/lib/python3.7/http/client.py", line 1272, in endheaders
             self._send_output(message_body, encode_chunked=encode_chunked)
           File "/usr/lib/python3.7/http/client.py", line 1032, in _send_output
             self.send(msg)
           File "/usr/lib/python3.7/http/client.py", line 972, in send
             self.connect()
           File "/usr/lib/python3.7/http/client.py", line 1447, in connect
             server_hostname=server_hostname)
           File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
             session=session
           File "/usr/lib/python3.7/ssl.py", line 870, in _create
             self.do_handshake()
           File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
             self._sslobj.do_handshake()
         ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'. (_ssl.c:1076)
         

• At this point, make sure that the cluster is not in Retreat Mode and the value is set to True and wait for around 3-5 minutes for the vCLS VMs to be redeployed again.
  • If the vCLS VMs were not redeployed:
    • Renew the vCenter certificates using the following KB: VxRail: Unable to Log In to vCenter Due to Expired Certificates
• If the DRS is still not working after renewing the vCenter certificates:
  • Put the cluster in retreat mode again to delete the vCLS VMs and then remove the cluster from retreat mode to redeploy the vCLS VMs and the issue should be solved
  • ​​​​​​​How to Disable vCLS on a Cluster via Retreat Mode(External Link)