VxRail: ESXi hosts run into a "HA error state" after implementing a Custom Certificate
Summary: Custom CA certificates are added to the ESXi hosts and vSphere High Availability (HA) stops working.
Symptoms
Custom CA certificates are added to the ESXi hosts and vSphere High Availability (HA) stops working.
/var/log/fdm.log:
2017-05-18T11:24:28.018Z error fdm[3A608B70] [Originator@6876 sub=Message opID=SWI-787207f7] [AcceptorImpl::FinishSSLAccept] Error N7Vmacore3St read) creating ssl stream or doing handshake
2017-05-18T11:24:28.145Z verbose fdm[FFD7FB70] [Originator@6876 sub=Election opID=SWI-60b7acd9] CheckVersion: Version[2] Other host GT : 90 >
2017-05-18T11:24:28.145Z verbose fdm[FFD7FB70] [Originator@6876 sub=Cluster opID=SWI-60b7acd9] [ClusterPersistence::VersionChange] version[2]
2017-05-18T11:24:28.145Z info fdm[FFD7FB70] [Originator@6876 sub=Cluster opID=SWI-60b7acd9] [ClusterPersistence::VersionChange] fetching versi
2017-05-18T11:24:28.145Z verbose fdm[FFD7FB70] [Originator@6876 sub=Election opID=SWI-60b7acd9] CheckVersion: Version[0] Other host Less : 260
2017-05-18T11:24:28.153Z error fdm[FFF45B70] [Originator@6876 sub=Message opID=SWI-66926e8] [MsgConnectionImpl::FinishSSLConnect] Error N7Vmac
--> PeerThumbprint: 3D:7E:55:CD:CF:9E:B1:C2:04:41:F6:59:2D:05:BB:49:7F:A7:AA:F3
--> ExpectedThumbprint: FE:B6:B6:44:65:DC:B7:70:C4:DD:0B:EA:CF:A1:5E:8A:13:50:1D:CA
--> ExpectedPeerName: host-87
--> The remote host certificate has these problems:
--> * Host name does not match the subject name(s) in certificate.
Cause
Resolution
1. Check the fdm.log on the primary host and copy the thumbprints for future reference.
2. Stop the vCenter Server service.
3. Connect to the vCenter Server database.
4. Take a backup of the vCenter Server database before making any changes.
5. Check if you can see the two thumbprints (from the fdm.log) in the VPX_HOST table.
6. Ensure that both values are the same as thumbprint from SSL certificate located in /etc/vmware/ssl/rui.crt
7. Commit the changes to the database.
8. Start the vCenter Server service and connect to it with the vSphere Client/Web Client.
9. Re-enable HA
To see both thumbprints for all hosts you can use below query:
select ID, DNS_NAME, IP_ADDRESS, EXPECTED_SSL_THUMBPRINT, HOST_SSL_THUMBPRINT from VPX_HOST;
It lists output similar to:
VCDB=# select ID, DNS_NAME, IP_ADDRESS, EXPECTED_SSL_THUMBPRINT, HOST_SSL_THUMBPRINT from VPX_HOST;
id | dns_name | ip_address | expected_ssl_thumbprint | host_ssl_thumbprint
To update thumbprints, you can use query similar to:
UPDATE VPX_HOST SET EXPECTED_SSL_THUMBPRINT=' DE:55:42:C7:81:2D:FA:D8:3C:73:4B:94:35:54:47:96:17:87:51:FF' where ID=37;
UPDATE VPX_HOST SET host_ssl_thumbprint=' DE:55:42:C7:81:2D:FA:D8:3C:73:4B:94:35:54:47:96:17:87:51:FF' where ID=37;
Additional Information
For more information, see: https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/6-5/vsphere-troubleshooting-6-5/troubleshooting-vcenter-server-and-the-vsphere-web-client/troubleshooting-vcenter-server-certificates/vsphere-ha-and-custom-ssl-certificates.html