VxRail: Node Add Failing with Status "Starting platform service…"
Summary: Node add is failing with error in UI "Starting platform service…"
Symptoms
Node add is failing with error: "Starting platform service…"
/var/log/microservice/{dayone/short.term/long.term} logs you find the following signature:
2021-10-26-12:24:33 microservice.do-host "[2021-10-26 12:24:33,318: ERROR/MainProcess] query is_ps_ready error"
2021-10-26-12:24:33 microservice.do-host "Traceback (most recent call last):"
2021-10-26-12:24:33 microservice.do-host " File ""/home/app/do_host_app/graphql/runtime.py"", line 173, in resolve_is_ps_ready"
2021-10-26-12:24:33 microservice.do-host " agent_info = await loader.load(('esxcli.vxrail.agent.get', host, auth))"
2021-10-26-12:24:33 microservice.do-host " File ""/home/app/do_host_app/connector/esxcli/esxcli_dataloader.py"", line 63, in __run_aysnc"
2021-10-26-12:24:33 microservice.do-host " with self.__get_esxcli_client(host, auth) as client:"
2021-10-26-12:24:33 microservice.do-host " File ""/usr/local/venv/lib/python3.6/site-packages/do_common/esxcli/esxcli_client2.py"", line 34, in __enter__"
2021-10-26-12:24:33 microservice.do-host " si = client.get_service_instance()"
..................
..................
..................
2021-10-26-12:24:33 microservice.do-host " File ""/usr/customized/lib/python3.6/http/client.py"", line 1448, in connect"
2021-10-26-12:24:33 microservice.do-host " server_hostname=server_hostname)"
2021-10-26-12:24:33 microservice.do-host " File ""/usr/local/venv/lib/python3.6/site-packages/gevent/_ssl3.py"", line 120, in wrap_socket"
2021-10-26-12:24:33 microservice.do-host " _session=session)"
2021-10-26-12:24:33 microservice.do-host " File ""/usr/local/venv/lib/python3.6/site-packages/gevent/_ssl3.py"", line 312, in __init__"
2021-10-26-12:24:33 microservice.do-host " raise x"
2021-10-26-12:24:33 microservice.do-host " File ""/usr/local/venv/lib/python3.6/site-packages/gevent/_ssl3.py"", line 308, in __init__"
2021-10-26-12:24:33 microservice.do-host " self.do_handshake()"
2021-10-26-12:24:33 microservice.do-host " File ""/usr/local/venv/lib/python3.6/site-packages/gevent/_ssl3.py"", line 667, in do_handshake"
2021-10-26-12:24:33 microservice.do-host " self._sslobj.do_handshake()"
2021-10-26-12:24:33 microservice.do-host " File ""/usr/customized/lib/python3.6/ssl.py"", line 689, in do_handshake"
2021-10-26-12:24:33 microservice.do-host " self._sslobj.do_handshake()"
2021-10-26-12:24:33 microservice.do-host "ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)"
2021-10-26-12:24:33 microservice.do-host "[2021-10-26 12:24:33,319: INFO/MainProcess] idrac not ready."
Cause
This can occur when the VMware Certificate Authority (VMCA) is not used to manage the ESXi certificates. In which case, ESXi is using thumbprint host certificates, instead of default VMCA certificates.
vCenter and VxRail Manager use the customer certificates, but the node uses the self-signed certificates.
Resolution
Starting from VxRail version 7.0.240, if the ESXi certificate is managed by 'custom' mode in vCenter, must manually download vCenter certificates to hosts and VxRail Manager. This is required to establish a trusted connection, if 'certificate_verification_enable' is 'true'.
Using the VMware Certificate Authority (VMCA), to manage certificates, would resolve this issue.
Alternatively, to work around this issue, check and disable certificate verification:
Check the certificate verification:
curl -X GET --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock http://127.0.0.1/rest/vxm/internal/configservice/v1/configuration/keys/certificate_verification_enable
Disable certificate verification:
curl --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock -X PUT -H "Content-Type: application/json" http://127.0.0.1/rest/vxm/internal/configservice/v1/configuration/keys/certificate_verification_enable -d '{"value": "false"}'