VxRail: vmware-vpxd service cannot be started after certificate update on vCenter

vCenter critical Services cannot be started after using vCenter Certificate Manager to reset all SSL Certificates.
After updating vCenter certificates, and checking running services, vmware-vpxd, and vmware-content-library are not running.

root@vcenter [ ~ ]# service-control --status --all
StartPending:
 vmware-vapi-endpoint vmware-vpxd-svcs
Running:
 applmgmt lwsmd vmafdd vmware-analytics vmware-cm vmware-eam vmware-postgres-archiver vmware-rhttpproxy vmware-statsmonitor vmware-vmon vmware-vpostgres vsphere-client vsphere-ui
Stopped:
 vmcam vmonapi vmware-certificatemanagement vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-pod vmware-rbd-watchdog vmware-sca vmware-sps vmware-topologysvc vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health vmware-vsm vsan-dps

Trying to start vmware-vpxd service you get an error similar to:

root@vcenter [ ~ ]# service-control --start vmware-vpxd
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd...
Error executing start on service vpxd. Details {
    "componentKey": null,
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "resolution": null,
    "problemId": null
}
Service-control failed. Error: {
    "componentKey": null,
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "resolution": null,
    "problemId": null
}

vCenter in /var/log/vmware/vmon/vmon-syslog.log:

2021-11-08T09:39:38.271729+00:00 warning vmon  ssl.CertificateError: hostname 'psc.xxxx.eg' doesn't match 'psc.xxx.xxx.eg'
2021-11-08T09:40:24.574482+00:00 notice vmon  <rhttpproxy-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-rhttpproxy/rhttpproxy-vmon-apihealth.py
2021-11-08T09:40:24.574698+00:00 notice vmon  <vapi-endpoint> Skip service health check. State STOPPED, Curr request 0
2021-11-08T09:40:24.574854+00:00 notice vmon  <vcha> Skip service health check. State STOPPED, Curr request 0
2021-11-08T09:40:24.574997+00:00 notice vmon  <vmware-vpostgres-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-vpostgres -f /dev/shm/vmware-postgres-health-status.xml
2021-11-08T09:40:24.816621+00:00 notice vmon  <vpxd-svcs> Skip service health check. State STOPPED, Curr request 0

vCenter in /storage/log/vmware/vpxd-svcs/vpxd-svcs.log:

2021-11-08T07:37:03.716Z [Thread-9  WARN  com.vmware.cis.server.util.impl.InitPoolTask  opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.VpxdClientException at attempt 19
2021-11-08T07:37:23.725Z [Thread-9  INFO  com.vmware.vim.sso.client.impl.SiteAffinityServiceDiscovery  opId=] Site affinity is disabled
2021-11-08T07:37:23.750Z [Thread-9  ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl  opId=] Error communicating to the remote server https://urldefense.com/v3/__https://psc.xxx.eg/sts/STSService/vsphere.local__;!!LpKI!zizBGmTnALcYXLnWIITladCn-NgryRjl_R2ufCq4LGkURpqvXMuRNpET1ze8_DkPVm9P9UE4Cw$ [psc[.]xxx[.]gov[.]eg]
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching psc.xxx.eg found.

1. Check if there is a DNS record mismatch and DNS connectivity. 
2. Ensure that each VM IP (vCenter, PSC, and VXM) has a matching single FQDN within the domain. 
3. If there are any duplicates NS records for the same IP, then the unnecessary NS records should be deleted.

Note: To check DNS Server configuration on each Redhat-based Linux VM see /etc/resolv.conf .

4. Ensure that vCenter certificates are not expired by running the following command line on the vCenter VM command-line interface:

root@vcenter [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

The command output should be similar to the following output:

STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Oct 29 12:42:18 2031 GMT
STORE TRUSTED_ROOTS
Alias : 50932a13985d9c33aa386868f12ba0da57f5eaff
            Not After : Oct 21 21:28:25 2029 GMT
Alias : 32c621e354a974e2b0c31abe17fdf4beedba113d
            Not After : Oct 29 11:29:49 2031 GMT
Alias : 342b168f6ab63a8facde75e187f8ab0a22a857e7
            Not After : Oct 29 12:42:18 2031 GMT
Alias : f5e20d029d8d7e3cda69079fcddc0cb5aa4d00b3
            Not After : Nov  3 10:41:37 2031 GMT
STORE TRUSTED_ROOT_CRLS
Alias : c4268b466f6dcefaff95ea32c85d9498819a9d9c
Alias : fda5f1c260738901752aaa0bcea5758d82e42ee6
Alias : e843a42ee0a5903a099c21cabf2d8b14747adf1e
Alias : f3721159c59455451478b401f80d23f996f40322
STORE machine
Alias : machine
            Not After : Oct 29 12:42:18 2031 GMT
STORE vsphere-webclient
Alias : vsphere-webclient
            Not After : Oct 29 12:42:18 2031 GMT
STORE vpxd
Alias : vpxd
            Not After : Oct 29 12:42:18 2031 GMT
STORE vpxd-extension
Alias : vpxd-extension
            Not After : Oct 29 12:42:18 2031 GMT
STORE SMS
Alias : sms_self_signed
            Not After : Oct 27 21:47:48 2029 GMT
STORE APPLMGMT_PASSWORD
STORE data-encipherment
Alias : data-encipherment
            Not After : Oct 21 21:28:25 2029 GMT
STORE BACKUP_STORE

5. Should there be any expired certificates, refer to KB VxRail: Unable to log in to vCenter due to expired certificates (Customer Correctable) to update expired certificates.

6. On the PSC, compare the local hostname with the name that is stored in MACHINE_SS

 /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
Output should be similar to following:
psc.xxx.eg

 /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SS
Output should be similar to following:
X509v3 Subject Alternative Name:
                email:email@acme.com, DNS:psc.xxx.eg

7. Compare the output above. If there is a mismatch, for example DNS:psc.xxx.eg.xxx.eg that was cached on the DNS Server before editing the DNS records, then proceed with the next steps.

• SSH to PSC VM and initiate the certificate-manager by running the following command

  root@psc [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager

• Use option 8 -> 8. Reset all Certificates.
• Follow this procedure:
  • Confirm “Do you want to generate all certificates using configuration file: Option[Y/N] "
  • Enter credentials
  • Enter values
  • Leave "IPAddress" field empty
  • Enter FQDN of PSC into "Hostname"
  • VMCA "Name" field is the name of the new Root CA being created (e.g. "VxRail CA")
  • Confirm "Continue operation: Option[Y/N] ?"
  • Confirm "Continue operation : Option[Y/N] ?"
• Restart all services on both PSC and vCenter

  service-control --stop --all
  service-control --start --all

8. Ensure that vCenter critical services are up and running:
 

root@vcenter [ ~ ]# service-control --status --all
Running:
 applmgmt lwsmd vmafdd vmonapi vmware-analytics vmware-certificatemanagement vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-topologysvc vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
 vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-updatemgr vmware-vcha vsan-dps