VxRail: LCM failed at finalizing task VcCertTask.

When upgrading to VxRail 7.0.37x, LCM failed at the finalizing task VcCertTask with the following error message:  

com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.

From the lcm-web.log:

2022-08-04 23:56:38,433 INFO [LCM] [lcm-core-0] c.v.l.t.SimpleUpgradeTaskExecutor [SimpleUpgradeTaskExecutor.java:49] Start to execute task : VcCertTask.
2022-08-04 23:56:38,433 INFO [LCM] [lcm-core-0] c.v.l.t.SimpleUpgradeTaskExecutor [SimpleUpgradeTaskExecutor.java:71] Task VcCertTask, execute timeout: 1200 seconds.
2022-08-04 23:56:38,433 INFO [LCM] [upgrade-task-0] c.e.m.m.e.DynamicReloadableResourceBundleMessageSource [DynamicReloadableResourceBundleMessageSource.java:17] Try to translate message: Update vCenter certificate.

2022-08-04 23:56:39,421 INFO [LCM] [upgrade-task-0] c.d.v.l.d.p.r.d.NanoServiceDO [NanoServiceDO.java:63] Call nano-service to update VC cert in VXM. ==> unix.http://127.0.0.1/rest/vxm/internal/operation/v1/vxm/download-vc-certs/execute, req body: {"vc_info":{"host":"<vCenter-FQDN>","username":"xxxxxxxxxx","password":"******","port":443},"auto_accept_vc_cert":true}
2022-08-04 23:56:44,189 ERROR [LCM] [upgrade-task-0] c.v.c.c.VcCertService [VcCertService.java:39] Error happened when update VC cert in vxm.500 INTERNAL SERVER ERROR: "{"result": {"error": {"code": "E3100_Security_CERT_03", "params": ["<vCenter-FQDN>", "unable to get local issuer certificate"], "message": "Failed to validate the certificate of <vCenter-FQDN>, the reason is: unable to get local issuer certificate"}}}<EOL>"

2022-08-04 23:59:32,437 INFO [LCM] [lcm-core-0] c.e.m.m.u.r.BaseVirtualApplianceUpgradeProfilePool [BaseVirtualApplianceUpgradeProfilePool.java:749] Performing UPGRADE at progress 46!
2022-08-04 23:59:32,437 INFO [LCM] [lcm-core-0] c.d.v.l.d.p.r.d.VxrailSystemDO [VxrailSystemDO.java:134] operation status cache hit LcmUpgrade-824fec9a-2a8f-437a-b778-7d152ed5afae
2022-08-04 23:59:32,543 INFO [LCM] [lcm-core-0] c.d.v.l.d.p.r.d.VxrailSystemDO [VxrailSystemDO.java:180] operation status cache update 213 LcmUpgrade-824fec9a-2a8f-437a-b778-7d152ed5afae
2022-08-04 23:59:32,543 WARN [LCM] [lcm-core-0] c.e.m.m.u.s.VirtualApplianceUpgradeService [VirtualApplianceUpgradeService.java:994] LCM Exception occurred during upgrade for virtual appliance 70372188 - {}
com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.
        at com.vce.lcm.api.LCMServiceImpl.handleUpgradeException(LCMServiceImpl.java:1745)
        at com.vce.lcm.api.LCMServiceImpl.performUpgrade(LCMServiceImpl.java:450)
        at com.emc.mystic.manager.upgrade.service.VirtualApplianceUpgradeService.runUpgrade(VirtualApplianceUpgradeService.java:972)
        at com.emc.mystic.manager.upgrade.service.DeployAndUpgradeRestServiceImpl.performVirtualApplianceUpgrade(DeployAndUpgradeRestServiceImpl.java:1748)
        at com.emc.mystic.manager.upgrade.service.PureUpgradeRestServiceImpl.runUpgrade(PureUpgradeRestServiceImpl.java:98)
        at com.emc.mystic.manager.upgrade.service.DeployAndUpgradeRestServiceImpl.lambda$performUpgrade$1(DeployAndUpgradeRestServiceImpl.java:781)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vce.lcm.exception.LCMInternalException: Error happened when update VC cert in vxm for 5 times.
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:85)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:127)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:133)
        at com.vce.lcm.core.composite.finalize.CompositeUpgradeFinalizeService.finalize(CompositeUpgradeFinalizeService.java:53)
        at com.vce.lcm.api.LCMServiceImpl.performCompositeUpgrade(LCMServiceImpl.java:1544)
        at com.vce.lcm.api.LCMServiceImpl.performUpgrade(LCMServiceImpl.java:448)
        ... 9 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:72)
        ... 14 common frames omitted
Caused by: com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.
        at com.vce.commons.core.VcCertService.updateVcCertInVxm(VcCertService.java:51)
        at com.vce.lcm.core.composite.finalize.VcCertTask.perform(VcCertTask.java:38)
        at com.vce.lcm.task.UpgradeTask.execute(UpgradeTask.java:58)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.lambda$execute$0(SimpleUpgradeTaskExecutor.java:65)
        ... 4 common frames omitted

From the short.term.log: 

"2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.616574231Z stdout F common.exceptions.OperationException: OperationException: {""message"": ""Exception(b'depth=0 C = US, ST = WA, L = Seattle, O = xxxx, OU = IT, CN = <VC-FQDN>
nverify error:num=20:unable to get local issuer certificate
n140230863038272:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:
n',)"", ""bundle"": """", ""prefix"": ""common.exceptions.OperationException"", ""key"": """", ""field"": """", ""error_code"": ""0"", ""exit_code"": ""-1"", ""params"": []}" "2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.616582687Z stdout F 2022-08-04 23:59:01,616 [INFO] <Dummy-17:140695313773896> certificate.py validate_site() (184): error reason in Exception(b'depth=0 C = US, ST = WA, L = Seattle, O = xxxxxx, OU = IT, CN = <VC-FQDN>\nverify error:num=20:unable to get local issuer certificate\n140230863038272:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:\n',) start pos and end pos: 109, 167"
"2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.61659078Z stdout F 2022-08-04 23:59:01,616 [ERROR] <Dummy-17:140695313773896> atomic.py execute() (204): Step GenerateVCCerts raise exception {""code"": ""E3100_Security_CERT_03"", ""params"": [""<VC-FQDN>"", ""unable to get local issuer certificate""], ""message"": ""Failed to validate the certificate of <VC-FQDN>, the reason is: unable to get local issuer certificate""}"

The VC Certificate Chain is not complete.

For example, certs/lin/00de08c9.1 downloaded from VC is missing the .0 cert file.

1. Log in to VCSA and rename the *.1 cert file to *.0:

cd /etc/vmware-vpx/docRoot/certs/
mv 00de08c9.1 00de08c9.0

2. Log in to VxM and rename the *.1 cert file to *.0:

cd /var/lib/vmware-marvin/trust/lin
mv 00de08c9.1 00de08c9.0

3. Run the command below on VxRail Manager to verify if it succeeded. If so, similar outputs to the following appear. Then, re-try the upgrade.

openssl s_client -connect <VC-FQDN>:443 -verify_return_error -brief -CApath /var/lib/vmware-marvin/trust/lin -verify_hostname <VC-FQDN>

CONNECTION ESTABLISHED
Protocol version: TLSv1.2
Ciphersuite: ECDHE-RSA-AES256-GCM-SHA384
Peer certificate: CN = HOSTNAME, C = US
Hash used: SHA512
Signature type: RSA
Verification: OK
Verified peername: HOSTNAME
Supported Elliptic Curve Point Formats: uncompressed:ansiX962_compressed_prime:ansiX962_compressed_char2
Server Temp Key: ECDH, P-256, 256 bits