VxRail: LCM kunne ikke afslutte opgaven VcCertTask.

Ved opgradering til VxRail 7.0.37x mislykkedes LCM ved færdiggørelsen af opgaven VcCertTask med følgende fejlmeddelelse:  

com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.

Fra lcm-web.log:

2022-08-04 23:56:38,433 INFO [LCM] [lcm-core-0] c.v.l.t.SimpleUpgradeTaskExecutor [SimpleUpgradeTaskExecutor.java:49] Start to execute task : VcCertTask.
2022-08-04 23:56:38,433 INFO [LCM] [lcm-core-0] c.v.l.t.SimpleUpgradeTaskExecutor [SimpleUpgradeTaskExecutor.java:71] Task VcCertTask, execute timeout: 1200 seconds.
2022-08-04 23:56:38,433 INFO [LCM] [upgrade-task-0] c.e.m.m.e.DynamicReloadableResourceBundleMessageSource [DynamicReloadableResourceBundleMessageSource.java:17] Try to translate message: Update vCenter certificate.

2022-08-04 23:56:39,421 INFO [LCM] [upgrade-task-0] c.d.v.l.d.p.r.d.NanoServiceDO [NanoServiceDO.java:63] Call nano-service to update VC cert in VXM. ==> unix.http://127.0.0.1/rest/vxm/internal/operation/v1/vxm/download-vc-certs/execute, req body: {"vc_info":{"host":"<vCenter-FQDN>","username":"xxxxxxxxxx","password":"******","port":443},"auto_accept_vc_cert":true}
2022-08-04 23:56:44,189 ERROR [LCM] [upgrade-task-0] c.v.c.c.VcCertService [VcCertService.java:39] Error happened when update VC cert in vxm.500 INTERNAL SERVER ERROR: "{"result": {"error": {"code": "E3100_Security_CERT_03", "params": ["<vCenter-FQDN>", "unable to get local issuer certificate"], "message": "Failed to validate the certificate of <vCenter-FQDN>, the reason is: unable to get local issuer certificate"}}}<EOL>"

2022-08-04 23:59:32,437 INFO [LCM] [lcm-core-0] c.e.m.m.u.r.BaseVirtualApplianceUpgradeProfilePool [BaseVirtualApplianceUpgradeProfilePool.java:749] Performing UPGRADE at progress 46!
2022-08-04 23:59:32,437 INFO [LCM] [lcm-core-0] c.d.v.l.d.p.r.d.VxrailSystemDO [VxrailSystemDO.java:134] operation status cache hit LcmUpgrade-824fec9a-2a8f-437a-b778-7d152ed5afae
2022-08-04 23:59:32,543 INFO [LCM] [lcm-core-0] c.d.v.l.d.p.r.d.VxrailSystemDO [VxrailSystemDO.java:180] operation status cache update 213 LcmUpgrade-824fec9a-2a8f-437a-b778-7d152ed5afae
2022-08-04 23:59:32,543 WARN [LCM] [lcm-core-0] c.e.m.m.u.s.VirtualApplianceUpgradeService [VirtualApplianceUpgradeService.java:994] LCM Exception occurred during upgrade for virtual appliance 70372188 - {}
com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.
        at com.vce.lcm.api.LCMServiceImpl.handleUpgradeException(LCMServiceImpl.java:1745)
        at com.vce.lcm.api.LCMServiceImpl.performUpgrade(LCMServiceImpl.java:450)
        at com.emc.mystic.manager.upgrade.service.VirtualApplianceUpgradeService.runUpgrade(VirtualApplianceUpgradeService.java:972)
        at com.emc.mystic.manager.upgrade.service.DeployAndUpgradeRestServiceImpl.performVirtualApplianceUpgrade(DeployAndUpgradeRestServiceImpl.java:1748)
        at com.emc.mystic.manager.upgrade.service.PureUpgradeRestServiceImpl.runUpgrade(PureUpgradeRestServiceImpl.java:98)
        at com.emc.mystic.manager.upgrade.service.DeployAndUpgradeRestServiceImpl.lambda$performUpgrade$1(DeployAndUpgradeRestServiceImpl.java:781)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vce.lcm.exception.LCMInternalException: Error happened when update VC cert in vxm for 5 times.
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:85)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:127)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:133)
        at com.vce.lcm.core.composite.finalize.CompositeUpgradeFinalizeService.finalize(CompositeUpgradeFinalizeService.java:53)
        at com.vce.lcm.api.LCMServiceImpl.performCompositeUpgrade(LCMServiceImpl.java:1544)
        at com.vce.lcm.api.LCMServiceImpl.performUpgrade(LCMServiceImpl.java:448)
        ... 9 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:72)
        ... 14 common frames omitted
Caused by: com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.
        at com.vce.commons.core.VcCertService.updateVcCertInVxm(VcCertService.java:51)
        at com.vce.lcm.core.composite.finalize.VcCertTask.perform(VcCertTask.java:38)
        at com.vce.lcm.task.UpgradeTask.execute(UpgradeTask.java:58)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.lambda$execute$0(SimpleUpgradeTaskExecutor.java:65)
        ... 4 common frames omitted

Fra short.term.log: 

"2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.616574231Z stdout F common.exceptions.OperationException: OperationException: {""message"": ""Exception(b'depth=0 C = US, ST = WA, L = Seattle, O = xxxx, OU = IT, CN = <VC-FQDN>
nverify error:num=20:unable to get local issuer certificate
n140230863038272:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:
n',)"", ""bundle"": """", ""prefix"": ""common.exceptions.OperationException"", ""key"": """", ""field"": """", ""error_code"": ""0"", ""exit_code"": ""-1"", ""params"": []}" "2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.616582687Z stdout F 2022-08-04 23:59:01,616 [INFO] <Dummy-17:140695313773896> certificate.py validate_site() (184): error reason in Exception(b'depth=0 C = US, ST = WA, L = Seattle, O = xxxxxx, OU = IT, CN = <VC-FQDN>\nverify error:num=20:unable to get local issuer certificate\n140230863038272:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:\n',) start pos and end pos: 109, 167"
"2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.61659078Z stdout F 2022-08-04 23:59:01,616 [ERROR] <Dummy-17:140695313773896> atomic.py execute() (204): Step GenerateVCCerts raise exception {""code"": ""E3100_Security_CERT_03"", ""params"": [""<VC-FQDN>"", ""unable to get local issuer certificate""], ""message"": ""Failed to validate the certificate of <VC-FQDN>, the reason is: unable to get local issuer certificate""}"

VC-certifikatkæden er ikke komplet.

For eksempel mangler certs/lin/00de08c9.1 , der er downloadet fra VC, .0-certfilen.

1. Log på VCSA, og omdøb *.1 CERT-fil til *.0:

cd /etc/vmware-vpx/docRoot/certs/
mv 00de08c9.1 00de08c9.0

2. Log på VxM, og omdøb *.1 CERT-fil til *.0:

cd /var/lib/vmware-marvin/trust/lin
mv 00de08c9.1 00de08c9.0

3. Kør kommandoen nedenfor på VxRail Manager for at kontrollere, om den lykkedes. Hvis det er tilfældet, vises lignende output som følgende. Prøv derefter at opgradere igen.

openssl s_client -connect <VC-FQDN>:443 -verify_return_error -brief -CApath /var/lib/vmware-marvin/trust/lin -verify_hostname <VC-FQDN>

CONNECTION ESTABLISHED
Protocol version: TLSv1.2
Ciphersuite: ECDHE-RSA-AES256-GCM-SHA384
Peer certificate: CN = HOSTNAME, C = US
Hash used: SHA512
Signature type: RSA
Verification: OK
Verified peername: HOSTNAME
Supported Elliptic Curve Point Formats: uncompressed:ansiX962_compressed_prime:ansiX962_compressed_char2
Server Temp Key: ECDH, P-256, 256 bits