วีเอ็กซ์เรล: LCM ล้มเหลวในการสิ้นสุดงาน VcCertTask

เมื่ออัปเกรดเป็น VxRail 7.0.37x LCM ล้มเหลวที่งาน VcCertTask ที่สิ้นสุดพร้อมข้อความแสดงข้อผิดพลาดต่อไปนี้:  

com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.

จาก lcm-web.log:

2022-08-04 23:56:38,433 INFO [LCM] [lcm-core-0] c.v.l.t.SimpleUpgradeTaskExecutor [SimpleUpgradeTaskExecutor.java:49] Start to execute task : VcCertTask.
2022-08-04 23:56:38,433 INFO [LCM] [lcm-core-0] c.v.l.t.SimpleUpgradeTaskExecutor [SimpleUpgradeTaskExecutor.java:71] Task VcCertTask, execute timeout: 1200 seconds.
2022-08-04 23:56:38,433 INFO [LCM] [upgrade-task-0] c.e.m.m.e.DynamicReloadableResourceBundleMessageSource [DynamicReloadableResourceBundleMessageSource.java:17] Try to translate message: Update vCenter certificate.

2022-08-04 23:56:39,421 INFO [LCM] [upgrade-task-0] c.d.v.l.d.p.r.d.NanoServiceDO [NanoServiceDO.java:63] Call nano-service to update VC cert in VXM. ==> unix.http://127.0.0.1/rest/vxm/internal/operation/v1/vxm/download-vc-certs/execute, req body: {"vc_info":{"host":"<vCenter-FQDN>","username":"xxxxxxxxxx","password":"******","port":443},"auto_accept_vc_cert":true}
2022-08-04 23:56:44,189 ERROR [LCM] [upgrade-task-0] c.v.c.c.VcCertService [VcCertService.java:39] Error happened when update VC cert in vxm.500 INTERNAL SERVER ERROR: "{"result": {"error": {"code": "E3100_Security_CERT_03", "params": ["<vCenter-FQDN>", "unable to get local issuer certificate"], "message": "Failed to validate the certificate of <vCenter-FQDN>, the reason is: unable to get local issuer certificate"}}}<EOL>"

2022-08-04 23:59:32,437 INFO [LCM] [lcm-core-0] c.e.m.m.u.r.BaseVirtualApplianceUpgradeProfilePool [BaseVirtualApplianceUpgradeProfilePool.java:749] Performing UPGRADE at progress 46!
2022-08-04 23:59:32,437 INFO [LCM] [lcm-core-0] c.d.v.l.d.p.r.d.VxrailSystemDO [VxrailSystemDO.java:134] operation status cache hit LcmUpgrade-824fec9a-2a8f-437a-b778-7d152ed5afae
2022-08-04 23:59:32,543 INFO [LCM] [lcm-core-0] c.d.v.l.d.p.r.d.VxrailSystemDO [VxrailSystemDO.java:180] operation status cache update 213 LcmUpgrade-824fec9a-2a8f-437a-b778-7d152ed5afae
2022-08-04 23:59:32,543 WARN [LCM] [lcm-core-0] c.e.m.m.u.s.VirtualApplianceUpgradeService [VirtualApplianceUpgradeService.java:994] LCM Exception occurred during upgrade for virtual appliance 70372188 - {}
com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.
        at com.vce.lcm.api.LCMServiceImpl.handleUpgradeException(LCMServiceImpl.java:1745)
        at com.vce.lcm.api.LCMServiceImpl.performUpgrade(LCMServiceImpl.java:450)
        at com.emc.mystic.manager.upgrade.service.VirtualApplianceUpgradeService.runUpgrade(VirtualApplianceUpgradeService.java:972)
        at com.emc.mystic.manager.upgrade.service.DeployAndUpgradeRestServiceImpl.performVirtualApplianceUpgrade(DeployAndUpgradeRestServiceImpl.java:1748)
        at com.emc.mystic.manager.upgrade.service.PureUpgradeRestServiceImpl.runUpgrade(PureUpgradeRestServiceImpl.java:98)
        at com.emc.mystic.manager.upgrade.service.DeployAndUpgradeRestServiceImpl.lambda$performUpgrade$1(DeployAndUpgradeRestServiceImpl.java:781)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vce.lcm.exception.LCMInternalException: Error happened when update VC cert in vxm for 5 times.
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:85)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:127)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:133)
        at com.vce.lcm.core.composite.finalize.CompositeUpgradeFinalizeService.finalize(CompositeUpgradeFinalizeService.java:53)
        at com.vce.lcm.api.LCMServiceImpl.performCompositeUpgrade(LCMServiceImpl.java:1544)
        at com.vce.lcm.api.LCMServiceImpl.performUpgrade(LCMServiceImpl.java:448)
        ... 9 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:72)
        ... 14 common frames omitted
Caused by: com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times.
        at com.vce.commons.core.VcCertService.updateVcCertInVxm(VcCertService.java:51)
        at com.vce.lcm.core.composite.finalize.VcCertTask.perform(VcCertTask.java:38)
        at com.vce.lcm.task.UpgradeTask.execute(UpgradeTask.java:58)
        at com.vce.lcm.task.SimpleUpgradeTaskExecutor.lambda$execute$0(SimpleUpgradeTaskExecutor.java:65)
        ... 4 common frames omitted

จาก short.term.log: 

"2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.616574231Z stdout F common.exceptions.OperationException: OperationException: {""message"": ""Exception(b'depth=0 C = US, ST = WA, L = Seattle, O = xxxx, OU = IT, CN = <VC-FQDN>
nverify error:num=20:unable to get local issuer certificate
n140230863038272:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:
n',)"", ""bundle"": """", ""prefix"": ""common.exceptions.OperationException"", ""key"": """", ""field"": """", ""error_code"": ""0"", ""exit_code"": ""-1"", ""params"": []}" "2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.616582687Z stdout F 2022-08-04 23:59:01,616 [INFO] <Dummy-17:140695313773896> certificate.py validate_site() (184): error reason in Exception(b'depth=0 C = US, ST = WA, L = Seattle, O = xxxxxx, OU = IT, CN = <VC-FQDN>\nverify error:num=20:unable to get local issuer certificate\n140230863038272:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:\n',) start pos and end pos: 109, 167"
"2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.61659078Z stdout F 2022-08-04 23:59:01,616 [ERROR] <Dummy-17:140695313773896> atomic.py execute() (204): Step GenerateVCCerts raise exception {""code"": ""E3100_Security_CERT_03"", ""params"": [""<VC-FQDN>"", ""unable to get local issuer certificate""], ""message"": ""Failed to validate the certificate of <VC-FQDN>, the reason is: unable to get local issuer certificate""}"

ห่วงโซ่ใบรับรอง VC ไม่สมบูรณ์

ตัวอย่างเช่น certs/lin/00de08c9.1 ที่ดาวน์โหลดจาก VC ไม่มีไฟล์ใบรับรอง .0

1. เข้าสู่ระบบ VCSA และเปลี่ยนชื่อ *.1 ไฟล์ cert ไปยัง *.0:

cd /etc/vmware-vpx/docRoot/certs/
mv 00de08c9.1 00de08c9.0

2. เข้าสู่ระบบ VxM และเปลี่ยนชื่อ *.1 ไฟล์ cert ไปยัง *.0:

cd /var/lib/vmware-marvin/trust/lin
mv 00de08c9.1 00de08c9.0

3. เรียกใช้คําสั่งด้านล่างบน VxRail Manager เพื่อตรวจสอบว่าสําเร็จหรือไม่ ถ้าเป็นเช่นนั้น ผลลัพธ์ที่คล้ายกันต่อไปนี้จะปรากฏขึ้น จากนั้นลองอัปเกรดอีกครั้ง

openssl s_client -connect <VC-FQDN>:443 -verify_return_error -brief -CApath /var/lib/vmware-marvin/trust/lin -verify_hostname <VC-FQDN>

CONNECTION ESTABLISHED
Protocol version: TLSv1.2
Ciphersuite: ECDHE-RSA-AES256-GCM-SHA384
Peer certificate: CN = HOSTNAME, C = US
Hash used: SHA512
Signature type: RSA
Verification: OK
Verified peername: HOSTNAME
Supported Elliptic Curve Point Formats: uncompressed:ansiX962_compressed_prime:ansiX962_compressed_char2
Server Temp Key: ECDH, P-256, 256 bits