VxRail: Health Check Fails For Test 'vxm_cert'
Summary: Health-check returns an event indicating that VxVerify found an issue with the VxRail Manager certificate.
Symptoms
This test checks the VxRail Manager certificate.
Checks performed:
- Is the certificate self-signed?
- For a self-signed certificate:
- Does ca.cnf contain "DNS" entry with the "FQDN"?
- Does the SAN include the "FQDN"?
- Has the certificate expired?
- Does the modulus of the server.crt match the modulus of the server.key
| Test Result | Result code | Result Interpretation |
|---|---|---|
|
Pass |
0 |
No issue with VxRail Manager certificate found. |
|
Warning |
1 |
VxRail Manager certificate issues found. See Cause section below. |
|
Failure |
2 |
VxRail Manager certificate issues found. Cause section below. |
|
Critical |
3 |
This test has no critical result. |
Cause
The 'vxm_cert' test checks the VxRail Manager certificate.
- For Warning, the result indicates the modulus of the server.crt does not match the modulus of the server.key.
- For Failure, the result indicates that the nonself-signed VxRail Manager certificate has expired and must be renewed prior to an upgrade attempt.
Acronym list:
- Domain Name System (DNS)
- Fully Qualified Domain Name (FQDN)
Files being used during the test:
- /etc/vmware-marvin/ssl/server.crt (VxRail Manager certificate)
- /etc/vmware-marvin/ssl/rootcert.crt (CA chain for VxRail Manager certificate)
- /etc/vmware-marvin/ssl/server.api.gateway.crt (copy of VxRail Manager certificate)
- /etc/vmware-marvin/ssl/ca.cnf (configuration file for self-signed VxRail Manager certificates)
- server.key
- Pre 7.0.400 - /etc/vmware-marvin/ssl/server.key
- 7.0.400+ read from
kubectl config
Test performed:
- Is server.crt self-signed?
- Yes:
- Check if server.crt, server.api.gateway.crt, and rootcert.crt are identical
- No: This fails the 'vxm_cert' test and no other VxRail Manager certificate checks are performed
- Check if the reverse DNS of VxRail Manager IP (FQDN) is contained in ca.cnf
- No: Below result expected if DNS entry missing or only short name found in ca.cnf file
| VxRM | Warning 198406 | vxm_cert: VXM FQDN missing/incomplete in ca.cnf .|
- No: Below result expected if DNS entry missing or only short name found in ca.cnf file
- Check if server.crt, server.api.gateway.crt, and rootcert.crt are identical
- No:
- Check if expired
- Yes: Fails the test
- The
vxverifyresult table shows| VxRM | Fail 198406 | vxm_cert: FAIL - found expired VXM custom certificate .|
- Check if expired
- Yes:
- Check if server.crt is trusted using rootcert.crt
- No: Flag the issue
- Has server.crt expired?
- Yes: Flag the issue
- Check if reverse DNS of VxRail Manager IP (FQDN) is found in server.crt Subject Alternative Names (SAN) list
- No: Flag the issue
- Check if the Modulus of server.crt matches the modulus of server.key
- No: Certificate replacement procedure was not followed correctly
- The
vxverifyresult table shows| VxRM | Warning 198406 | vxm_cert: Modulus mismatch between server.crt and server.key. .|
- The
- No: Certificate replacement procedure was not followed correctly
- If any flags are raised, the test returns a 'FAIL'
The vxv.log contains "VXM certificate validity check - FAIL".
Fix:
- If the certificate is self-signed, see Resolution 1.
- For nonself-signed certificates, see Resolution 2.
All info is found in vxv.log, use grep command to search for 'vxm_cert'.
Verbose log Example from Lab:
vcluster101-vxrm:/home/mystic # less vxv.log | grep vxm_cert 2022-11-01 14:34:55-INFO [vxm_cert] Testing VXM certificate validity check 2022-11-01 14:34:55-INFO [dns_fqdn_ip] 172.168.10.50 gethostbyaddr returning FQDN vcluster101-vxrm.vv001.local 2022-11-01 14:34:55-INFO [vxm_cert] VXM certificate self-signed check: True 2022-11-01 14:34:55-DEBUG [vxm_cert] -> modulus of certificate read 2022-11-01 14:34:55-DEBUG [vxm_cert] -> modulus of certificate read 2022-11-01 14:34:55-DEBUG [vxm_cert] -> modulus of certificate read 2022-11-01 14:34:55-DEBUG [vxm_cert] -> modulus of certificate read 2022-11-01 14:34:55-INFO [vxm_cert] VXM self-sign certs pass: server, api gateway and rootcert content identical! 2022-11-01 14:34:55-INFO [vxm_cert] Check if FQDN 'vcluster101-vxrm.vv001.local' is found in ca.cnf 2022-11-01 14:34:55-DEBUG [vxm_cert] ca.cnf DNS SAN: ['DNS.1 = vcluster101-vxrm.vv001.local'] 2022-11-01 14:34:55-INFO [vxm_cert] FQDN 'vcluster101-vxrm.vv001.local' found in 'vcluster101-vxrm.vv001.local': True 2022-11-01 14:34:55-INFO [vxm_cert] VXM FQDN in ca.cnf : True 2022-11-01 14:34:55-INFO [vxm_cert] -> Certificate vcluster101-vxrm.vv001.local, self-signed: True, Issuer: 'vcluster101-vxrm.vv001.local', expired: False 2022-11-01 14:34:55-INFO [vxm_cert] -> Found 1 certificate(s) in chain 2022-11-01 14:34:55-INFO [vxm_cert] Certificate CN='vcluster101-vxrm.vv001.local', Issuer: 'vcluster101-vxrm.vv001.local', SAN: ['vcluster101-vxrm.vv001.local'], self-signed: True, expired: False 2022-11-01 14:34:55-INFO [vxm_cert] Certificate is valid against provided trust chain! 2022-11-01 14:34:55-INFO [vxm_cert] -> No expired certs found. 2022-11-01 14:34:55-INFO [vxm_cert] Certificate will expire in 761 days (2024-12-02 13:52:43) 2022-11-01 14:34:55-INFO [vxm_cert] Certificate version: v3 2022-11-01 14:34:55-DEBUG [vxm_cert] .. cert has SAN: ['vcluster101-vxrm.vv001.local'] 2022-11-01 14:34:55-INFO [vxm_cert] FQDN 'vcluster101-vxrm.vv001.local' found in 'vcluster101-vxrm.vv001.local': True 2022-11-01 14:34:55-DEBUG [vxm_cert] -> modulus of certificate read 2022-11-01 14:34:55-DEBUG [vxm_cert] VXM cert modulus : 737606798658210345552395023242313575498349980168484844388104681760467073883011018654519552113639865930937060725956161028970536990609868865205746383328200764864894981142328568328657935452055209671631324383458709981382958650198551830486 143325734179025886034284279986239071265103840351119056608085586215383171845712455515672064475301529905881788883636903724318736502128949857904091221042682404407112606348900344815441050547380775230077999536636557738641283900720744678777412009298019876243746380791172035001376949864207929904577617722929532669596 969605085603057343287908041826751812176348357320164911109115951288171761502258221092274323790009201505777693273716701833573608884839819707266105827978472998034615720643262714593852230449266964049807291368579426111835780256915666430267382209552126939824155889043819078701590538547302381769757869412847241459155 555459521172889268714285038687298213572400125782935108262701577461219143109838771620714629597117036018317636913785163251545629790329095499202728776464552170726432853116188495914370652200727134390240043211871565088281127910113500064626019413290428798648699326217103710460104457195146414570080660613595135695795 756289083984604070476681971839564047103254900324978466550121916659163629 2022-11-01 14:34:55-DEBUG [vxm_cert] -> modulus of private key read 2022-11-01 14:34:55-DEBUG [vxm_cert] VXM private key modulus : 737606798658210345552395023242313575498349980168484844388104681760467073883011018654519552113639865930937060725956161028970536990609868865205746383328200764864894981142328568328657935452055209671631324383458709981382958650198551830486 143325734179025886034284279986239071265103840351119056608085586215383171845712455515672064475301529905881788883636903724318736502128949857904091221042682404407112606348900344815441050547380775230077999536636557738641283900720744678777412009298019876243746380791172035001376949864207929904577617722929532669596 9696050856030573432879080418267518121763483573201649111091159512881717615022582210922743237900092015057776932737167018335736088848398197072661 05827978472998034615720643262714593852230449266964049807291368579426111835780256915666430267382209552126939824155889043819078701590538547302381769757869412847241459155555459521172889268714285038687298213572400125782935108262701577461219143109838771620714629597117036018317636913785163251545629790329095499202728776464552170726432853116188495914370652200727134390240043211871565088281127910113500064626019413290428798648699326217103710460104457195146414570080660613595135695795756289083984604070476681971839564047103254900324978466550121916659163629 2022-11-01 14:34:55-INFO [vxm_cert] VXM certificate self-signed: True; Expired: False; Trusted: True; SAN: ['vcluster101-vxrm.vv001.local']; Modulus: True 2022-11-01 14:34:55-DEBUG [vxm_cert] Has certificate extended usage (serveAuth & clientAuth) configured check.... 2022-11-01 14:34:55-DEBUG [vxm_cert] -> Found OID: 1.3.6.1.5.5.7.3.1 2022-11-01 14:34:55-DEBUG [vxm_cert] -> Found OID: 1.3.6.1.5.5.7.3.2 2022-11-01 14:34:55-INFO [vxm_cert] -> Iterated through all extended usage OIDs 2022-11-01 14:34:55-INFO [vxm_cert] VXM has serverAuth OID : True 2022-11-01 14:34:55-INFO [vxm_cert] VXM has clientAuth OID : True 2022-11-01 14:34:55-INFO [vxm_cert] VXM cert OID list: : <Extensions([<Extension(oid=<ObjectIdentifier(oid=2.5.29.19, name=basicConstraints)>, critical=False, value=<BasicConstraints(ca=False, path_length=None)>)>, <Extension(oid=<ObjectIdentifier(oid=2.5.29.37, name=extendedKeyUsage)>, critical=False, value=<ExtendedKeyUsage([<ObjectIdentifier(oid=1.3.6.1.5.5.7.3.1, name=serverAuth)>, <ObjectIdentifier(oid=1.3.6.1.5.5.7.3.2, name=clientAuth)>])>)>, <Extension(oid=<ObjectIdentifier(oid=2.5.29.17, name=subjectAltName)>, critical=False, value=<SubjectAlternativeName(<GeneralNames([<DNSName(value='vcluster101-vxrm.vv001.local')>, <IPAddress(value=172.168.10.50)>])>)>)>])> 2022-11-01 14:34:55-INFO [vxm_cert] VXM certificate validity check - PASS vcluster101-vxrm:/home/mystic/ #
In the above example, we see a "PASS" message for the certificate validity check.
Resolution
Resolution 1 (for self-signed certificates)
Review the More Info section to see if a quick fix is available.
Follow the steps from KB 20625 VxRail: How to apply for a new certificate for VxRail Manager to create a new VxRail Manager certificate. (A Dell Support account is required to view this article)
Resolution 2 (for external certificates)
Apply Resolution 1 and then use VxRail plug-in to replace the certificate with a nonself-signed certificate following the certificate requirements from KB 20625 for self-signed certs:
- The certificate should be Version 3 containing extended usage (serverAuth, clientAuth)
[ v3_req ] basicConstraints = CA:false extendedKeyUsage = serverAuth, clientAuth subjectAltName = @alt_names
- The common Name should be FQDN of VxRail Manager
commonName = <vxm-fqnd>
- SAN (alt names) should include VxRail Manager FQDN and IP
[ alt_names ] DNS.1 = <vxm-fqnd> IP.1 = <vxm-ip>
Additional Information:
Depending on the cause found in vxv.log:
- The certificate is self-signed but not matching server.api.gateway.crt and rootcert.crt
- vxv.log entry found:
WARNING [vxm_cert] VXM certs issue!!! Check /etc/vmware-marvin/ssl *.crt files: server, server.api.gateway and rootcert - Must be identical!
- Fix:
Check if server.crt is valid- Run:
Pre 7.0.350openssl x509 -in /etc/vmware-marvin/ssl/server.crt -nocert -serial -subject -issuer -ext subjectAltName -startdate -enddate -fingerprint -sha1
7.0.350+openssl x509 -in /etc/vmware-marvin/ssl/server.crt -nocert -serial -subject -issuer -ext subjectAltName -startdate -enddate -fingerprint -sha256
- Expected result:
- The Serial of the certificate
- A line containing the FQDN in lines 'subject', issuer' and DNS from 'X509v3 Subject Alternative Name'
- Lines showing the validity period containing 'notBefore' and 'notAfter'
- The fingerprint of the certificate
- Example from Lab:
vcluster101-vxrm:/home/mystic # openssl x509 -in /etc/vmware-marvin/ssl/server.crt -nocert -subject -issuer -ext subjectAltName -startdate -enddate -fingerprint -sha256 subject=C = US, ST = local, L = vsphere, O = VMware, OU = VxRailApplianceServer, CN = vcluster101-vxrm.vv001.local issuer=C = US, ST = local, L = vsphere, O = VMware, OU = VxRailApplianceServer, CN = vcluster101-vxrm.vv001.local X509v3 Subject Alternative Name: DNS:vcluster101-vxrm.vv001.local, IP Address:172.168.10.50 notBefore=Feb 25 14:47:14 2022 GMT notAfter=May 15 14:47:14 2024 GMT SHA256 Fingerprint=A3:7E:D6:E9:44:AE:31:7A:5B:41:AC:1F:F5:BF:D8:02:2E:E8:A4:07:D9:99:05:EC:A0:85:7C:76:29:FA:5F:CF vcluster101-vxrm:/home/mystic #
- If server.crt is OK (correct FQDN, IP, date./time of test within validity period), compare against server.api.gateway.crt and rootcert.crt
- Run the openssl command against server.api.gateway.crt and rootcert.crt
- The expected output is a different output than against the server.crt
- Fix copying the server.crt over the offending file
- Example from Lab
- Check the files:
vcluster101-vxrm:/home/mystic # openssl x509 -in /etc/vmware-marvin/ssl/server.crt -serial -nocert -subject -issuer -ext subjectAltName -startdate -enddate -fingerprint -sha256 serial=F2262B1EB62DFF22 subject=C = US, ST = local, L = vsphere, O = VMware, OU = VxRailApplianceServer, CN = vcluster101-vxrm.vv001.local issuer=C = US, ST = local, L = vsphere, O = VMware, OU = VxRailApplianceServer, CN = vcluster101-vxrm.vv001.local X509v3 Subject Alternative Name: DNS:vcluster101-vxrm.vv001.local, IP Address:172.168.10.50 notBefore=Feb 25 14:47:14 2022 GMT notAfter=May 15 14:47:14 2024 GMT SHA256 Fingerprint=A3:7E:D6:E9:44:AE:31:7A:5B:41:AC:1F:F5:BF:D8:02:2E:E8:A4:07:D9:99:05:EC:A0:85:7C:76:29:FA:5F:CF vcluster101-vxrm:/home/mystic # openssl x509 -in /etc/vmware-marvin/ssl/server.api.gateway.crt -serial -nocert -subject -issuer -ext subjectAltName -startdate -enddate -fingerprint -sha256 serial=DEE85F0B1E5964DC subject=C = US, ST = local, L = vsphere, O = VMware, OU = VxRailApplianceServer, CN = vcluster101-vxrm.vv001.local issuer=C = US, ST = local, L = vsphere, O = VMware, OU = VxRailApplianceServer, CN = vcluster101-vxrm.vv001.local X509v3 Subject Alternative Name: DNS:vcluster101-vxrm.vv001.local, IP Address:172.168.10.50 notBefore=Jan 27 15:08:32 2022 GMT notAfter=Apr 16 15:08:32 2024 GMT SHA256 Fingerprint=B9:03:A9:FA:C1:44:06:E9:16:7F:F3:55:67:D1:BF:D7:9A:F6:FC:D0:50:FF:31:08:B1:50:FE:32:FC:C6:E1:09 vcluster101-vxrm:/home/mystic # openssl x509 -in /etc/vmware-marvin/ssl/rootcert.crt -serial -nocert -subject -issuer -ext subjectAltName -startdate -enddate -fingerprint -sha256 serial=F2262B1EB62DFF22 subject=C = US, ST = local, L = vsphere, O = VMware, OU = VxRailApplianceServer, CN = vcluster101-vxrm.vv001.local issuer=C = US, ST = local, L = vsphere, O = VMware, OU = VxRailApplianceServer, CN = vcluster101-vxrm.vv001.local X509v3 Subject Alternative Name: DNS:vcluster101-vxrm.vv001.local, IP Address:172.168.10.50 notBefore=Feb 25 14:47:14 2022 GMT notAfter=May 15 14:47:14 2024 GMT SHA256 Fingerprint=A3:7E:D6:E9:44:AE:31:7A:5B:41:AC:1F:F5:BF:D8:02:2E:E8:A4:07:D9:99:05:EC:A0:85:7C:76:29:FA:5F:CF vcluster101-vxrm:/home/mystic # - We can see that even the FQDN matches in 'subject', 'issuer' and SAN, that the serial, fingerprint and dates of validity of server.api.gateway.crt differ from the server.crt
- Copy server.crt over server.api.gateway.crt and rootcert.crt
cp /etc/vmware-marvin/ssl/server.crt /etc/vmware-marvin/ssl/server.api.gateway.crt cp /etc/vmware-marvin/ssl/server.crt /etc/vmware-marvin/ssl/rootcert.crt systemctl restart vmware-marvin systemctl restart runjars
Follow steps 3c from KB 20625 - VxRail: How to apply for a new certificate for VxRail Manager (A Dell Support account is required to view this article)
- Check the files:
- Run the openssl command against server.api.gateway.crt and rootcert.crt
- If server.crt is not ok
- Create a new self-signed VxRail Manager certificate - follow KB 20625 VxRail: How to apply for a new certificate for VxRail Manager (A Dell Support account is required to view this article)
- Run:
- vxv.log entry found: