Dell VxRail: VxRail Plugin Shows Error "The provided vCenter credentials are not valid"

VxRail 7.0.x or 8.0.x plugin does not work. It shows the error:

The provided vCenter credentials are not valid

Check the vCenter management account username and password to confirm that they are correct and valid.

• Scenario 1: There is a name resolution issue in microservices, specifically do-cluster.

/var/log/microservice_log/short.term.log shows Temporary failure in name resolution or No address associated with hostname.

"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737166669Z stderr F 2022-06-23 13:54:04,736 [ERROR] <Dummy-152:139828974536264> executor.py resolve_or_error() (456): An error occurred while resolving field ClusterDomainOwnerQuery.cluster"
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737174033Z stderr F Traceback (most recent call last):"
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737177269Z stderr F File ""/home/app/api/graphql_query/resolver/ClusterResolver.py"", line 21, in get_cluster"
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737179684Z stderr F si = soap_client.get_service_instance()"
...
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737280516Z stderr F File ""/usr/local/venv/lib64/python3.6/site-packages/do_common/socks_proxy.py"", line 40, in create_connection"
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737283191Z stderr F sock_addr_info = get_sorted_sock_addr_info(host, port)"
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737285535Z stderr F File ""/usr/local/venv/lib64/python3.6/site-packages/do_common/socks_proxy.py"", line 21, in get_sorted_sock_addr_info"
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737287589Z stderr F sock_addrs = socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM)"
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737289854Z stderr F File ""/usr/local/venv/lib64/python3.6/site-packages/gevent/_socketcommon.py"", line 230, in getaddrinfo"
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737291978Z stderr F addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags)"
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737294412Z stderr F File ""/usr/local/venv/lib64/python3.6/site-packages/gevent/resolver/thread.py"", line 63, in getaddrinfo"
...
"2022-06-23 13:54:05,524" microservice.do-cluster "2022-06-23T13:54:04.737321073Z stderr F socket.gaierror: [Errno -3] Temporary failure in name resolution"

• Scenario 2: An invalid CRL file was downloaded from vCenter trusted root CA certificates.

2022-05-24T14:04:31.381+0000 INFO  [main] com.vce.commons.core.ssl.MarvinTrustManager MarvinTrustManager.reloadAllowedAuthorities:148 - PostConstruct:reloadAllowedAuthorities
2022-05-24T14:04:31.385+0000 INFO  [main] com.vce.commons.core.ssl.MarvinTrustManager MarvinTrustManager.reloadAllowedAuthorities:166 - Allowing authority permanently for cert /var/lib/vmware-marvin/trust/lin/42727c5a.0
2022-05-24T14:04:31.386+0000 INFO  [main] com.vce.commons.core.ssl.MarvinTrustManager MarvinTrustManager.reloadCrl:186 - reloadCrl
2022-05-24T14:04:31.398+0000 ERROR [main] org.springframework.web.context.ContextLoader ContextLoader.initWebApplicationContext:313 - Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'backupEVCSettingAction': Unsatisfied dependency expressed through field 'vcConnectionService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean
with name 'VCConnectionServiceImpl': Unsatisfied dependency expressed through field 'connectionService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'connectionHelper': Unsatisfied dependency expressed through field 'connectionFactory'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vcConnectionService':
 Unsatisfied dependency expressed through field 'marvinTrustManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'marvinTrustManager': Invocation of init method failed; nested exception is java.security.cert.CRLException: Empty input

vxm:/home/mystic # ls -l /var/lib/vmware-marvin/trust/lin/
total 8
-rw-r--r-- 1 tcserver pivotal 1489 May 17 10:47 42727c5a.0
-rw-r--r-- 1 tcserver pivotal  0 May 17 10:47 42727c5a.r0

• Scenario 3: Error certificate verify failed

/var/log/microservice_log/short.term.log

2022-03-21-08:06:25 microservice.do-cluster "ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)"
2022-03-21-08:06:25 microservice.do-cluster ""
2022-03-21-08:06:25 microservice.do-cluster "During handling of the above exception, another exception occurred:"
2022-03-21-08:06:25 microservice.do-cluster ""
2022-03-21-08:06:25 microservice.do-cluster "Traceback (most recent call last):"
2022-03-21-08:06:25 microservice.do-cluster " File ""/usr/local/venv/lib64/python3.6/site-packages/graphql/execution/executor.py"", line 452, in resolve_or_error"
2022-03-21-08:06:25 microservice.do-cluster " return executor.execute(resolve_fn, source, info, **args)"
2022-03-21-08:06:25 microservice.do-cluster " File ""/usr/local/venv/lib64/python3.6/site-packages/graphql/execution/executors/sync.py"", line 16, in execute"
2022-03-21-08:06:25 microservice.do-cluster " return fn(*args, **kwargs)"
2022-03-21-08:06:25 microservice.do-cluster " File ""/home/app/api/graphql_query/cluster_do_query.py"", line 59, in resolve_cluster"
2022-03-21-08:06:25 microservice.do-cluster " cluster = ClusterResolver.get_cluster(vc_conn_info_input, cluster_argument)"
2022-03-21-08:06:25 microservice.do-cluster " File ""/home/app/api/graphql_query/resolver/ClusterResolver.py"", line 23, in get_cluster"
2022-03-21-08:06:25 microservice.do-cluster " raise GraphQLError('Failed to connect to vCenter {}'.format(vc_conn_info_input.get('host')))"
2022-03-21-08:06:25 microservice.do-cluster "graphql.error.base.GraphQLError: Failed to connect to vCenter None"

• Scenario 4: vCenter's SSL certificate is not fully qualified.

/var/log/microservice_log/short.term.log

2022-04-20-06:00:16 microservice.do-cluster " File ""/usr/lib64/python3.6/ssl.py"", line 694, in do_handshake"
2022-04-20-06:00:16 microservice.do-cluster " match_hostname(self.getpeercert(), self.server_hostname)"
2022-04-20-06:00:16 microservice.do-cluster " File ""/usr/lib64/python3.6/ssl.py"", line 331, in match_hostname"
2022-04-20-06:00:16 microservice.do-cluster " % (hostname, dnsnames[0]))"
2022-04-20-06:00:16 microservice.do-cluster "ssl.CertificateError: hostname 'VC_FQDN' does not match 'VC_IP'"

vCenter's cert does not contain FQDN in Subject Alternative Name, it may contain IP address only:
echo | openssl s_client -connect <vc_fqdn>:443 2>/dev/null |openssl x509 -noout -text

X509v3 Subject Alternative Name: 
                IP Address:xx.xx.xx.xx

Example in lab:
X509v3 Subject Alternative Name:
                DNS:xxxxxxx, IP Address:xx.xx.xx.xx

• Scenario 5: Microservices are unable to connect to vCenter with error No route to host.

Command firewall-cmd --reload was run in VxRail Manager recently.
/var/log/microservice_log/short.term.log

"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540928759Z stderr F   File ""/usr/local/venv/lib64/python3.6/site-packages/pyVmomi/SoapAdapter.py"", line 1039, in connect"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540932851Z stderr F     http_client.HTTPSConnection.connect(self)"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540936399Z stderr F   File ""/usr/lib64/python3.6/http/client.py"", line 1444, in connect"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540939008Z stderr F     super().connect()"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.54094136Z stderr F   File ""/usr/lib64/python3.6/http/client.py"", line 956, in connect"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540943688Z stderr F     (self.host,self.port), self.timeout, self.source_address)"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540946025Z stderr F   File ""/usr/local/venv/lib64/python3.6/site-packages/do_common/socks_proxy.py"", line 71, in create_connection"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540948554Z stderr F     raise err"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540950893Z stderr F   File ""/usr/local/venv/lib64/python3.6/site-packages/do_common/socks_proxy.py"", line 61, in create_connection"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540955153Z stderr F     sock.connect((sa[0], sa[1]))"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540958727Z stderr F   File ""/usr/local/venv/lib64/python3.6/site-packages/socks.py"", line 47, in wrapper"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.54096135Z stderr F     return function(*args, **kwargs)"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540963723Z stderr F   File ""/usr/local/venv/lib64/python3.6/site-packages/socks.py"", line 780, in connect"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.54096604Z stderr F     super(socksocket, self).connect((dest_addr, dest_port))"
"2024-01-20 04:18:35,083" microservice.do-cluster "2024-01-20T04:18:34.540968413Z stderr F OSError: [Errno 113] No route to host"

Scenario 1: There is a name resolution issue in microservices.

1. Restart VxRail Manager's dnsmasq service:

service dnsmasq stop
service dnsmasq start

2. If restart nsmasq service does not help, check whether the VxRail manager DNS server is configured with an external public DNS, for example 8.8.8.8.
3. If it still fails to do name resolution, contact Dell Support and quote this article number 000214621 to run DNS check tool.

Scenario 2: Invalid CRL file downloaded from vCenter trusted root CA certificates

Follow the article VxRail: Unable to Import vCenter Root Certificates Due to Empty or Corrupted CRL Files to delete empty or corrupted CRL files from vCenter and reimport vCenter trusted root certificates to VxRail Manager.

Scenario 3: Error certificate verify failed

Contact Dell Support and quote this article number 000157888 to resolve certificate issue.

Scenario 4: vCenter's SSL cert is not fully qualified.

Regenerate vCenter Server machine SSL cert > SubjectAltName must contain DNS Name=machine_FQDN.

Scenario 5: Microservices are unable to connect to vCenter with error No route to host.

Restart rke2 server by running the below two commands:

bash /usr/local/bin/rke2-killall.sh
systemctl start rke2-server

Or

Reboot VxRail Manager.