VxRail: vCenter Remote https Connections Exceed Max Allowed Value After Reboot

In a customer-supplied vCenter environment (also known as an external vCenter), the vCenter can manage multiple VxRail clusters.
After a vCenter reboot, you may observe an error stating that the VxRail managers failed to connect to the vCenter.

Check the VxRail manager file /var/log/mystic/web.log. Look for the text Remote host terminated the handshake as this shows the VxRail manager failed to connect to the vCenter.

2023-05-16 09:35:37.378+0000 ERROR [myScheduler-6] com.graphql_java_generator.client.QueryExecutorImpl QueryExecutorImpl.doJsonRequestExecution:148 - {"message":"Failed to connect to vCenter None"," locations":[{"line":1,"column":8,"sourceName":null}],"description":null,"validationErrorType":null,"queryPath":null,"errorType":null,"path":["cluster"],"extensions":null}

2023-05-16 09:35:37.378+0000 INFO  [myScheduler-6] com.vce.commons.domainowner.cluster.VCRepository VCRepository.getDeploymentTypeAndHostSummary:135 - failed to get deployment type and host summary.com.graphql_java_generator.client.response.GraphQLExecutionException: 1 errors occured: {"message":"Failed to connect to vCenter None","locations":[{"line":1,"column":8,"sourceName":null}],"description":null,"validationErrorType":null,"queryPath":null,"errorType":null,"path":["cluster"],"extensions":null}

Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
        at sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1696) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1514) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416) ~[?:?]

Check the vCenter file /var/log/vmware/envoy/envoy.log. Look for the text remote https connections exceed max allowed: 2048. You then see the following message showing that the vCenter is closing connections from VxRail manager IP address x.x.x.x.

2023-05-16T08:55:26.142Z warning envoy[2436] [Originator@6876 sub=filter] [C28710] remote https connections exceed max allowed: 2048
2023-05-16T08:55:26.142Z warning envoy[2436] [Originator@6876 sub=filter] [C28710] closing connection TCP<x.x.x.x:52018, y.y.y.y:443>

After the vCenter is rebooted, all VxRail managers under that vCenter try to re-connect to it. Each VxRail Manager may try to initiate between 80 to 110 https connections. The vCenter default setting for the maximum number of allowed https connections is 2048. If the number of VxRail Managers trying to connect to a single vCenter is too large, then the vCenter cannot handle the number of connection requests. The end result is the maximum allowed connections are exceeded, and the vCenter starts to close the connections from some of the VxRail Managers.

VxRail 7.0.480 has an enhancement to reduce the https connections to vCenter during vCenter reboot.
If you still encounter this issue, follow the below steps as a workaround.

If the vCenter managed VxRail clusters are less than 25, follow VMware article 344920 HTTPS Connection Exhaustion in envoy logs of the vCenter server to increase the maximum RemoteHttpsConnections limit to 3072 and then restart vCenter rhttpproxy service.

If the vCenter manages more than 25 VxRail clusters, do not try to increase the maximum RemoteHttpsConnections limit to a number larger than 3072. VMware does not recommendthis, and it may negatively impact the vCenter performance.

Follow the below steps to work around:

1. Power off all the VxRail Managers to allow the vCenter to free up the https connections.
2. Boot up 10 VxRail managers and wait for the connection number to be stable. You can run the below command on the vCenter to monitor the connection number.

netstat -tnep | grep envoy | grep "<VCSA IP address>:443"| wc -l

3. Boot up another five VxRail managers and wait for the connection number to be stable.
4. Repeat step 3 until all VxRail managers are booted up.