NVP vProxy: VM backup fails due to VMware View error "Unable to fetch data from vCenter"

Summary: VM backups are failing due to a connection issue with the vCenter server, and this article explains the resolution.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

vCenter backups are failing due to connection issues with vCenter.

  • Receive error in the NMC similar to KB: NVP-vProxy: View update fails, error "Unable to fetch data from vCenter: EOF was observed that violates the protocol."
  • NetWorker VMware View error received: 
    Unable to fetch data from vCenter: EOF was observed that violates the protocol.  The client probably provided invalid authentication information.
  • Connection issue with vCenter results in SSL errors. SSL error in the vProxy /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-vddk.log file. 
    VDDK INFO 2024-10-07T21:01:42.616+05:30 error -[04510] [Originator@6876 sub=IO] HandshakeCb; <SSL(<io_obj p:0x00007f2694008bc8, h:68, <TCP 'xxx.xxx.xxx.xxx : 60194'>, <TCP 'xxx.xxx.xxx.xxx : 443'>>)>; error: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read: The connection was closed by the remote end during handshake.)
VDDK INFO 2024-10-07T21:01:42.749+05:30 error -[10320] [Originator@6876 sub=transport] Cannot connect to Server: SSL Exception: error:140000DB:SSL routines:SSL routines:short read: The connection was closed by the remote end during handshake..
2024-10-07T15:31:42Z NOTICE: VDDK INFO 2024-10-07T21:01:42.791+05:30 warning -[10320] [Originator@6876 sub=transport] Fail to fetch ddb with double encoding, error(Cannot connect to server)
  • The vCenter server's /var/log/vmware/envoy/envoy.log contains: 
    2024-10-09T22:32:56.814Z warning envoy[3248] [Originator@6876 sub=filter] [Tags: "ConnectionId":"737396"] remote https connections exceed max allowed: 2048
2024-10-09T22:32:56.814Z warning envoy[3248] [Originator@6876 sub=filter] [Tags: "ConnectionId":"737396"] closing connection TCP<xxx.xxx.xxx.xxx:55718, xxx.xxx.xxx.xxx:443>

 

Cause

The root cause of the issue is a connection problem between the vCenter server and the NetWorker server, leading to a failure in vCenter backups. The VMware View error message "EOF was observed that violates the protocol" indicates a connection issue with vCenter.

The vbackupd-vddk.log file from the vProxy server shows an SSL error with the message:

"SSL Exception: error:140000DB:SSL routines:SSL routines:short read: The connection was closed by the remote end during handshake."

The VMware envoy log shows the same error, indicating a connection exhaustion issue.

"remote https connections exceed max allowed: 2048" suggests that the vCenter server is exceeding the maximum allowed connections to the vProxy server.

The closing connection message: "closing connection TCP<xxx.xxx.xxx.xxx:55718, xxx.xxx.xxx.xxx:443>" indicates that the vProxy server is closing the connection due to excessive connections.

 

Resolution

By default NetWorker will inventory a vCenter every 15 minutes. The vCenter inventory process will also occur anytime a user connects to the NetWorker server from the NetWorker Management Console, and anytime a VMware protection policy starts. The default "nsrvim" inventory interval can be increased up to 60 minutes, see: NVP vProxy: NetWorker nsrvim process runs every 15 minutes causing a high workload on vCenter Server and potential VPXD unavailability

If the issue persists even after increasing the NSRVIM_TIME_INTERVAL to 60 minutes, perform the following:

NOTE: This issue is impacting NetWorker VMware Protection; however, it is occurring on the vCenter server. To resolve it, contact your VMware Team or VMware Support to address the connection problem between the vCenter and NetWorker servers.

The resolution to the vCenter backup failure issue is to address the SSL connection exhaustion error. Follow these steps to resolve the issue:

  1. Follow VMware KB 344920 This hyperlink is taking you to a website outside of Dell Technologies.
  2. Contact VMware Support: If the above KB does not resolve the issue, contact VMware support for further assistance.
CAUTION: Disabling SSL verification may compromise the security of your environment. It is recommended to only disable SSL verification as a last resort and only after consulting with VMware support.

After implementing the above steps, verify that the issue has been resolved by checking the vCenter Server and vProxy logs for any SSL-related errors. If the issue persists, contact VMware Support for further assistance.

 

Additional Information

Verify that the NetWorker server can connect to the vCenter HTTPS port (443) over SSL:

openssl s_client -connect VCENTER_ADDRESS:443
NOTE: openssl is a third-party utility. It is typically included by default on Linux operating systems; however, it is typically not included by default on Windows operating systems. This requires installation of openssl.

NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations

 

Affected Products

NetWorker
Article Properties
Article Number: 000236865
Article Type: Solution
Last Modified: 23 Jan 2025
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.