NetWorker: Unable to fetch data from vCenter: SSL_ERROR_SYSCALL Error observed by underlying SSL/TLS BIO

• The following error message is returned in the NetWorker Management Console (NMC) when performing a VMware View refresh:

• The NetWorker server and vCenter Server versions meet compatibility requirements. See the NetWorker compatibility matrix available through: Support for NetWorker | Manuals & Documents 
• The NetWorker server can resolve the vCenter's IP address, Fully Qualified Domain Name (FQDN), and short name correctly.

nslookup ADDRESS

• The NetWorker server can reach port 443 on the vCenter server:

Linux:

curl -v vCenter_Address:443

Test-NetConnection -ComputerName vCenter_Address -port 443

Firewall Configuration: Ensure that your firewall is not blocking SSL traffic. Even if port 443 is open, packet-level filtering might be preventing SSL traffic.

Network Issues: The error might be due to a network issue where the connection is being reset by the peer. This can happen if there is an intermediate device (like a firewall or router) that is interfering with the connection.

The SSL_ERROR_SYSCALL error happens when the TCP handshake completes, but a TCP reset (RST) packet is received, ending the connection during the SSL phase.

The network or firewall admin must check for firewall rules blocking or interrupting SSL connections between the NetWorker server and vCenter server on port 443. If there are any rules in place, disable them temporarily to see if the issue is resolved in NetWorker. If disabling the rules allows VMware View to refresh and backups to complete, adjust firewall or routing rules to maintain connections between NetWorker server and vCenter.
 

Packet Capture Tools

The network administrator can also use packet capturing tools (tcpdump, Wireshark) from the NetWorker server and vCenter. When the issue is reproduced review the packet captures to see if the vCenter server is closing the inventory session.

tcpdump command example:

nohup tcpdump -i any -s 0 -C 500 -w /tmp/`hostname`_`date -I`.pcap &

OR:

nohup tcpdump -i any host NW_SERVER_IP and host VCENTER_SERVER_IP -s 0 -C 500 -w /tmp/`hostname`_`date -I`.pcap &

• nohup option indicates that the command is run in the background until the process ID (PID) is terminated with the kill command.
• -i specifies interface, you can use any, or specify a network interface name, such as eth0.
• host is used to filter the capture. In the above example, this is used to filter the traffic for only the IP addresses of the NetWorker server and vCenter server. If these hosts have multiple IP addresses, then the IP addresses used during NetWorker communication must be specified.
• -s 0 specifies a snap length of 65535 (the entire frame is captured).
• -C 500 option indicates a file size of 500,000,000 bytes.
• -w option indicates the output file location. The output file shown is automatically generated with the system hostname and YYYY-MM-DD that it was run. A .pcap file can be analyzed in Wireshark.

Linux NetWorker server and vCenter server: https://www.tcpdump.org/manpages/tcpdump.1.html 

Wireshark tshark command example:

cd "C:\Program Files\Wireshark"

.\tshark.exe -D

.\tshark.exe -i{Interface Number} -a files:{number of files} -b duration:{file duration in seconds} -f "dst host DST_IP_ADDRESS and src host SRC_IP_ADDRESS" -w tshark_capture.pcapng

DST_IP_ADDRESS: Replace this with the DNS resolvable IP address of the vCenter server. This should be the DNS resolvable IP address for the vCenter hostname used to add the vCenter server to NetWorker.
SRC_IP_ADDRESS: Replace this with the DNS resolvable IP address of the NetWorker server.

Example:

See: https://www.wireshark.org/docs/man-pages/tshark.html 

Procedure:

The packet capture method used depends on the operating systems involved. For Windows NetWorker servers, use Wireshark or tshark as detailed above. For Linux NetWorker servers and the vCenter server appliance, use tcpdump as instructed above.

1. Start a packet capture on both the NetWorker server and vCenter server. This must capture the communication attempt between the NetWorker server and vCenter server.
2. From the NetWorker Management Console (NMC), perform a VMware View refresh.
   1. Go to Protection->VMware View
   2. Right-click the vCenter server.
   3. Click Refresh.
3. When the SSL_ERROR_SYSCALL error appears. Make note of the time on the NetWorker server and vCenter server.
4. Stop the packet captures.
5. Review the packet captures for any TCP reset (RST) packets between the NetWorker server and vCenter server. 

Make note of:

• The DNS resolvable IP address of the NetWorker server and vCenter server.
• The time zones of both the NetWorker server and vCenter server (in case different).
• The timestamp on the NetWorker server when the SSL_ERROR_SYSCALL appeared in the NMC.
  • The rendered /nsr/logs/daemon.raw on the NetWorker server logs a nsrvim failure when this error appears. The daemon.raw must be rendered locally on the NetWorker server to render the timestamps in the server's time zone.
    NetWorker: How to use nsr_render_log to render .raw log files

Openssl connection from the NetWorker server to the vCenter server may produce a similar failure. Openssl is not typically available on Windows hosts; however, it can be installed from third-party providers. Openssl is commonly on Linux hosts by default. Example command for Linux:

openssl s_client -connect VCENTER_IP:443 -showcerts < /dev/null

Additional articles:

NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations
NVP vProxy: VMware View Does Not Refresh and all VM Backups Fail
NetWorker: VMware View Refresh Fails "Dispatcher Request Timed Out"