NetWorker: Unable to fetch data from vCenter: SSL_ERROR_SYSCALL Error observed by underlying SSL/TLS BIO

• The following error message is returned in the NetWorker Management Console (NMC) when performing a VMware View refresh:

• The NetWorker server and vCenter Server versions meet compatibility requirements. See the NetWorker compatibility matrix available through: https://elabnavigator.dell.com/eln/modernHomeAutomatedTiles?page=NetWorker
• The NetWorker server can resolve the vCenter's IP address, Fully Qualified Domain Name (FQDN), and shortname correctly.

nslookup ADDRESS

• The NetWorker server can reach port 443 on the vCenter server:

Linux:

curl -v vCenter_Address:443

Test-NetConnection -ComputerName vCenter_Address -port 443

The network or firewall admin must check for firewall rules blocking or interrupting SSL connections between the NetWorker server and vCenter server on port 443. If there are any rules in place, disable them temporarily to see if the issue is resolved in NetWorker. If disabling the rules allows VMware View to refresh and backups to complete, adjust firewall or routing rules to maintain connections between NetWorker server and vCenter.
 

Packet Capture Tools

The network administrator can also use packet capturing tools (tcpdump, Wireshark) from the NetWorker server and vCenter. When the issue is reproduced review the packet captures to see if the vCenter server is closing the inventory session.

tcpdump command example:

nohup tcpdump -i any -s 0 -C 500 -w /tmp/`hostname`_`date -I`.pcap &

• nohup option indicates that the command is run in the background until the process ID (PID) is terminated with the kill command.
• -i specifies interface, you can use any, or specify a network interface name, such as eth0.
• -s 0 specifies a snap length of 65535 (the entire frame is captured).
• -C 500 option indicates a file size of 500,000,000 bytes.
• -w option indicates the output file location. The output file shown is automatically generated with the system hostname and YYYY-MM-DD that it was run. A .pcap file can be analyzed in Wireshark.

Linux NetWorker server and vCenter server: https://www.tcpdump.org/manpages/tcpdump.1.html 

Wireshark tshark command example:

cd "C:\Program Files\Wireshark"

.\tshark.exe -D

.\tshark.exe -i{Interface Number} -a files:{number of files} -b duration:{file duration in seconds} -f "dst host DST_IP_ADDRESS and src host SRC_IP_ADDRESS" -w tshark_capture.pcapng

DST_IP_ADDRESS: Replace this with the DNS resolvable IP address of the vCenter server. This should be the DNS resolvable IP address for the vCenter hostname used to add the vCenter server to NetWorker.
SRC_IP_ADDRESS: Replace this with the DNS resolvable IP address of the NetWorker server.

Example:

See: https://www.wireshark.org/docs/man-pages/tshark.html 

Procedure:

The packet capture method used depends on the operating systems involved. For Windows NetWorker servers, use Wireshark or tshark as detailed above. For Linux NetWorker servers and the vCenter server appliance, use tcpdump as instructed above.

1. Start a packet capture on both the NetWorker server and vCenter server. This must capture the communication attempt between the NetWorker server and vCenter server.
2. From the NetWorker Management Console (NMC), perform a VMware View refresh.
   1. Go to Protection->VMware View
   2. Right-click the vCenter server.
   3. Click Refresh.
3. When the SSL_ERROR_SYSCALL error appears. Make note of the time on the NetWorker server and vCenter server.
4. Stop the packet captures.
5. Review the packet captures for any TCP reset (RST) packets between the NetWorker server and vCenter server. 

Make note of:

• The DNS resolvable IP address of the NetWorker server and vCenter server.
• The time zones of both the NetWorker server and vCenter server (in case different).
• The timestamp on the NetWorker server when the SSL_ERROR_SYSCALL appeared in the NMC.
  • The rendered /nsr/logs/daemon.raw on the NetWorker server logs a nsrvim failure when this error appears. The daemon.raw must be rendered locally on the NetWorker server to render the timestamps in the server's time zone.
    NetWorker: How to use nsr_render_log to render .raw log files

NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations
NVP vProxy: VMware View Does Not Refresh and all VM Backups Fail