NVP vProxy: VMware View Does Not Refresh and all VM Backups Fail

• VMware vCenters are added to the NetWorker server for performing NetWorker VMware Protection vProxy backups.
• The NetWorker Management Console > Protection > VMware View tab fails to refresh:

Error fetching vCenter information for: vCenter_Name

Reason(s): Unable to fetch data from vCenter: End of file or no input: Operation interrupted or timed out (3600 s receive delay) (3600 s send delay).

• The automated nsrvim process used to query vCenter inventory is failing. The daemon.log on the NetWorker server reports:

106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 6508 3964 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: nsrvim starting on NetWorker_Hostname (process 6252). 
106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 6508 3964 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Connecting to NetWorker on 'NetWorker_Hostname'. 
106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 6508 3964 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Querying NSR hypervisor resource 'vCenter_Hostname' 
106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 6508 3964 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Connecting to service at https://vCenter_Hostname/sdk 
106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 6508 3964 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Starting session with infrastructure services daemon. 
106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 5648 7592 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Querying for inventory at https://vCenter_Hostname/sdk 
106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 7952 8524 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Querying for inventory at https://vCenter_Hostname/sdk 
106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 3624 4728 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Querying for inventory at https://vCenter_Hostname/sdk 
106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 6508 3964 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Querying for inventory at https://vCenter_Hostname/sdk 
106637 MM/DD/YYYY HH:MM:SS AM/PM  1 3 0 2228 7760 0 NetWorker_Hostname nsrdisp_nwbg RAP notice job 'nsrvim' progress message: End of file or no input: Operation interrupted or timed out (3600 s receive delay) (3600 s send delay)

Linux: /nsr/logs/daemon.raw
Windows: C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw
NetWorker: NetWorker: How to use nsr_render_log 
NetWorker: NetWorker: How to automatically render daemon.raw to daemon.log in real time

• Virtual machine (VM) backups from the vCenter fail with a similar error message reported in the backup session log:

MM/DD/YYYY HH:MM:SS AM/PM Failed to run nsrvim, error: Unable to fetch data from vCenter: End of file or no input: Operation interrupted or timed out (3600 s receive delay) (3600 s send delay).
MM/DD/YYYY HH:MM:SS AM/PM Failed to get work items. Will retry in 360 seconds.
MM/DD/YYYY HH:MM:SS AM/PM Starting nsrvim.
MM/DD/YYYY HH:MM:SS AM/PM Calling the nsrvim program to collect the inventory data.
MM/DD/YYYY HH:MM:SS AM/PM Setting default timeout 1800.
MM/DD/YYYY HH:MM:SS AM/PM Using a timeout of 1800 seconds for the nsrvim request. Minimum timeout is 360 seconds. Maximum timeout is 3600 seconds.
MM/DD/YYYY HH:MM:SS AM/PM Failed to run nsrvim, error: Unable to fetch data from vCenter: End of file or no input: Operation interrupted or timed out (3600 s receive delay) (3600 s send delay).
MM/DD/YYYY HH:MM:SS AM/PM Unable to fetch data from vCenter: End of file or no input: Operation interrupted or timed out (3600 s receive delay) (3600 s send delay)
MM/DD/YYYY HH:MM:SS AM/PM Action backup vmware-vproxy 'backup' with job id 1769899 is exiting with status 'failed', exit code 1
MM/DD/YYYY HH:MM:SS AM/PM Action has finished with failures.

Linux: /nsr/log/policy/Policy_Name/Workflow_Name
Windows: C:\Program Files\EMC NetWorker\nsr\logs\policy\Policy_Name\Workflow_Name

• The NetWorker server can reach port 443 on the vCenter Server:

Windows (PowerShell): tnc vCenter_Hostname -port 443
Linux: curl -v vCenter_Hostname:443
NetWorker command: nsrports -t vCenter_Hostname -p 443

• ​​Errors and symptoms described may be consistent or appear intermittently.

The error that is returned is that the operation is interrupted or timed out. However, the timeout is 3600 s (1 hour) and the error appears before the 1-hour threshold is passed. The process is being interrupted.

• Network routing or firewall issue.
• vCenter server is closing the connection before the nsrvim inventory process completes.

Case 1 scenario: The firewall is configured with Adaptive Rule which allows NetWorker server to connect to vCenter over 443, but closes connection while NetWorker's nsrvim process is inventorying vCenter.

Case 2 scenario: vCenter server is sending a FIN segment during the nsrvim protocol data unit (PDU). 

The NetWorker server's nsrvim process is used to query vCenter servers for VMware resources. This process runs every 15 minutes by default, during NMC VMware View refreshes, or when a VM protection job starts.

The network or firewall admin must check for firewall rules blocking or interrupting connections between the NetWorker server and vCenter server on port 443. If there are any rules in place, disable them temporarily to see if the issue is resolved in NetWorker. If disabling rules allows VMware View to refresh and backups to complete, adjust firewall or routing rules to maintain NetWorker server and vCenter connections. 

Required ports and network topology diagrams are detailed in the NetWorker VMware Integration Guide, available on Dell Support at NetWorker Support Information.

The network administrator can also use packet capturing tools (tcpdump, Wireshark) from the NetWorker server and vCenter. When the issue is reproduced review the packet captures to see if the vCenter server is closing the inventory session.

tcpdump command example:

nohup tcpdump -i any -s 0 -C 500 -w /tmp/`hostname`_`date -I`.pcap &

• nohup option indicates that the command is run in the background until the process ID (PID) is terminated with the kill command.
• -i specifies interface, you can use any, or specify a network interface name, such as eth0.
• -s 0 specifies a snap length of 65535 (the entire frame is captured).
• -C 500 option indicates a file size of 500,000,000 bytes.
• -w option indicates the output file location. The output file shown is automatically generated with the system hostname and YYYY-MM-DD that it was run. A .pcap file can be analyzed in Wireshark.

Linux NetWorker server and vCenter server: https://www.tcpdump.org/manpages/tcpdump.1.html

Wireshark tshark command example:

.\tshark.exe -D

.\tshark.exe -i{Interface Number} -a files:{number of files} -b duration:{file duration in seconds} -f "dst host DST_IP_ADDRESS and src host SRC_IP_ADDRESS" -w tshark_capture.pcapng

DST_IP_ADDRESS: Replace this with the Domain Name System (DNS) resolvable IP address of the vCenter server. This should be the DNS resolvable IP address for the vCenter hostname used to add the vCenter server to NetWorker.
SRC_IP_ADDRESS: Replace this with the DNS resolvable IP address of the NetWorker server.

Example:

See: https://www.wireshark.org/docs/man-pages/tshark.html

Enable nsrdispd debug and reproduce the failure:

1. If the NetWorker server is Linux, run the above command on the NetWorker server. Run the above command on the vCenter server appliance. If the NetWorker server is Windows, use Wireshark to capture traffic while reproducing the issue.
2. When reproducing the issue in NetWorker, enable nsrdispd debug and run the nsrvim command with debug.

dbgcommand -n nsrdispd Debug=9

nsrvim -D7 -d vCenter_Hostname > {Path_to_output_file} 2<&1

nve:~ # dbgcommand -n nsrdispd Debug=9
Process ID List : 14600
Processing PID:14600
nve:~ # nsrvim -D7 -d vcsa.amer.lan > /tmp/nsrvim.out 2<&1
nve:~ # ls -l /tmp | grep nsrvim
-rw------- 1 root      root      60025 May 22 10:18 nsrvim.out
nve:~ #

3. The error is reported in the NetWorker server's daemon.raw.

Linux: /nsr/logs/daemon.raw
Windows: C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw
NetWorker: NetWorker: How to use nsr_render_log 
NetWorker: NetWorker: How to automatically render daemon.raw to daemon.log in real time 

4. Disable the nsrdispd debug and stop the tcpdump:

dbgcommand -n nsrdispd Debug=0

ps -ef | grep tcpdump

kill -9 PID_of_tcpdump

nve:~ # dbgcommand -n nsrdispd Debug=0
Process ID List : 14600
Processing PID:14600
nve:~ # ps -ef | grep tcpdump
root     29439 29267  0 10:01 pts/0    00:00:00 tcpdump -i any -s 0 -C 500 -w /tmp/nve_2024-05-22.pcap
root     29882 29267  0 10:13 pts/0    00:00:00 grep --color=auto tcpdump
nve:~ # kill -9 29439
nve:~ # ps -ef | grep tcpdump
root     29890 29267  0 10:13 pts/0    00:00:00 grep --color=auto tcpdump
[1]+  Killed                  nohup tcpdump -i any -s 0 -C 500 -w /tmp/`hostname`_`date -I`.pcap
nve:~ # ps -ef | grep tcpdump
root     29893 29267  0 10:13 pts/0    00:00:00 grep --color=auto tcpdump
nve:~ # ls -l /tmp | grep pcap
-rw------- 1 root      root    5464064 May 22 10:13 nve_2024-05-22.pcap
nve:~ #

Review packet captures to see if the vCenter server or a network device is closing the connection.

NetWorker 19.10 and later allows the backup administrator to change the nsrvim interval. The default interval is 15 minutes; however, this can be extended to any interval up to 60 minutes. This feature is detailed in NVP vProxy: NetWorker nsrvim process runs every 15 minutes causing a high workload on vCenter Server and potential VPXD unavailability.

Additional information about network requirements and troubleshooting can be found in: NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations