NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations

Systems and ports that are involved for VMware Virtual Machine (VM) backup and restore operations.

Per the NetWorker Security Configuration Guide, NetWorker uses a direct socket connection to communicate and move data across the network to the required service with minimal overhead. While NetWorker opens some ports for TCP and UDP, NetWorker only requires TCP ports. UDP ports are optional, except for SNMP which uses UDP port 161 and 162.

The port table and network topology that is displayed in this KB was pulled directly from the NetWorker VMware Integration Guide. The VMware Integration and Security Configuration Guides can be found on the Dell Support NetWorker Product page.

Port Requirements

Domain Name System (DNS)

Ensure to check that DNS is resolved correctly for the systems involved, including the Fully Qualified Domain Name (FQDN), Short Name, and IP Address (Reverse Lookup).

nslookup FQDN
nslookup SHORT_NAME
nslookup IP_ADDRESS

See article: NetWorker: Name Resolution Troubleshooting Best Practices

It is also advised checking system hosts files. Host file entries may conflict or override a DNS query, or lead to an incorrect IP address or hostname.

• Linux Systems: /etc/hosts
• Windows Systems: C:\Windows\System32\drivers\etc\hosts

NetWorker Server

The NetWorker nsrports command can be used to confirm name resolution and port connectivity between the systems involved. See the above table regarding which ports and target system should be specified when testing communication.

nsrports -t target_system -p port

Linux systems can use the curl command to test connectivity:

curl -v target_system:port

Windows systems can use Test-NetConnection PowerShell cmdlet:

Test-NetConnection -ComputerName target_system  -port port

vProxy Appliance

The ProxyHC (Health Check) utility can be run on the vProxy to check port connectivity between systems, see article: NVP-vProxy: How to use health check tool ProxyHC on vProxy appliance.

The vProxy appliance can also use the curl command to test connectivity. See the above table regarding which ports and target system should be specified when testing communication.

curl -v target_system:port

ESXi Server

The netcat (nc) command can be used on ESXi hosts to check port connectivity. See the above table regarding which ports and target system should be specified when testing communication.

nc -zv target_system port

See article NVP vProxy: NetWorker VIB to Open NFS Ports For vProxy FLR.

vCenter Server

vCenter servers use operating system-based connection commands. See the above table regarding which ports and target system should be specified when testing communication.

curl -v target_system:port

Virtual Machines

Whether testing communication from a VM for the Data Protection Restore Client, NetWorker Web User Interface (NWUI), or connectivity with Data Domain, the command used depends on the operating system of the virtual machine. Linux systems can use the curl command to test connectivity:

curl -v target_system:port

Windows systems can use Test-NetConnection PowerShell cmdlet:

Test-NetConnection -ComputerName target_system  -port port

• NetWorker server -> vProxy Appliance
• NetWorker server/Storage Node -> Data Domain
• vProxy Appliance -> Data Domain
• vProxy Appliance -> vCenter server

Linux Hosts

The following steps can be performed on Linux NetWorker servers, storage nodes, vProxy appliance, and/or the vCenter Server appliance.

sudo su -

nohup ping ADDRESS | while read l; do echo "$(date) $l"; done >> /nsr/logs/$(hostname)_ping.log &

ps -ef | grep ping

kill -9 PID_OF_PING

[root@nsr ~]# ps -ef | grep ping
gdm         4066    1993  0 Nov15 tty1     00:00:18 /usr/libexec/gsd-housekeeping
root      215151  215146  0 Nov18 ?        00:16:25 /opt/nsr/rabbitmq-server-3.11.16/erts-13.2.2/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -B i -- -root /opt/nsr/rabbitmq-server-3.11.16 -bindir /opt/nsr/rabbitmq-server-3.11.16/erts-13.2.2/bin -progname erl -- -home /nsr/rabbitmq -- -pa  -noshell -noinput -s rabbit boot -boot start_sasl -syslog logger [] -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false
root      467940  467115  0 16:10 pts/3    00:00:00 ping nsr-vproxy02.amer.lan
root      468141  467115  0 16:11 pts/3    00:00:00 grep --color=auto ping
[root@nsr ~]# kill -9 467940

Windows Hosts

The following steps can be performed on Windows NetWorker servers and/or remote storage node.

1. Create a .bat script (timestamped_ping.bat) containing the following. Replace ADDRESS with the IP address or DNS resolvable FQDN of the remote host. Change the output location to another path if NetWorker is not installed in the default install path, or if you want to direct the output somewhere else.

@echo off
ping -t ADDRESS |find /v ""|cmd /q /v:on /c "for /l %%a in (0) do (set "data="&set /p "data="&if defined data echo(!date! !time! !data!)" >> "C:\Program Files\EMC NetWorker\nsr\logs\ping.out" 2<&1

2. Open an Administrator command-prompt in the directory that the script was saved.

3. Run the script:

timestamped_ping.bat

4. Leave the script running and reproduce the backup or restore issue. 
5. Once the issue is reproduce. Stop the script by using CTRL+C in the command prompt the script is running in.
6. Review the C:\Program Files\EMC NetWorker\nsr\logs\ping.out file (or location that you specified) for any network issues (dropped packets, high latency, so forth) which overlap with the timestamp from when the backup or restore issue is observed.  

NOTE: The NetWorker Performance Optimization Planning Guide states that latency should not exceed 50 ms between the NetWorker server/storage node and Data Domain. This can result in poor throughput. Higher latency can result in sessions closing unexpectedly.