NVP-vProxy: Backup Failed to Disable Storage Migration for Virtual Machine with "VDDK Error: 1: Unknown error" Failed to Connect, No Route to Host
Summary: Virtual Machine (VM) backups performed through the NetWorker VMware Protection (NVP) vProxy appliance fail with "Backup failed to disable storage migration for virtual machine with 'VDDK Error: 1: Unknown error." The vProxy's vbackupd-vddk.log indicates a network connection issue between the vProxy and vCenter. ...
Symptoms
VM backups performed with the NetWorker VMware Protection (NVP) vProxy appliance fail
All VMs from a specific vProxy (sometimes specific vCenter) fail with the following error logged in the policy logs:
YYYY-MM-DDTHH:mm:SSZ ERROR: [###] Failed to disable storage migration for virtual machine "VM-MOREF": VDDK Error: 1: Unknown error.
YYYY-MM-DDTHH:mm:SSZ INFO: [###] For the past 2 minutes there have been errors attempting to disable Storage vMotion, will continue attempts for another 12 minutes.
YYYY-MM-DDTHH:mm:SSZ WARN: [###] Error disabling storage migration: VDDK Error: 1: Unknown error., retrying
Linux: /nsr/logs/policy/POLICY-NAME/WORKFLOW-NAME/ACTION-NAME/
Windows: C:\Program Files\EMC NetWorker\nsr\logs\POLICY-NAME\WORKFLOW-NAME\ACTION-NAME\
The vProxy appliance's /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-vddk.log contains the following "no route to host" messages when attempting to disable storage migration:
YYYY-MM-DDTHH:mm:SSZ NOTICE: VDDK INFO YYYY-MM-DDTHH:mm:SS+TZ warning -[03208] [Originator@6876 sub=IO.Connection] Failed to connect; <io_obj p:0x00007fa3d802a3c8, h:16, <TCP 'VPROXY-IPADDR : 33754'>, <TCP 'VCENTER-IPADDR : 443'>>, e: 113(No route to host), duration: 3032msec
YYYY-MM-DDTHH:mm:SSZ NOTICE: VDDK INFO YYYY-MM-DDTHH:mm:SS+TZ warning -[03208] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007fa3cc026850, TCP:VCENTER-IPADDR:443>, (null), duration: 3033msec, N7Vmacore15SystemExceptionE(No route to host)
YYYY-MM-DDTHH:mm:SSZ NOTICE: VDDK INFO --> [context]XXXXXXXXXX/XXXXXXXXXX[/context]
YYYY-MM-DDTHH:mm:SSZ NOTICE: VDDK INFO VixDiskLib: VixDiskLib_PrepareForAccess: Failed to start session. Cannot get compatible version for vim at 5938.
YYYY-MM-DDTHH:mm:SSZ NOTICE: VDDK INFO VixDiskLib: VixDiskLib_PrepareForAccess: Deactivate Storage VMotion failed. Error 1 (Unknown error) (Cannot get compatible version for vim) at 5949.
YYYY-MM-DDTHH:mm:SSZ NOTICE: VDDK INFO VixDiskLib: VixDiskLib_Disconnect: Disconnect.
A network connection test from the vProxy appliance fails to connect to the vCenter server with the same "no route to host" error.
ping vCenter-Hostnamecurl -v vCenter-Hostname:443
Cause
Resolution
Ensure that the vProxy networking information is correct.
- Connect to the vProxy using SSH as the admin account.
- Once connected, switch to root:
sudo su -
- Is the correct IP address assigned?
ifconfig -a
- Are the correct DNS servers configured?
cat /etc/resolv.conf
- Is the correct default gateway configured?
route
- Check the vProxies /etc/hosts file to see if there are any incorrect entries, pointing the vProxy to the wrong IP address for the vCenter. Remove any incorrect entries.
cat /etc/hosts
config_network script, detailed in article 157759: NVP vProxy: How to manually change the vProxy Appliance network interface settings
- The vProxy must be able to resolve the vCenter hostname as it is configured under NetWorker's VMware View. If DNS is not working or not configured, create /etc/hosts file entries on the vProxy appliances to resolve the vCenter hostname to the IP address.
- If there are no issues observed with the vProxy's network configuration, engage the VMware and Network administrators to review the VMware/networking configuration for any routing issues preventing successful communication between the vProxy and VMware hosts on port 443.
For more information about required communication for VM backup and restore operations, see the following documentation:
- Article 203350: NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations
- The NetWorker VMware Integration Guide, available on Dell Support at https://www.dell.com/support/home/product-support/product/networker/docs