NVP vProxy: NBD VM backup fails with "VDDK Error: 14009 - The server refused connection."
Summary: NetWorker VMware Protection (NVP) is configured using the Network Block Device (NBD) transport mode for one or more Virtual Machines (VM). The VM backup fails, reporting "VDDK Error: 14009 - The server refused connection." ...
Symptoms
Virtual Machine (VM) backups performed using Network Block Device (NBD) fail, reporting:
Failed to open source VMDK '[DATASTORE_NAME] #ABC#-#ABC#-#ABC#/VM_NAME.vmdk': VDDK Error: 14009 – The server refused connection.
The vProxy's /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-vddk.log reports:
YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO Cnx_Connect: Error message: Failed to connect to server ESXi_ADDRESS:902 YYYY-MM-DDTHH:mm:SS NOTICE: VDDK WARN [NFC ERROR]NfcNewAuthdConnectionEx: Failed to connect: Failed to connect to server ESXi_ADDRESS:902 YYYY-MM-DDTHH:mm:SS NOTICE: VDDK WARN [NFC ERROR]NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server ESXi_ADDRESS:902 YYYY-MM-DDTHH:mm:SS NOTICE: VDDK WARN [NFC ERROR]NfcEstablishAuthCnxToServer: Failed to create new AuthD connection: Failed to connect to server ESXi_ADDRESS:902 YYYY-MM-DDTHH:mm:SS NOTICE: VDDK WARN [NFC ERROR]Nfc_BindAndEstablishAuthdCnx3: Failed to create a connection with server ESXi_ADDRESS: Failed to connect to server ESXi_ADDRESS:902 YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO NBD_ClientOpen: Couldn't connect to ESXi_ADDRESS:902 Failed to connect to server ESXi_ADDRESS:902 YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO DISKLIB-DSCPTR: DescriptorOpenNbd: Failed to open NBD extent 'vpxa-nfc://[DATASTORE_NAME] #ABC#-#ABC#-#ABC#/VM_NAME.vmdk@ESXi_ADDRESS:902': NBD_ERR_NETWORK_CONNECT YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO DISKLIB-LINK : DiskLinkOpen: Failed to open 'vpxa-nfc://[DATASTORE_NAME] #ABC#-#ABC#-#ABC#/VM_NAME.vmdk@ESXi_ADDRESS:902': : NBD_ERR_NETWORK_CONNECT YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO DISKLIB-CHAIN : DiskChainOpen: "vpxa-nfc://[DATASTORE_NAME] #ABC#-#ABC#-#ABC#/VM_NAME.vmdk@ESXi_ADDRESS:902": failed to open: NBD_ERR_NETWORK_CONNECT. YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO DISKLIB-LIB : Failed to open 'vpxa-nfc://[DATASTORE_NAME] #ABC#-#ABC#-#ABC#/VM_NAME.vmdk@ESXi_ADDRESS:902' with flags 0xe NBD_ERR_NETWORK_CONNECT (2338). YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO VixDiskLib: Detected DiskLib error 2338 (NBD_ERR_NETWORK_CONNECT). YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO VixDiskLib: Failed to open disk vpxa-nfc://[DATASTORE_NAME] #ABC#-#ABC#-#ABC#/VM_NAME.vmdk@ESXi_ADDRESS:902!52a04cd0-8c56-ec6f-f705-ba0d58328830. Error 14009 (The server refused connection) (DiskLib error 2338: NBD_ERR_NETWORK_CONNECT) at 6396. YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_OpenEx: Cannot open disk [DATASTORE_NAME] #ABC#-#ABC#-#ABC#/VM_NAME.vmdk. Error 14009 (The server refused connection) (DiskLib error 2338: NBD_ERR_NETWORK_CONNECT) at 7072. YYYY-MM-DDTHH:mm:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_Open: Cannot open disk [DATASTORE_NAME] #ABC#-#ABC#-#ABC#/VM_NAME.vmdk. Error 14009 (The server refused connection) at 7146.
Hot-add backups of VMs on the same ESXi host may complete successfully.
NBD backups of other VMs on ESXi hosts (not observing this connection issue) may complete successfully.
Cause
The vProxy VM is unable to connect to ports 902 and/or 443 on the ESXi host the VM resides on.
curl -v ESXi_HOSTNAME:902 curl -v ESXi_HOSTNAME:443
Example:
nsr-vproxy01:~ # curl -v esx01.amer.lan:902 * Trying 192.168.9.114:902... * connect to 192.168.9.114 port 902 failed: Connection refused * Failed to connect to esx01.amer.lan port 902 after 3 ms: Couldn't connect to server * Closing connection 0 curl: (7) Failed to connect to esx01.amer.lan port 902 after 3 ms: Couldn't connect to server nsr-vproxy01:~ # curl -v esx01.amer.lan:443 * Trying 192.168.9.114:443... * connect to 192.168.9.114 port 443 failed: Connection refused * Failed to connect to esx01.amer.lan port 443 after 2 ms: Couldn't connect to server * Closing connection 0 curl: (7) Failed to connect to esx01.amer.lan port 443 after 2 ms: Couldn't connect to server
Or use the ProxyHC utility to check all port requirements from the vProxy to each ESXi host in the VMware environment:
/home/admin/ProxyHC port
ProxyHC utility is not on the vProxy by default, you must download it from the Support Tools site and copy it to the vProxy appliance. Instructions are provided in: NVP-vProxy: How to use health check tool ProxyHC on vProxy appliance
Resolution
This issue is occurring outside of NetWorker. Consult with the network and VMware administrator to resolve the network connection issues.
Port requirements and network troubleshooting steps are detailed in: NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations
Alternatively, if this issue is only impacting NBD backups, check to see if the VM supports the hot-add transport mode instead.
- The VM must reside on a VMware datastore that the vProxy's ESXi host has direct access to.
- The VM must not have any IDE disks.
- The VM must use hardware version 7 or later.
- Other limitations and considerations are documented in the NetWorker VMware Integration Guide: https://www.dell.com/support/product-details/product/networker/docs