vProxy: FLR is failing for all clients with the error "Unable to create datastore"
Summary: The NetWorker VMware Protection integration is configured with the vProxy Appliance. The File level Recovery fails consistently using the NetWorker Management Console indicating that it was unable to create the Network File System (NFS) data store. ...
Symptoms
nsrvproxy_flr: vProxy Log: YYYY/MM/DD HH:MM:SS INFO: [@(#) Build number: ##] Unmounting after mount failure: Unable to create datastore '[VPROXY_NAME]-1487848168' using '[DATA_DOMAIN]:/data/col1/coral/[VPROXY_NAME]-[SESSION_ID]': ServerFaultCode: An error occurred during host configuration.
OR
nsrvproxy_flr: vProxy Log: YYYY/MM/DDTHH:MM:SS INFO: [@(#) Build number: 123] Unmounting after mount failure: Unable to create datastore 'EMC-FLR-[VPROXY_NAME]-026473-1645810870' using 'DD_NAME:/data/col1/networker/FLR-[VPROXY_NAME]-[SESSION_ID]': ServerFaultCode: An error occurred during host configuration. Operation failed, diagnostics report: Mount failed: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Unable to connect to NFS server.
Cause
The FLR workflow creates an NFS export on the Data Domain, which is used as a VMware Data Store on the target ESXi hosts.
A firewall prevents one or more ESXi hosts in the VMware Cluster from connecting to the NFS server (Data Domain) to mount the NFS Data Store.
Resolution
Ensure that TCP access is available over NFS ports 3009, 2049, and 2052 to the Data Domain from all ESXi hosts in the Cluster.
To test ESXi port connections, use netcat (nc) with the following flags:
For TCP tests:
nc -zv address port
[root@esxi:~] nc -zv ddve.amer.lan 3009 Connection to ddve.amer.lan 3009 port [tcp/sunrpc] succeeded! [root@esxi:~] nc -zv ddve.amer.lan 2049 Connection to ddve.amer.lan 2049 port [tcp/nfs] succeeded! [root@esxi:~] nc -zv ddve.amer.lan 2052 Connection to ddve.amer.lan 2052 port [tcp/*] succeeded!
Sometimes, ports are open on the network firewall but blocked on the ESXi host's OS firewall, resulting in a "connection timed out" message. Validate this by temporarily disabling the ESXi firewall. The ESXi firewall can be disabled with the following.
[root@esxi:~] esxcli network firewall get Default Action: DROP Enabled: true Loaded: true [root@esxi:~] esxcli network firewall set --enabled false [root@esxi:~] esxcli network firewall refresh
The vProxy appliance must also be able to reach NFS ports on the DD. These can be checked using the curl command:
curl -v address:port
Additional Information
Port requirements are documented in the NetWorker VMware Integration Guide and: NVP vProxy: Troubleshooting Network Connectivity For Backup and Restore Operations