NVP vProxy: VM Backup Fails to Download VM's .vmx file HTTP 500 Internal Server Error

During a NetWorker VMware Protection (NVP) VM backup, the backup fails and the following error is logged in the VM backup session log:

• NetWorker Server:

Linux: /nsr/logs/policy/POLICY_NAME/WORKFLOW_NAME/JOBID_VM-NAME_TIMESTAMP.log
Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\policy\POLICY_NAME\WORKFLOW_NAME\JOBID_VM-NAME_TIMESTAMP.log
 

 

• vProxy Appliance:

/opt/emc/vproxy/runtime/logs/recycle/vbackupd/DATE/BackupVmSessions-SESSION_ID.log

YYYY-MM-DDTHH:mm:SS ERROR:  [NETWORKER-BUILD] UUID-VCENTER-NAME:VM-MOREF: HTTP Get request failed to download config file '[DATASTORE-NAME] VM-NAME/VM-NAME.vmx' using URL 'https://VCENTER-NAME/folder/VM-NAME/VM-NAME.vmx?dcPath=DATACENTER-NAME&dsName=DATASTORE-NAME'. HTTP Status code: 500.
YYYY-MM-DDTHH:mm:SS ERROR:  [NETWORKER-BUILD] Failed to download VM config file "[DATASTORE-NAME] VM-NAME/VM-NAME.vmx" into saveset file "VM-MOREF-config-file-0.cfg".

The /var/log/hostd.log on the VMs (ESXi) host reports the following:

YYYY-MM-DDTHH:mm:SS Wa(164) Hostd[79374775]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/UUID/VM-NAME/VM-NAME.vmx] File - failed to get objectId, '/vmfs/volumes/UUID/VM-NAME/VM-NAME.vmx': Operation not supported (11)

This error coincides with when the HTTP 500 error is observed during the VMs backup. 

The HTTP 500 (Internal Server Error) in the VM backup session log is returned from the VMware environment. The vProxy backup workflow sends an HTTP GET request to get the VM's configuration files; however the vCenter is responding with HTTP 500 (Internal Server Error)

The hostd error message "Operation not supported (11)" typically indicates an issue accessing or manipulating the specified virtual machine file in VMware.

There are several causes which could factor into this error; however, most are at the ESXi level:

• File permissions -The ESXi host does not have the necessary permissions to access the file.
• Datastore Accessibility - Connectivity issues between the ESXi host and datastore.
• Management Agents - The ESXi host's hostd and vpxa agents are encountering issues.
• Disk space issues on the ESXi host
• VM configuration file issues
• ESXi host requires updates

NetWorker VMware Protection (NVP) Permissions Verification: 

The ProxyHC utility can be used to validate backup access permissions. The ProxyHC utility is not provided on the vProxy appliance by default.

See NVP-vProxy: How to use health check tool ProxyHC on vProxy appliance

./ProxyHC perm

Correct any missing permissions issues identified by ProxyHC.
  This issue can occur when VMware encryption is enabled, but the vCenter user account for NetWorker lacks cryptographic permissions: NetWorker NVP-vProxy: Backup error: Failed to download the VM configuration file. HTTP Status code: 500

When VMware encryption is enabled, the user account must have the following permissions:

• Cryptographic operations > Add disk
• Cryptographic operations > Direct access
• Cryptographic permissions > Register VM

The VMware administrator must perform permission verification of the files on the ESXi host.

Datastore Accessibility:

From the VMware vSphere Client, check the Datastore tab for any accessibility errors. In case there is an issue where connectivity is intermittent, check the vSphere Event Console for any errors or failures regarding the datastore access.

Management Agents:

Check the ESXi host's hostd and vpxa agent files for any errors. In some instances, restarting the agents can resolve the issue:

/etc/init.d/hostd restart
/etc/init.d/vpxa restart

ESXi disk space issues:

Ensure that there is sufficient disk space available on the ESXi host. Use the vdf -h command to check disk space.

VM configuration file corruption:

Verify access to the .VMX file outside of the backup workflow. Using the .vmx file URL from the VM session log, use one or more of the following options to validate the VMX file's integrity:

• Re-register the VM.
  • Right-click the VM and click Remove from Inventory (DO NOT DELETE FROM DISK!)
  • Browse the Datastore, locate the VM's .vmx file, and add it back to the inventory.
• On the vProxy appliance open an SSH session, run the following curl command. Replace the vCenter username with the user account used to perform VM backups. Replace the URL with the URL identified in the VM session log:

curl -v -k --user VCENTER_USER_ACCOUNT "VMX_URL"

admin@nsr-vproxy02:~> curl -v -k --user administrator@vsphere.local "https://vcsa.amer.lan/folder/05b58d65-4eef-c1d2-5070-00505606604d/rhel-client03.amer.lan.vmx?dcPath=vSAN%2520Datacenter&dsName=vsanDatastore"
Enter host password for user 'administrator@vsphere.local':
*   Trying 192.168.9.111:443...
* Connected to vcsa.amer.lan (192.168.9.111) port 443 (#0)
...
< HTTP/2 200
...
CONTENTS OF .VMX File
* Connection #0 to host vcsa.amer.lan left intact

• From a web browser that has access to the vCenter address. Enter/Paste the URL from the VM session log. Enter the credentials of the vCenter user account used for backups:

VMware vCenter and ESXi versioning:

Consult the NetWorker compatibility matrix for your NetWorker version. https://elabnavigator.dell.com/eln/modernHomeAutomatedTiles?page=NetWorker

In the NetWorker All Components guide, see NetWorker NVP (Proxy) Compatibility Matrix.
Use the latest vProxy major release supported by your NetWorker version and compatible with your ESXi version.
Ideally the vCenter Server and ESXi hosts should be on the same version and update release.

Workaround:

Either of the following workarounds can be tested; however, each option may only serve as a temporary solution:

• Perform a Host and Storage VMware vSphere vMotion of any VM reporting these failures. After vMotioning the VM, perform a backup from NetWorker.
• Reboot the ESXi hosts which contain the VMs reporting these issues during backups. After the ESXi host has come back online, confirm if VM backups are successful for VMs residing on this host.

In either case, VMware support is recommended to root-cause this issue.