NVP vProxy: Troubleshooting VM backups fail with "VDDK Error 13: You do not have access rights to this file"

VMware Virtual Machine backups fail with:

Failed to backup virtual disk "Hard disk #".  Failed to initialize Block Reader.  Failed to open source VMDK '[DATASTORE] VM_NAME/VM_NAME.vmdk': VDDK Error: 13: You do not have access rights to this file.

This message can be found in the Virtual Machine (VM) backup session log as seen through the NetWorker Management Console's (NMC) backup action details. The VM session logs are also stored in the following locations:

NetWorker Server: /nsr/logs/policy/POLICY_NAME/WORKFLOW_NAME/
vProxy Appliance: /opt/emc/vproxy/runtime/logs/recycle/vbackupd/DATE

This Virtual Disk Development Kit (VDDK) error is reported to vProxy by VMware. Error 13 is a general access error indicating that there are either permissions or communications issues preventing the vProxy from successfully accessing the VM files. 

Getting ProxyHC and moving it to the vProxy:

ProxyHC can be collected from the customer facing NetWorker Tools page: https://central.dell.com/solutions/Networker-Tools

Copying ProxyHC to the vProxy appliance:

The package can be copied to the vProxy appliance using an SCP agent (WinSCP, FileZilla, and so on). 

scp source admin@vProxy_address:/home/admin/ProxyHC

Using ProxyHC:

sudo su -

chmod +x /home/admin/ProxyHC

/home/admin/ProxyHC auth port perm tasks disk cleanup hotadd

If the cleanup flag indicated that operations were performed (disks removed, locks released, annotations cleared), re-attempt backup. If any failures are reported, review the vProxy's /tmp/proxy-hc.log. The output identifies what is missing and needed or isolates 'you do not have access rights' to a specific cause. References for specific causes can be found under "Additional Info."
 

KBs with specific causes for VDDK Error 13:

• Datastore Access: NVP vProxy: VM Backups Fail With "VDDK Error: 13: You do not have access rights to this file unable to open virtual disk in hot add mode"
• IDE disks: vProxy restore fails with "Failed to initialize Block Writer" and "VDDK Error: 13: You do not have access rights to this file."
• VMware Content-Read Based Cache (CBRC) is enabled: NVP vProxy: VM backups fail with VDDK Error 13 "You do not have access rights" when CBRC is enabled
• SCSI: NetWorker NVP-vProxy: Backups fail with the error: VDDK Error: 13: "You do not have access rights to this file." "Failed to allocate SCSI target ID"
• Permissions: NVP vProxy: VM backup fails with "Failed to open source VMDK '[DATASTORE] UUID/VM_NAME.vmdk': VDDK Error: 13: You do not have access rights to this file."
• Disk Already Attached: NVP-vProxy: VM disk failing to backup with "VDDK Error: 13: You do not have access rights to this file....mount point already exists"

Extra Resources:

• ProxyHC (Full KB): NVP-vProxy: How to use health check tool ProxyHC on vProxy appliance
• vProxy Versions table: NVP vProxy: NetWorker vProxy Version Information
• vCenter Builds: https://kb.vmware.com/s/article/2143838
• ESXi Builds: https://kb.vmware.com/s/article/2143832
• NetWorker compatibility matrix: https://elabnavigator.dell.com/eln/modernHomeAutomatedTiles?page=NetWorker

Other options for ProxyHC:

usage: ProxyHC vCenter [-h] --name NAME --user USER [--pwd PWD] [--vm VM]
                       [--nsruser NSRUSER] [--nsrpwd NSRPWD]
                       [--nsrauthc NSRAUTHC] [--allproxies ALLPROXIES]
                       [--DDhost DDHOST]
                       {auth,port,perm,disk,hotadd,migration,listsession,annotation,cleanup,kb524127,all,layout,tasks,datastore,vmsnapchain,vmdisksize,termsoapsession}
                       ...

positional arguments:
  {auth,port,perm,disk,hotadd,migration,listsession,annotation,cleanup,kb524127,all,layout,tasks,datastore,vmsnapchain,vmdisksize,termsoapsession}
                        sub-command help
    auth                test authentication against vCenter server
    port                check open ports to ESXi hosts in cluster and data
                        domain systems
    perm                check vCenter access permissions
    disk                check & release extra disks from vProxy
    hotadd              check for hotadd access to provided VM
    migration           check vCenter for stale migration locks
    listsession         print login session list to log
    annotation          check vCenter for stale backup locks(annotations)
    cleanup             remove left over artifacts(extra disks,
                        locks/annotations) and find migration locks.
    kb524127            check for invalid state of vProxy VM
    all                 All vCenter tools(combination of port, perm and
                        cleanup)
    layout              check for description of files that make up VM
    tasks               check orphaned tasks in vCenter
    datastore           look for datastores with unsupported names
    vmsnapchain         check if any protected VM has excessive child disks
    vmdisksize          check if any protected VM has irregular disk size i.e.
                        trailing .5 KB
    termsoapsession     terminate soap session to vCenter from NWS

optional arguments:
  -h, --help            show this help message and exit
  --name NAME           hostname of the vCenter server
  --user USER           user name for login to vCenter server
  --pwd PWD             password for login to vCenter server
  --vm VM               DNS name of VM(as shown in vSphere)
  --nsruser NSRUSER     user name for login to NetWorker server
  --nsrpwd NSRPWD       password for login to NetWorker server
  --nsrauthc NSRAUTHC   Remote AuthC server
  --allproxies ALLPROXIES
                        specify to do work on peer vProxies registered
  --DDhost DDHOST       provide DD IP/hostname for port connectivity tests