NetWorker vProxy: Backups failed to initialize Block Reader. Failed to open source VMDK

The NetWorker VMware Protection (NVP) integration is configured with the vProxy appliance. The virtual machine (VM) backups intermittently fail to open the VM's source VMDK with the hot add transport. The scheduled VMware Protection Workflow execution will impacts different VMs during every execution.

The vProxy backup session log shows:

YYYY-MM-DDTHH:MM:SS INFO:   [@(#) Build number: 66] Data Mover: Hard disk 1: Preparing for data movement.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Data Mover: Hard disk 1: VDDK: Failed to open source VMDK "[Datastore] VM_name/VM_name-000002.vmdk": VDDK Error: 13: You do not have access rights to this file.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Data Mover: Hard disk 1: Failed to initialize Block Reader.
YYYY-MM-DDTHH:MM:SS INFO:   [@(#) Build number: 66] Data Mover: Hard disk 1: Cleaned up after data movement.
YYYY-MM-DDTHH:MM:SS INFO:   [@(#) Build number: 66] Data Mover: Hard disk 3: Preparing for data movement.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Failed to backup virtual disk "Hard disk 1".
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Data Mover: Hard disk 3: VDDK: Failed to open source VMDK "[Datastore] sVM_name/VM_name_2-000002.vmdk": VDDK Error: 13: You do not have access rights to this file.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Data Mover: Hard disk 3: Failed to initialize Block Reader.
YYYY-MM-DDTHH:MM:SS INFO:   [@(#) Build number: 66] Data Mover: Hard disk 3: Cleaned up after data movement.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Failed to backup virtual disk "Hard disk 3".

The /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-vddk.log shows:

YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_OpenEx: Open a disk.
YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO YYYY-MM-DDTHH:MM:SS error -[21154] [Originator@6876 sub=transport] Mounting 42229cff-3484-fb68-ed83-674d7a152728 using transport hotadd failed : .
YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO YYYY-MM-DDTHH:MM:SS error -[21154] [Originator@6876 sub=transport] Cannot use mode hotadd to access [Datastore] VM_name/VM_name_2-000002.vmdk: Cannot mount using this method. (Mounting 42229cff-3484-fb68-ed83-674d7a152728 using transport hotadd failed : .)
YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO VixDiskLib: Unable to locate appropriate transport mode to open disk. Error 13 (You do not have access rights to this file) (Mounting 42229cff-3484-fb68-ed83-674d7a152728 using transport hotadd failed : .) at 5294.
YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_OpenEx: Cannot open disk [Datastore] VM_name/VM_name_2-000002.vmdk. Error 13 (You do not have access rights to this file) (Mounting 42229cff-3484-fb68-ed83-674d7a152728 using transport hotadd failed : .) at 5434.
YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_Open: Cannot open disk [Datastore] VM_name/VM_name_2-000002.vmdk. Error 13 (You do not have access rights to this file) at 5478.YYYY-MM-DDTHH:MM:SS INFO:   [@(#) Build number: 66] Data Mover: Hard disk 1: Preparing for data movement.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Data Mover: Hard disk 1: VDDK: Failed to open source VMDK "[Datastore] VM_name/VM_name-000002.vmdk": VDDK Error: 13: You do not have access rights to this file.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Data Mover: Hard disk 1: Failed to initialize Block Reader.
YYYY-MM-DDTHH:MM:SS INFO:   [@(#) Build number: 66] Data Mover: Hard disk 1: Cleaned up after data movement.
YYYY-MM-DDTHH:MM:SS INFO:   [@(#) Build number: 66] Data Mover: Hard disk 3: Preparing for data movement.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Failed to backup virtual disk "Hard disk 1".
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Data Mover: Hard disk 3: VDDK: Failed to open source VMDK "[Datastore] sVM_name/VM_name_2-000002.vmdk": VDDK Error: 13: You do not have access rights to this file.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Data Mover: Hard disk 3: Failed to initialize Block Reader.
YYYY-MM-DDTHH:MM:SS INFO:   [@(#) Build number: 66] Data Mover: Hard disk 3: Cleaned up after data movement.
YYYY-MM-DDTHH:MM:SS ERROR:  [@(#) Build number: 66] Failed to backup virtual disk "Hard disk 3".

There are 100 VM backup sessions running on the vCenter, additional VMs may show in a queued state.

There are too many concurrent hot add sessions during the scheduled VMware Protection Workflow execution.  The "Performance and scalability" section of the NetWorker VMware Integration guide recommends less than 100 concurrent backup sessions per vCenter Server. NetWorker documentation is available through Support for NetWorker | Manuals & Documents 

Reduce the number of concurrent hot add sessions during the scheduled VMware Protection Workflow execution, by removing vProxies or changing the available hot add sessions per vProxy.