NVP vProxy: VM Disk is Removed or Restored as an Empty Drive

A VM restore performed using the NetWorker VMware Protection (NVP) vProxy appliance. One of the following conditions appears:

• The VM restore fails, and the VM shows a missing or zeroed out disk.
• The VM restore succeeds; however, the recovered VM fails to power on because the OS disk is missing.
• The VM restore succeeds, and the VM powers on; however, one of the data disks is missing from the recovered VM.

The recovered VM shows "zeroed out" disk:

Figure 1: VM Shows a Missing or Zeroed Out Disk.

VM Restore fails with the following error seen in the VM recover session log:

YYYY-MM-DDTHH:MM:SSZ DEBUG:  [105] Data Mover: Hard disk 3: Reporting disk status error: Failed to initialize Block Writer.  Failed to open source VMDK '[DATASTORE] VM_NAME/VM_NAME_2.vmdk': VDDK Error: 13: You do not have access rights to this file.

The vProxy's /opt/emc/vproxy/runtime/logs/vrecoverd/vrecoverd-vddk.log may show a "Not found" alert.

YYYY-MM-DDTHH:MM:SSZ NOTICE: VDDK INFO YYYY-MM-DDTHH:MM:SSZ3Z error -[13331] [Originator@6876 sub=transport] Unmounting 42126e5f-f18a-8243-01ea-6a29456f8ecb using transport hotadd failed : Got bad HTTP response during feteching file (Not Found).

The vProxy VMs vmware.log shows a vMotion was initiated during the restore.

YYYY-MM-DDTHH:MM:SSZ In(05) vmx - SVMotion: Enter Phase 1
....
YYYY-MM-DDTHH:MM:SSZ In(05) worker-2107995 - SVMotionCleanupThread: Cleanup complete.

VMware Distributed Resource Scheduler (DRS) is enabled. The vProxy VM is vMotioned in the VMware environment during the restore operation. When a VM is vMotioned, it claims any disks attached to it. During a VM restore, the VM's disks attach to the vProxy. If a vMotion occurs during the restore, the disks are renamed to belong to the vProxy.

Depending on how many disks the VM has and when the vMotion occurs on the vProxy, it is possible that some disks may restore correctly. When vMotion starts, any pending disks fail. The VM disk is renamed, so the recovery process cannot find the VMware Virtual Machine Disk (VMDK) using the VM name.

This is expected functionality based on vMotion design. See VMware article 1029513.

Automatic vMotion operations should be avoided for vProxy VMs. If a vProxy VM is vMotioned mid backup or restore, it may cause unexpected issues. Disable DRS on the vProxy VM.

Consult with your VMware administrator or support team for making these changes.

Similar issues may appear due to Change Block Tracking (CBT) File (CTK) corruption: NVP-vProxy: VM recovery is marked as successful but the VM does not boot with the error: Missing Operating System