NVP-vProxy: VM backup fails with "Unable to create non-quiesced snapshot after a retry attempt: Detected an invalid snapshot configuration"

The Backup session log shows:

...
YYYY-MM-DD HH:MM:SS INFO:   [167] Creating snapshot of virtual machine ...
YYYY-MM-DD HH:MM:SS ERROR:  [167] Unable to create non-quiesced snapshot after a retry attempt: Detected an invalid snapshot configuration.
YYYY-MM-DD HH:MM:SS ERROR:  [167] Unable to create snapshot.
YYYY-MM-DD HH:MM:SS INFO:   [167] Enabling storage migration of virtual machine ...
YYYY-MM-DD HH:MM:SS INFO:   [167] Storage migration successfully enabled: "vm-4030" @ "my_vm_name.my_domain.com" 
YYYY-MM-DD HH:MM:SS WARN:   [167] Virtual machine left locked by backup session because cleanup is required.  Snapshot was not removed.
...

VM Backup session logs can be found on the NetWorker server in the following locations:

• Linux: /nsr/logs/policy/POLICY_NAME/WORKFLOW_NAME/
• Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\policy\POLICY_NAME\WORKFLOW_NAME

The VM backup logs follow the NetWorker server jobsdb retention (default 72 hours). After the retention period has passed the logs are removed. The logs are retained longer on the vProxy appliance in the following location:

• /opt/emc/vproxy/runtime/logs/recycle/vbackupd/DATE

The vmware-###.log in the VMs "datastore folder" in VMware shows:

...
YYYY-MM-DD HH:MM:SS| vcpu-0| I125: SNAPSHOT: SnapshotBranchDisk: Failed to branch disk: '/vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME.vmdk' -> '/vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME-000004.vmdk' : The file already exists (39)
YYYY-MM-DD HH:MM:SS| vcpu-0| I125: DiskLib_IsVMFSSparseSupported: vmfssparse is not supported on /vmfs/volumes/DATASTORE_UUID/VM_NAME: f532.
YYYY-MM-DD HH:MM:SS| vcpu-0| I125: DISKLIB-LIB_CREATE   : DiskLibCreateCreateParam: Selecting the default child type as SeSparse for /vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME-000005.vmdk.
YYYY-MM-DD HH:MM:SS| vcpu-0| I125: DISKLIB-LIB_CREATE   : DiskLibCreateCreateParam: seSparse grain size is set to 8 for '/vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME-000005.vmdk'
YYYY-MM-DD HH:MM:SS| vcpu-0| I125: SNAPSHOT: SnapshotBranchDisk failed for '/vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME.vmdk' : A required file was not found (7).
YYYY-MM-DD HH:MM:SS| vcpu-0| I125: DISKLIB-VMFS  : "/vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME-flat.vmdk" : closed.
YYYY-MM-DD HH:MM:SS| vcpu-0| I125: SNAPSHOT: SnapshotBranch failed: A required file was not found (7).
...

The issue can occur when there are orphaned temporary snapshot descriptor files in the virtual machine data store directory. These snapshot descriptor files appear as ####.-000001.vmdk, ####.-000001-sesparse.vmdk and ####.-000001-ctk.vmdk.

This issue is occurring within VMware. The steps to remove the orphaned temporary snapshot descriptor files are documented in VMware article: "Detected an invalid snapshot configuration" error creating a snapshot   

The following steps must be performed by the VMware Administrator, if additional assistance is required, VMware support must be engaged.

1. Browse affected VMs Datastore to identify the temporary descriptor files. Name may appear as: 
   • ####-000001.vmdk
   • ####-000001-sesparse.vmdk
   • ####-000001-ctk.vmdk 
2. Identify a folder with enough space to move the temporary descriptor files and create a new folder
   • vmfs/volumes/VM_####_DATASTORE5/FOLDERNAME/
3. Move the temporary descriptor files to the newly created folder
4. Take a snapshot of the VM and validate the snapshot created successfully.
5. Delete all the snapshots with the "Delete All Snapshots" option.
6. Validate snapshot deleted.
7. Validate there are no descriptor files in the VM datastore folder.
8. Let the scheduled vProxy Backups Run.