NVP-vProxy: Backup stops responding with 'Excessive Depth in Document' error

The Backup Administrator configures VMware backups using the NVP vProxy appliance. Backups initially complete successfully against all vProxy appliances. Backups then stop responding against one specific vProxy. Other vProxy appliances continue processing work orders normally.

The backup log shows the backups stall at different stages for different VMs. The unresponsive behavior is not limited to a single VM.

The /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-engine.log continuously shows an XML parser error:

noname.xml:7: parser error : Extra content at the end of the document
a67fc16d47</uuid><contentId>d3df551ae74b3a5e1618dd2fa62962ef</contentId><parent>

The vmconfig command-line utility reset-backup, disable-migration, and enable-migration options generates a depth error for certain VMs:

noname.xml:7: parser error : Excessive depth in document: 256 use XML_PARSE_HUGE option
93c9a65840</uuid><contentId>ca3f9d42bfd056ff3eb2db1f8072aa4b</contentId><parent>

A virtual machine in the protection group has a significant number of files in its datastore directory. In one case, a VM had over 2000 files that were two years old.

The vProxy backup workflow gathers VM configuration details at the start of each backup. It creates an XML file that includes filenames from the VM datastore directory. The XML data is stored in the BackupVmSessions.gob file.

When the datastore directory contains excessive files, the XML exceeds the parser depth limit of 256. The parser error corrupts the BackupVmSessions.gob file. The corrupted file makes the vbackupd.service unable to process any backup work orders.

1. Remove the Affected VM From the Protection Group:

• • Identify the VM with excessive files in its datastore directory.
  • Remove the VM from the protection group.

2. Recreate the BackupVmSessions.gob file:

• • SSH to the affected vProxy appliance as root.
  • Check the status of the vbackupd service:

systemctl status vbackupd.service

• • Stop the vbackupd service:

systemctl stop vbackupd.service

• • Rename the corrupted gob file:

mv /opt/emc/vproxy/runtime/state/vbackupd/BackupVmSessions.gob /opt/emc/vproxy/runtime/state/vbackupd/BackupVmSessions.gob.bak

• • Start the vbackupd service: 

systemctl start vbackupd.service

• • Check the service status:

systemctl status vbackupd.service

• • Verify progress in the vbackupd log:

tail -f /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-engine.log

3. Clear VM Lock Flags if needed: If any VMs have stale backup lock or migration flags, use the vmconfig utility to clear them:

/opt/emc/vproxy/bin/vmconfig -v <vcenter_fqdn> -u '<vcenter_user>' -p '<vcenter_password>' -c "reset-backup" -l moref -k <vm_moref>

4. Clean the VM Datastore Directory:

• • Review the datastore directory of the removed VM. Remove old or unnecessary files to reduce the file count.
  • After cleanup, add the VM back to the protection group if needed. Verify the backup completes without the parser error.

In some instances, the vSphere Web Console reports the affected virtual machines requires consolidation. In the vSphere Web Client, select the vCenter Server VM's tab, then use the Manage Columns option to show the Needs Consolidation column. If any of the VMware Protection Policies protected virtual machines require consolidation, the virtual machines should be consolidated using the vSphere Client.

For more information about how to consolidate a virtual machine, view Brocade knowledge base article 316575 (External Link).