Avamar: The VM image level backup is failing with snapshot quiesce error
Summary: This article provides information about Virtual Machine (VM) image level backup that is failing with snapshot quiesce error.
Symptoms
The VM image backup is failing with snapshot quiesce error as shown below.
avvcbimage Warning <16277>: vSphere Task failed (application quiesce): 'An error occurred while quiescing the virtual machine. See the virtual machine's event log for details.
avvcbimage Warning <43652>: Snapshot 'Avamar-1577539864a83b60c2d81fb0e6234f5e956e1bedb51d697138' creation for VM '[XXYY-XXYY01] testxyz2473_1/testxyz2473_1.vmx' task creation encountered a quiesce problem
avvcbimage Error <43650>: File level consistent snapshot quiesce failure.
When taking a manual snapshot, it is working well.
Cause
There was a resource issue in the VM environment with disk I/O and memory usage. VMware vCenter version 6.5 and later introduced a new Volume Shadow-copy Services (VSS) feature which is called extended snapshots (CreateSnapshotEx_Task) superseding old feature "CreateSnapshot_Task."
A note from the VMware release notes is as below.
The release 6.5 includes support for additional Volume Shadow-copy Service (VSS) backup configurations.
- The
vssBackupType VSS_BT_COPYwas previously used as the default forCreateSnapshot_Taskbut NowVSS_BT_FULL, VSS_BT_INCREMENTAL, VSS_BT_DIFFERENTIAL,andVSS_BT_LOGare available as well. - The
vssBackupContextis introduced to enforce application or file system quiescing. - The timeout (default = 15 minutes) for quiescing VMs can be configured anywhere from 5 minutes to 4 hours.
To support these new features and configurations, the function CreateSnapshotEx_Task was added to the vSphere API, superseding CreateSnapshot_Task.
Resolution
When the VM team tried for the extended snapshot manually using vCenter Managed Object Browser (MOB), it was failed due to VMware related issues. The backup issue got resolved after the VMware team fixed all VMware related issues including disk I/O issue on that VM and we were able to take snapshot both from MOB and Avamar backup.