Avamar: an error occurred while saving the snapshot when backing VMware virtual machines
Summary: Avamar: an error occurred while saving the snapshot when backing VMware virtual machines
Symptoms
When doing image level backup for VMware virtual machines, you may see the backup fails and you notice the followings:
1. In the Avamar backup log you may see this error:
avvcbimage Info <0000>: Snapshot 'Avamar-xxxxxxxxxxxxxxxxxx' creation for VM '[<Datastore_Name>] VM_Name/VM_Name.vmx' task still in progress, sleep for 3 sec
avvcbimage Error <16279>: vSphere Task failed (snapshot error=45): 'fault.GenericVmConfigFault.summary'.
avvcbimage Error <14635>: Snapshot 'Avamar-xxxxxxxxxxxxxxxxxx' creation for VM '[<Datastore_Name>] VM_Name/VM_Name.vmx' task creation encountered a problem.
2. In the vCenter task of the virtual machine you may see this error:
An error occurred while taking a snapshot:msg.snapshot.error-QUIESCINGERROR
3. The same problem happens again when you take a non-quiesce backup of the virtual machine using --quiesce_fs=false flag in Avamar.
4. When you manually create a backup in vCenter, the creation fails.
5. You are not able to storage vMotion the problematic virtual machine.
6. In the vmware.log of the virtual machine, you may see error about storage space:
vcpu-0| I120: DISKLIB-VMFS : "/vmfs/volumes/xxxxxxxxxxxxxxxx/VM_Name/VM_Name-flat.vmdk" : closed.
| vcpu-0| W110: File_GetFreeSpace: Couldn't statfs /vmfs/volumes/xxxxxxxxxxxxxxxx/VM_Name
| vcpu-0| I120: SNAPSHOT: SnapshotCheckSpaceAndSnapshotNum: Error getting the free space on the storage disk
7. There is plenty of space on the datastore which can be verified both on the storage side and in vCenter.
Cause
Resolution
https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-D0595EB3-D20C-4951-88EF-5AFB0BF2398D.html
Note: Rescanning a datacenter might take a long time. During this time, vMotion might not be operable and some datastore operations might be disallowed. Consult your VMware admin before you rescan the storage.