NetWorker: VM Backups Fail with Error 'A Digest Operation has Failed' during SNAPSHOT Creation
Summary: NetWorker vProxy (NVP) Virtual Machine (VM) backups cancel with snapshot errors, reporting a failed digest operation; log shows corrupt digest metadata causing snapshot failure on Horizon clone VMs. ...
Symptoms
NetWorker vProxy is used to protect the vSphere environment.
Virtual Machine (VM) Backups intermittently fail with the unique error "Unable to create snapshot. Unable to create a non-quiesced snapshot after a retry attempt: An error occurred while saving the snapshot: A digest operation has failed."
The VM's vmware.log file of the impacted VM:
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DIGESTLIB-FILE : DigestLib_FlushMetaDataToDisk: digest header of parent digest has CID 0x31a5ac17 new CID of child digest is 0xb1f97036, parent digest may be corrupt, failing meta-data update on child. YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710/VMNAME01-000001-digest-sesparse.vmdk" : closed. YYYY-MM-DDTHH:MM:SS Er(02) vcpu-0 - DISKLIB-LIB_CREATE : DiskLibPostCreateChild: could not init child digest disk: Invalid parameter YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710/VMNAME01-000002-digest-sesparse.vmdk" : closed. YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-VMFS : "vsan://520220993a84143a-59c06e4db535f859/03e5fe67-0e62-18cc-461c-043201a83710" : closed. YYYY-MM-DDTHH:MM:SS Er(02) vcpu-0 - DISKLIB-LIB_CREATE : DiskLibCreateChildSparse: Failed in post processing: A digest operation has failed YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-VMFS : "vsan://520220993a84143a-59c06e4db535f859/03e5fe67-0e62-18cc-461c-043201a83710" : open successful (1114113) size = 0, hd = 0. Type 20 YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-VMFS : "vsan://520220993a84143a-59c06e4db535f859/03e5fe67-0e62-18cc-461c-043201a83710" : closed. YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - OBJLIB-VSANOBJ: VsanObjUnlinkInt: Unlinking '/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710' -- 'vsan://520220993a84143a-59c06e4db535f859/03e5fe67-0e62-18cc-461c-043201a83710' (flag: 256). YYYY-MM-DDTHH:MM:SS148Z In(05) vcpu-0 - DISKLIB-LIB : DiskLibUnlinkInt: Disk delete successfully completed { result:0, Msg: 'The operation completed successfully', fileName:'/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710/VMNAME01-000002.vmdk'} YYYY-MM-DDTHH:MM:SS Er(02) vcpu-0 - DISKLIB-LIB_CREATE : DiskLib_CreateChild: Failed to create child disk: A digest operation has failed YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710/VMNAME01-000002-digest-sesparse.vmdk" : open successful (1114113) size = 0, hd = 0. Type 19 YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710/VMNAME01-000002-digest-sesparse.vmdk" : closed. YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-LIB : DiskLibUnlinkInt: Disk delete successfully completed { result:0, Msg: 'The operation completed successfully', fileName:'/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710/VMNAME01-000002-digest.vmdk'} YYYY-MM-DDTHH:MM:SS Er(02) vcpu-0 - SNAPSHOT: SnapshotBranchDisk: Failed to branch disk: '/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710/VMNAME01-000001.vmdk' -> '/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710/VMNAME01-000002.vmdk' : A digest operation has failed (66) YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/vsan:520220993a84143a-59c06e4db535f859/2504ec67-4847-6d24-2a39-043201a83710/CAP-HC-CRYSTAL_10-000001-digest-sesparse.vmdk" : closed. YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-VMFS : "vsan://520220993a84143a-59c06e4db535f859/504bee67-f2b3-217e-6646-043201518600" : closed. YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - SNAPSHOT: SnapshotBranch failed: A digest operation has failed (5).
vmware.log files are located in the VM's datastore folder in VMware.
Cause
The issue has been observed on Horizon persistent clone VMs.
Resolution
This issue and its resolution has been addressed in VMware KB Article 373339 provided below. See the KB article below for workaround instructions.
Cannot take snapshots of VMs due to the error "An error occurred while taking a snapshot: A digest operation has failed."
https://knowledge.broadcom.com/external/article/373339/cannot-take-a-snapshot-with-the-error-a.html