Avamar: Hyper V RCT Backups Fail with Errors Due to Orphaned Checkpoints

Avamar Hyper-V RCT backups may fail with errors similar to the following:
 

avhypervvss Info <43419>: Creating backup CheckPoint with 'Copy' guest backup type
2019-04-15 00:05:05 avhypervvss Info <43420>: Creating backup CheckPoint with 'Application' consistenty level
avhypervvss Error <43426>: Checkpoint creation for Virtual machine 'DELLEMC-VM<XXXXSS-XXXX-XXXX-XXXX-XXXXXX>' has failed and will be skipped from backup. For details, please check 'winclustersvc.log'.
avhypervvss Error <43426>: Checkpoint creation for Virtual machine 'AVAMAR-DEV<XXXXSS-XXXX-XXXX-XXXX-XXXXXX>' has failed and will be skipped from backup. For details, please check 'winclustersvc.log'.
avhypervvss Error <43426>: Checkpoint creation for Virtual machine 'DDR-Dev2<XXXXX-XXXX-XXXX-XXXX-XXXXXX>' has failed and will be skipped from backup. For details, please check 'winclustersvc.log'.
avhypervvss Warning <43427>: No targets available for backup after checkpoint process.

Hyper-V RCT backups may complete successfully for a day or two. However, the last successful backup is leaving orphaned checkpoint files (*.avhdx) in the virtual machine (VM) directories.  

A checkpoint of the VMs is created temporarily during the backup. After it finishes, the Hyper-V plug-in informs the Windows operating system (OS) to merge the AVHD checkpoint into the VM's parent virtual hard disk (VHD).

Sometimes, the checkpoints do not merge into the main VHD after the backups complete. From the Event Viewer log (Application and Services logs -> Microsoft -> Windows -> Hyper-V VMMS), the following error may display:
 

"'DDR-Dev2' background disk merge failed to complete: Account restrictions are preventing this user from signing in. For example: blank passwords aren't allowed, sign-in times are limited, or a policy restriction has been enforced. (0x8007052F). (Virtual machine ID XXXXX-XXXX-XXXX-XXXX-XXXXXX) "

In the worst-case scenario, the checkpoints might not be visible either in the Hyper-V Manager or from the PowerShell commands.

Both Hyper-V and the Avamar plug-in are unaware of the orphaned checkpoints. The next Avamar backups fail as the Windows OS is unable to create new checkpoint AVHD files due to the existing orphaned checkpoints.

 

From the Avamar logs, checkpoint cleanup was run from the Avamar Hyper-V plugin side successfully. However, there was an error simultaneously in Event Viewer, that it failed to merge the disk which caused the checkpoint deletion error.

From the Winclustersvc.log file, the snippet below shows that checkpoint cleanup was done successfully:
 

winclustersvc INFO<0000>:  Cleaning up checkpoint created on group AvamarBackup_SERVER-app-20190510000504-0: checkpoint \\SERVER\root\virtualization\v2:Msvm_SnapshotCollection.CollectionID="DXXXXDX-XXCE-XXDX-XABF-XXXDXXXXXX" . 
winclustersvc INFO<0000>:  DestroySnapshot executed successfully 

The plug-in calls the Microsoft API to delete the checkpoint, and only after it returns successfully does the plugin log the above message.  

But simultaneous Event Viewer logs give the following error for all the VMs, which caused the checkpoint to not get deleted:
 

 "'DDR-Dev2' background disk merge failed to complete: Account restrictions are preventing this user from signing in. For example: blank passwords aren't allowed, sign-in times are limited, or a policy restriction has been enforced. (0x8007052F). (Virtual machine ID XXXXX-XXXX-XXXX-XXXX-XXXXXX) "

Open a support case with Microsoft for more troubleshooting of the orphaned checkpoint issue.

Here are some workarounds which may resolve the issue:

1. Install the latest patches and updates on the Hyper-V host machines using Windows Update.
2. Restart the Hyper-V Virtual Machine Management (VMMS) service before the start of the Avamar backup. The Log on as a service Group Policy Object property is not synchronized with the VMMS service after creating a scheduled task. When the VMMS service is restarted, it tries to forcefully synchronize with the Microsoft logon service.
3. Disable the User rights assignment/Log on as a Service property in the Group Policy Object (GPO). Afterwards, run the following command on the Hyper-V machine using Windows command or PowerShell prompt: 

GPUpdate / Force 

Reference:  https://social.technet.microsoft.com/Forums/en-US/1dc1a820-be03-4c24-9f85-5793eb6c1d4a/background-merge-fails-replication-state-critical?forum=winserverhyperv  (External Link)