Avamar: Hyper-V VM failures during backups

Individual virtual machines (VM) or even entire Cluster Shared Volumes (CSVs) may go offline or crash. Sometimes, the VMs cannot recover automatically on their own because they could not access the boot disks upon restarting. The VMs must be manually power-cycled to recover successfully.

While running Hyper-V backups on a CSV cluster, a VM or multiple VMs may fail when a CSV ownership change occurs. This is particularly the case with Hyper-V CSV backups in multiproxy configurations with the Change CSV Ownership feature enabled.

In the Cluster Event logs, Error 5120 might be seen: 

"Cluster Share Volume "Volume1"('hvcluster-dell') has entered a paused state because of '(a00000a9)'.  All I/O will temporarily be queued until a path to the volume is reestablished.

When a multiproxy Hyper-V cluster is registered using the Avamar Cluster Wizard, the option Change ownership of CSVs to one or more Proxy Nodes is available. The Change CSV Ownership feature allows the backup process to automatically change CSV ownership of the VM to the proxy node that performs the backups. Backup performance noticeably slows when the proxy node is not the owner of the CSV being backed up. The backup process must then stream data from the owner node to the primary node. It then sends the backup to the Avamar or Data Domain server. As a result, backup performance depends on network performance between cluster nodes. Backups are faster and more efficient when a proxy node owns the CSV and backup data streams directly from the proxy node to the backup server.

However, in some environments this performance feature may cause individual VMs or even whole CSV volumes to go offline when an ownership change occurs. This mostly happens when the disk input/output rate is too high.  

Remove or comment out (using #) the "--change-csv-ownership" flag in the avhypervvss.cmd file using a text editor such as Notepad. This file is located in the C:\Program Files\avs\var directory on the primary proxy node. Then save and close the file.

However, disabling this feature may result in slower backup performance. Enable and increase the multistream count in the dataset as a workaround.