Avamar - Hyper-V VMs crash during backup
Summary: Knowledgebase (KB) article explaining why Hyper-V VMs crash during Avamar backups.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
Individual virtual machines (VM) or even entire Cluster Shared Volumes (CSVs) may go offline or crash. Sometimes, the VMs cannot recover automatically on their own because it could not access the boot disks on restart. The VMs then have to be manually power-cycled to recover successfully.
While running Hyper-V backups on a CSV cluster, a VM or multiple VMs may fail when a CSV ownership change occurs. This is particularly the case with Hyper-V CSV backups in a multi-proxy configuration with the "Change CSV Ownership" feature enabled.
In the Cluster Event logs, Error 5120 might be seen:
While running Hyper-V backups on a CSV cluster, a VM or multiple VMs may fail when a CSV ownership change occurs. This is particularly the case with Hyper-V CSV backups in a multi-proxy configuration with the "Change CSV Ownership" feature enabled.
In the Cluster Event logs, Error 5120 might be seen:
"Cluster Share Volume "Volume1"('hvcluster-dell') has entered a paused state because of '(a00000a9)'. All I/O will temporarily be queued until a path to the volume is reestablished.
Cause
When a multi-proxy Hyper-V cluster is registered using the Avamar Cluster Wizard, the option "Change ownership of CSVs to one or more Proxy Nodes" is available. The "Change CSV Ownership" feature allows the backup process to automatically change CSV ownership of the VM to the proxy node that is performing the backups. When the proxy node is not the owner of the CSV being backed up, it notably slows down backup performance. The backup process must then stream data from the owner node to the primary node. It then finally sends the backup to the Avamar or Data Domain server. As a result, backup performance depends on network performance between cluster nodes. Backups are faster and more efficient when a proxy node owns the CSV and backup data streams directly from the proxy node to the backup server.
However, in some environments this performance feature may cause individual VMs or even whole CSV volumes to go offline when the ownership change occurs. This mostly happens when the disk input/utput rate is too high.
However, in some environments this performance feature may cause individual VMs or even whole CSV volumes to go offline when the ownership change occurs. This mostly happens when the disk input/utput rate is too high.
Resolution
Remove or comment out (using #) the "--change-csv-ownership" flag in the avhypervvss.cmd file using a text editor such as Notepad. This file is located in the C:\Program Files\avs\var directory on the primary proxy node. Then save and close the file.
However, disabling this feature may result in slower backup performance. Enable and increase the multi-stream count in the dataset as a workaround.
However, disabling this feature may result in slower backup performance. Enable and increase the multi-stream count in the dataset as a workaround.
Article Properties
Article Number: 000165926
Article Type: Solution
Last Modified: 11 Oct 2024
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.