Avamar: Hyper-V Backup Fails with 'widows Structured Exception ACCESS_VIOLATION' Error

Hyper-V cluster backups may fail or complete with the following errors:

2021-01-13 01:18:07 avhypervvss Error <42448>: 1 remote client(s) failed to complete operation on the remote file targets
2021-01-13 01:18:07 avhypervvss Info <43400>: Attempting to commit checkpoint for targets with backup success...
2021-01-13 01:18:07 avhypervvss Info <43409>: Attempting to convert backup CheckPoint to ReferrencePoint.
2021-01-13 01:18:07 avhypervvss Error <43712>: Failed to backup Virtual Machine 'PS001 <VM GUID>'
2021-01-13 01:18:07 avhypervvss Error <43712>: Failed to backup Virtual Machine 'CN001 <VM GUID>'
2021/01/13-09:18:45.98200 [hypervvss_assist]  ERROR: <41036> threadimpl::runthread() - Windows structured exception ACCESS_VIOLATION in threadID 12828
2021-01-13 01:18:45 avhypervvss Warning <41036>: threadimpl::runthread() - Windows structured exception ACCESS_VIOLATION in threadID 12828
2021/01/13-09:18:46.07500 [hypervvss_assist]  ERROR: <41037> 0: OPENSSL_Applink()+0x277287 [0xa1683e87]

The avhypervvss.exe hotfix binaries have not been properly applied to the cluster nodes.

If they are not applied in the correct order, it results in intermittent backup failures.

Unregister, uninstall, and reinstall Avamar client software and Hyper-V plug-in on all cluster nodes. And then apply the latest patches, depending on the Avamar version.

Before deploying the hotfix binaries, confirm that there are no active backups or restores running on clients.

1. Run Avamar cluster configuration wizard and unregister the cluster client.
2. Uninstall Avamar Hyper-V plug-in on all nodes of the cluster.
3. Uninstall Avamar Windows client on all nodes of the cluster.
4. Restart the node if the Hyper-V Granular Level Recovery (GLR) plug-in configured on any of the nodes.
5. Clean up the stale "vmgroups" if any on all nodes of the cluster. Engage DELL support if necessary.
6. Install Avamar Hyper-V plugin software version 19.10.100-135 on all nodes of the cluster, and on all nodes of the cluster.

   1. Go into the services.msc console on each node and stop the "DELL Avamar Plug-in Service for Windows" service on each node.
   2. Rename the original winclustersvc.exe to winclustersvc-ORIG.exe.
   3. Rename the original winclustersvc.tlb to winclustersvc-ORIG.tlb
   4. Rename the original avhypervvss.exe to avhypervvss-ORIG.exe
   5. Copy the new hotfix (HF) binaries avhypervvss.exe, winclustersvc.exe, and winclustersvc.tlb to the "bin" folder on all nodes.
   6. Restart the "DELL Avamar Plug-in Service for Windows" service on all the cluster nodes.
   7. Confirm that the version is correct. Run the following command:

      avhypervvss.exe --version

      Output should look similar to the following:

      Version:     19.10.100-135_HF338820

7. Run Avamar cluster configuration wizard and configure the cluster client.
8. Set back flags and configuration in avhypervvss.cmd and avtar.cmd and run backups.

For Hyper-V RCT backups, configuration of the "shared storagepath parameter" is required from v19.4 and higher. 

Use the following article for instructions on the configuration and installation of the latest hotfixes for these newer Avamar versions: Avamar: Hyper-V VSS plug-in: Config and Install of Latest Cumulative HFs for 19.4 and later-