Avamar: Hyper-V Backup Fails with 'windows Structured Exception ACCESS_VIOLATION' Error

Hyper-V cluster backups may fail or complete with the following errors:

2021-01-13 01:18:07 avhypervvss Error <42448>: 1 remote client(s) failed to complete operation on the remote file targets
2021-01-13 01:18:07 avhypervvss Info <43400>: Attempting to commit checkpoint for targets with backup success...
2021-01-13 01:18:07 avhypervvss Info <43409>: Attempting to convert backup CheckPoint to ReferrencePoint.
2021-01-13 01:18:07 avhypervvss Error <43712>: Failed to backup Virtual Machine 'PS001 <VM GUID>'
2021-01-13 01:18:07 avhypervvss Error <43712>: Failed to backup Virtual Machine 'CN001 <VM GUID>'
2021/01/13-09:18:45.98200 [hypervvss_assist]  ERROR: <41036> threadimpl::runthread() - Windows structured exception ACCESS_VIOLATION in threadID 12828
2021-01-13 01:18:45 avhypervvss Warning <41036>: threadimpl::runthread() - Windows structured exception ACCESS_VIOLATION in threadID 12828
2021/01/13-09:18:46.07500 [hypervvss_assist]  ERROR: <41037> 0: OPENSSL_Applink()+0x277287 [0xa1683e87]

The avhypervvss.exe hotfix binaries have not been properly applied to the cluster nodes. If they are not applied in the correct order, the result leads to intermittent backup failures.

Unregister, uninstall, and reinstall Avamar client software and Hyper-V plug-in on all cluster nodes. And then apply the latest patches, depending on the Avamar version.

Before deploying the hotfix binaries, confirm that there are no active backups or restores running on clients. Save the flags and configuration set in avhypervvss.cmd and avatar.cmd.

1. Run Avamar cluster configuration wizard and unregister the cluster client.
2. Uninstall Avamar Hyper-V plug-in on all nodes of the cluster.
3. Uninstall Avamar Windows client on all nodes of the cluster.
4. Restart the node if GLR plug-in configured on any of the nodes.
5. Cleanup stale VM groups if any on all nodes of the cluster. Engage DELL support if necessary.

6. Install Avamar Hyper-V plugin software version 19.10.100-135 on all nodes of the cluster, and on all nodes of the cluster. 
   1. Go into the services.msc console on each node and stop the "DELL Avamar Plug-in Service for Windows" service on each node.
   2. Rename the original winclustersvc.exe to winclustersvc-ORIG.exe.
   3. Rename the original winclustersvc.tlb to winclustersvc-ORIG.tlb
   4. Rename the original avhypervvss.exe to avhypervvss-ORIG.exe
   5. Copy the new HF binaries avhypervvss.exe, winclustersvc.exe, and winclustersvc.tlb to "bin" folder on all nodes.
   6. Restart the "DELL Avamar Plug-in Service for Windows" service on all the cluster nodes.
   7. Confirm that the version is correct. Run the following command:

avhypervvss.exe --version

Output should look similar to the following:

Version:     19.10.100-135_HF338820

  7. Run Avamar cluster configuration wizard and configure the cluster client.

8. Set back flags and configuration in avhypervvss.cmd and avatar.cmd and run backups.

For the Hyper-V backups completing with errors but not available for restore in the Avamar Administrator console, the fix has been incorporated into Avamar v19.3 hotfix #327477 and higher.

For Hyper-V RCT backups, configuration of the "shared storagepath parameter" is required starting in v19.4 and higher. Use the following KB article with instructions for configuration and installation of the latest hotfixes for these newer Avamar versions:

KB Avamar Hyper-V VSS plug-in: Config and Install of Latest Cumulative HFs for 19.4 and later-