Avamar: Hyper-V incremental backups failing with "invalid token" errors

Summary: Avamar: Hyper-V incremental backups failing with "invalid token" errors

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Symptoms

Full backups complete successfully. Incremental backups of individual VMs complete successfully. However, doing incremental backups of more than one virtual machine (VM) fails. 

2020-03-31 23:16:20 avhypervvss Error <19328>: [AVAMAR-PROD] is unable to forward CTL sub-workorder to remote client [HYPERV-HOST03].
2020-03-31 23:16:20 avhypervvss Error <19366>: Unable to send sub-workorders to all remote clients
2020-03-31 23:18:44 avhypervvss Error <41659>: Backup did not complete successfully 

On the secondary node, we may see these errors in the logs (usually in C:\Program Files\avs\var directory):

avhypervvss Error <43488>: Backup stream generator encountered an internal error. PAX api 'beginfile:extents' reported an error 32.
avhypervvss Error <43490>: Failed in begin file C:\ClusterStorage\Volume23\WEBT01\WEBT01-OS.vhdx , for incremental backup
avhypervvss Error <43486>: Backup stream generator encountered an internal error. Failed to stream file '00000009\C:\ClusterStorage\Volume23\WEBT01\WEBT01-OS.vhdx2EA970B2-B8D9-49BB-A466-5B0AE624ADF6'.
avhypervvss Error <43488>: Backup stream generator encountered an internal error. PAX api 'endfile:extents' reported an error 32.
avhypervvss Error <43473>: Processing workorder DR VM Group 01-1587484697586#201 (pid:3032-Hyper-V VSS) in stream mode failed.
20avhypervvss Error <43475>: Sub-process spawn for workorder DR VM Group 01-1587484697586#201 (pid:3032-Hyper-V VSS) returned 536870920. Plugin exited with 'code 536870920: cannot establish connection with server (possible network or DNS failure)'

2020-04-21 11:04:10 avtar Error <41439>: Using invalid token:5ba93c9db0cff93f52b521d7420e43f6eda2784f
2020-04-21 11:04:10 avtar Error <10542>: Data Domain server "DDR.DELLEMC.com" open failed DDR result code: 4904, desc: Invalid API argument.
2020-04-21 11:04:10 avtar Error <10509>: Problem logging into the DDR server:'', only GSAN communication was enabled. 

Cause

Hyper-V Resilient Change Tracking (RCT) extents were in the token request sent to Data Domain. If there are too many RCT extents, the DDR token request fails.
The avagent software fix disables the inclusion of the RCT extents in the token request as they are not necessary.

Resolution

To resolve this issue.
  • Upgrade the Avamar Server to version 19.4.100-124 or later
  • The fix is incorporated in Avamar version 19.4 and later software releases

Additional Information

Hyper-V backups might still fail even with the avagent software fix applied on all cluster nodes. If that is the case, then other issues might be contributing to the problem. These issues such as java processes that stopped responding or DDRmaintlog rotation failures on the Avamar Utility Node need Avamar support investigation. 

Affected Products

Avamar, Avamar Server
Article Properties
Article Number: 000071777
Article Type: Solution
Last Modified: 13 Jun 2024
Version:  6
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.