Avamar: Hyper-V incremental backups failing with "invalid token" errors
Summary: Avamar: Hyper-V incremental backups failing with "invalid token" errors
Symptoms
Full backups complete successfully. Incremental backups of individual VMs complete successfully. However, doing incremental backups of more than one virtual machine (VM) fails.
2020-03-31 23:16:20 avhypervvss Error <19328>: [AVAMAR-PROD] is unable to forward CTL sub-workorder to remote client [HYPERV-HOST03]. 2020-03-31 23:16:20 avhypervvss Error <19366>: Unable to send sub-workorders to all remote clients 2020-03-31 23:18:44 avhypervvss Error <41659>: Backup did not complete successfully
On the secondary node, we may see these errors in the logs (usually in C:\Program Files\avs\var directory):
avhypervvss Error <43488>: Backup stream generator encountered an internal error. PAX api 'beginfile:extents' reported an error 32. avhypervvss Error <43490>: Failed in begin file C:\ClusterStorage\Volume23\WEBT01\WEBT01-OS.vhdx , for incremental backup avhypervvss Error <43486>: Backup stream generator encountered an internal error. Failed to stream file '00000009\C:\ClusterStorage\Volume23\WEBT01\WEBT01-OS.vhdx2EA970B2-B8D9-49BB-A466-5B0AE624ADF6'. avhypervvss Error <43488>: Backup stream generator encountered an internal error. PAX api 'endfile:extents' reported an error 32. avhypervvss Error <43473>: Processing workorder DR VM Group 01-1587484697586#201 (pid:3032-Hyper-V VSS) in stream mode failed. 20avhypervvss Error <43475>: Sub-process spawn for workorder DR VM Group 01-1587484697586#201 (pid:3032-Hyper-V VSS) returned 536870920. Plugin exited with 'code 536870920: cannot establish connection with server (possible network or DNS failure)' 2020-04-21 11:04:10 avtar Error <41439>: Using invalid token:5ba93c9db0cff93f52b521d7420e43f6eda2784f 2020-04-21 11:04:10 avtar Error <10542>: Data Domain server "DDR.DELLEMC.com" open failed DDR result code: 4904, desc: Invalid API argument. 2020-04-21 11:04:10 avtar Error <10509>: Problem logging into the DDR server:'', only GSAN communication was enabled.
Cause
Hyper-V Resilient Change Tracking (RCT) extents were in the token request sent to Data Domain. If there are too many RCT extents, the DDR token request fails.
The avagent software fix disables the inclusion of the RCT extents in the token request as they are not necessary.
Resolution
- Upgrade the Avamar Server to version 19.4.100-124 or later
- The fix is incorporated in Avamar version 19.4 and later software releases
Additional Information
Hyper-V backups might still fail even with the avagent software fix applied on all cluster nodes. If that is the case, then other issues might be contributing to the problem. These issues such as java processes that stopped responding or DDRmaintlog rotation failures on the Avamar Utility Node need Avamar support investigation.