Avamar: Hyper-V incremental backups failing with "INTERNAL ERROR: <0001> assert error"

Summary: Avamar Hyper-V incremental backups failing with "INTERNAL ERROR: <0001> assert error"

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

An Avamar Hyper-V incremental backup fails with the following errors
avhypervvss Error <19328>: [AV-HYPERV] is unable to forward CTL sub-workorder to remote client [HV-HOST1].
avhypervvss Error <19366>: Unable to send sub-workorders to all remote clients
avhypervvss Error <41659>: Backup did not complete successfully.

From the HV-HOST1 logs we see the following errors:
2020/01/30-10:33:34.39000 [[hypervvss_workorder_thread_pool]Thread#1]  INTERNAL ERROR: <0001> assert error (!(de->has_error())), hypervvss_avtar_handler.cpp line 1301
2020/01/30-10:33:34.39000 [[hypervvss_workorder_thread_pool]Thread#1]  INTERNAL ERROR: <0001> assert error (!(dep.getdirelemp()->is_directory() == true)), hypervvss_avtar_handler.cpp line 1305
...
...
2020-01-30 02:33:34 avhypervvss Error <43497>: Backup stream generator encountered an internal error. Failed to open file 'C:\ClusterStorage\volume1\virtualmachinedisks\test_vm\Snapshots\CCB901D5-7313-45F2-887D-CA319176C6A7.vmcx'.
2020-01-30 02:33:34 avhypervvss Error <43486>: Backup stream generator encountered an internal error. Failed to stream file '00000337\C:\ClusterStorage\volume1\virtualmachinedisks\test_vm\Snapshots\CCB901D5-7313-45F2-887D-CA319176C6A7.vmcx'.
2020-01-30 02:33:34 avhypervvss Error <43473>: Processing workorder Hyper Daily Backups Schedule-HyperV Daily Backups-1580356800022#201 (pid:3032-Hyper-V VSS) in stream mode failed.
2020-01-30 02:33:35 avhypervvss Error <43475>: Sub-process spawn for workorder Hyper Daily Backups Schedule-HyperV Daily Backups-1580356800022#201 (pid:3032-Hyper-V VSS) returned 536870925. Plugin exited with 'code 536870925: externally cancelled'
2020-01-30 02:33:36 avhypervvss Error <43499>: Failed to process backup workorder Hyper Daily Backups Schedule-HyperV Daily Backups-1580356800022#201. Failed to save backup files.
2020-01-30 02:33:36 avhypervvss Error <41591>: Failed to process backup workorder. Failed to save backup files.
2020-01-30 02:33:40 avhypervvss Error <41705>: Failed to save backup files. Failed to save backup files.
2020-01-30 02:33:40 avhypervvss Error <13810>: Unable to forward a ctl message to the remote client '10.0.0.5', result code: 3.
2020-01-30 02:33:40 avhypervvss Error <41659>: Backup did not complete successfully.

In the avtar log there will be instances of the following error:
avtar]  ERROR: <0001> backstreamdir::childdone error merging history stream data during phase_do_hidden while processing directory '00000328\C:\ClusterStorage\volume2\vmharddisk'

This leads to a FATAL error:
2020-01-30 02:33:35 avtar FATAL <0000>: Backup 'Hyper Daily Backups Schedule-HyperV Daily Backups-1580356800022#201' to DDR not sealed due to previous errors
 

Cause

The issue is due to the inter-avagent communication between the primary and secondary nodes timing out. 

By default the time limit is 1 minute.  

However, sometimes this process takes longer.  This is likely where the number of RCT extents being parsed during the Hyper-V RCT incremental backup is large.

Resolution

Edit the avhypervvss.cmd file in the var directory and add the flag --request-ctl-timeout=600000. 

This value increases the default time limit for inter-avagent communication from 1 to 167 minutes.

For more information on adding this type of flag to Avamar, see How to supply specialized flags to Avamar backups.

Additional Information

More information about the issue can be found in KB 491152.
Avamar: Hyper-V Multi-proxy backups are failing due to timeout on communication between Primary and Secondary proxy

Affected Products

Avamar

Products

Avamar, Avamar Plug-in for Hyper-V VSS, Avamar Server
Article Properties
Article Number: 000071523
Article Type: Solution
Last Modified: 03 Mar 2025
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.