Avamar: Hyper-V incremental backups failing with "INTERNAL ERROR: <0001> assert error"
Resumen: Avamar Hyper-V incremental backups failing with "INTERNAL ERROR: <0001> assert error"
Síntomas
An Avamar Hyper-V incremental backup fails with the following errors
avhypervvss Error <19328>: [AV-HYPERV] is unable to forward CTL sub-workorder to remote client [HV-HOST1].
avhypervvss Error <19366>: Unable to send sub-workorders to all remote clients
avhypervvss Error <41659>: Backup did not complete successfully.
From the HV-HOST1 logs we see the following errors:
2020/01/30-10:33:34.39000 [[hypervvss_workorder_thread_pool]Thread#1] INTERNAL ERROR: <0001> assert error (!(de->has_error())), hypervvss_avtar_handler.cpp line 1301
2020/01/30-10:33:34.39000 [[hypervvss_workorder_thread_pool]Thread#1] INTERNAL ERROR: <0001> assert error (!(dep.getdirelemp()->is_directory() == true)), hypervvss_avtar_handler.cpp line 1305
...
...
2020-01-30 02:33:34 avhypervvss Error <43497>: Backup stream generator encountered an internal error. Failed to open file 'C:\ClusterStorage\volume1\virtualmachinedisks\test_vm\Snapshots\CCB901D5-7313-45F2-887D-CA319176C6A7.vmcx'.
2020-01-30 02:33:34 avhypervvss Error <43486>: Backup stream generator encountered an internal error. Failed to stream file '00000337\C:\ClusterStorage\volume1\virtualmachinedisks\test_vm\Snapshots\CCB901D5-7313-45F2-887D-CA319176C6A7.vmcx'.
2020-01-30 02:33:34 avhypervvss Error <43473>: Processing workorder Hyper Daily Backups Schedule-HyperV Daily Backups-1580356800022#201 (pid:3032-Hyper-V VSS) in stream mode failed.
2020-01-30 02:33:35 avhypervvss Error <43475>: Sub-process spawn for workorder Hyper Daily Backups Schedule-HyperV Daily Backups-1580356800022#201 (pid:3032-Hyper-V VSS) returned 536870925. Plugin exited with 'code 536870925: externally cancelled'
2020-01-30 02:33:36 avhypervvss Error <43499>: Failed to process backup workorder Hyper Daily Backups Schedule-HyperV Daily Backups-1580356800022#201. Failed to save backup files.
2020-01-30 02:33:36 avhypervvss Error <41591>: Failed to process backup workorder. Failed to save backup files.
2020-01-30 02:33:40 avhypervvss Error <41705>: Failed to save backup files. Failed to save backup files.
2020-01-30 02:33:40 avhypervvss Error <13810>: Unable to forward a ctl message to the remote client '10.0.0.5', result code: 3.
2020-01-30 02:33:40 avhypervvss Error <41659>: Backup did not complete successfully.
In the avtar log there will be instances of the following error:
avtar] ERROR: <0001> backstreamdir::childdone error merging history stream data during phase_do_hidden while processing directory '00000328\C:\ClusterStorage\volume2\vmharddisk'
This leads to a FATAL error:
2020-01-30 02:33:35 avtar FATAL <0000>: Backup 'Hyper Daily Backups Schedule-HyperV Daily Backups-1580356800022#201' to DDR not sealed due to previous errors
Causa
By default the time limit is 1 minute.
However, sometimes this process takes longer. This is likely where the number of RCT extents being parsed during the Hyper-V RCT incremental backup is large.
Resolución
This value increases the default time limit for inter-avagent communication from 1 to 167 minutes.
For more information on adding this type of flag to Avamar, see How to supply specialized flags to Avamar backups.
Información adicional
Avamar: Hyper-V Multi-proxy backups are failing due to timeout on communication between Primary and Secondary proxy