Avamar: Hyper-V Level 0 backup with RCT is slow
Summary: Hyper‑V Level 0 backup with Resilient Change Tracking (RCT) runs at only about130 GB an hour with high CPU. This is because the plug‑in writes 64 KB pieces while the pipe is limited to 16 KB, reducing throughput. ...
Symptoms
Avamar Hyper-V Full backup with RCT is slow.
The backup of the same Virtual Machine (VM) using traditional Volume Shadow Copy Service (VSS) cluster client shows a better performance (four to five times faster).
Here is a snippet from the full backup log. From the avtar log, the CPU usage is high, however the average GB per hour rate is about 130:
2019-01-30 10:45:56 avtar Info <8688>: Status 2019-01-30 10:45:56, 6 files, 2 folders, 5,606 GB (6 files, 22.19 GB, 0.66% new) 1065MB 130% CPU 6019469459456 input bytes, 00000001\<PATH_TO_VHDX_DISK> 2019-01-30 11:01:03 avtar Info <8688>: Status 2019-01-30 11:01:03, 6 files, 2 folders, 5,633 GB (6 files, 22.19 GB, 0.66% new) 1144MB 129% CPU 6048233734144 input bytes, 00000001\<PATH_TO_VHDX_DISK> 2019-01-30 11:16:09 avtar Info <8688>: Status 2019-01-30 11:16:09, 6 files, 2 folders, 5,660 GB (6 files, 22.19 GB, 0.66% new) 1107MB 131% CPU 6077491691520 input bytes, 00000001\<PATH_TO_VHDX_DISK> 2019-01-30 11:31:16 avtar Info <8688>: Status 2019-01-30 11:31:16, 6 files, 2 folders, 5,692 GB (6 files, 22.19 GB, 0.66% new) 1141MB 138% CPU 6111952240640 input bytes, 00000001\<PATH_TO_VHDX_DISK>
The synthetic avtar degenerate and Data Domain ddpconcheck tests do not indicate any bottleneck in the disk or network area. They also show that the boost layer can send the information at more than 500+ GB per hour.
Cause
Avamar Hyper-V plug-in writes a 64KB piece size pax stream to avtar, however, the pipe was resized to 16KB which made the performance lower than expected.
Resolution
Upgrade the Avamar Server to version 19.4.100-124 or later.
The fix is in Avamar 19.4 and later software releases which introduce multi read and buffer methods in avtar.