Avamar: Virtual Machine backup is slow and times out due to hardware resource bottleneck
Summary: An Avamar guest backup of a Virtual Machine runs slowly, due to a hardware resource bottleneck.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
A virtual machine which has an Avamar client installed in the guest operating system experiences erratic or poor performance.
If the backup reaches the end of the backup window without completing, Avamar reports that the backup fails with "time out - end."
Example:-
Below, we review the logs for a client which experienced erratic behavior since it was virtualized.
Below, we demonstrate a fast backup and a slow backup for the same client.
The log of the fast backup shows a peak performance rate of 2,291,881 files per hour. The file and data change rate in this backup is low.
The log status messages show that over the course of the backup, the file scan rate is constant with no fast or slow periods.
CPU utilization is also reasonably constant remaining between 40-60%.
Below let us demonstrate this graphically using an internal Dell tool which parses Avamar client logs and charts the data.
These graphs show that, under good conditions, the backup runs quickly.
The storage system is feeding avtar with files at a rate that keeps the CPU at a moderately high utilisation level.
Next we review the log for the slower backup. This backup is for the same client data but under different virtual hardware loading conditions. We see the files/hour performance is <400,000 files/hour.
Since the change rate is low, similar to the fast backup, the backup behaviors are comparable.
Below the log, a graphical representation shows that the backup performance begins quickly, then slows down (area highlighted within the red box).
The slow files-per-hour performance correlates precisely with the time period 11am to 7pm.
Note how from 11am-7pm CPU utilization drops from well above 40% to around 5-10%.
After 7pm CPU utilization fluctuates from 15-20% and files per hour performance increases, although not as high as during the initial part of the backup.
If the backup reaches the end of the backup window without completing, Avamar reports that the backup fails with "time out - end."
Example:-
Below, we review the logs for a client which experienced erratic behavior since it was virtualized.
Below, we demonstrate a fast backup and a slow backup for the same client.
The log of the fast backup shows a peak performance rate of 2,291,881 files per hour. The file and data change rate in this backup is low.
2015-09-14 00:49:20 avtar Info <5156>: Backup #12 timestamp
2015-09-14 00:49:20, 13,050,798 files, 1,352,497 folders, 9,671 GB (37,919 files, 437.1 MB, 0.00% new)
2015-09-14 00:49:20 avtar Info <6083>: Backed-up 9,671 GB in 341.66 minutes: 1,698 GB/hour (2,291,881 files/hour)
The log status messages show that over the course of the backup, the file scan rate is constant with no fast or slow periods.
CPU utilization is also reasonably constant remaining between 40-60%.
Below let us demonstrate this graphically using an internal Dell tool which parses Avamar client logs and charts the data.
These graphs show that, under good conditions, the backup runs quickly.The storage system is feeding avtar with files at a rate that keeps the CPU at a moderately high utilisation level.
Next we review the log for the slower backup. This backup is for the same client data but under different virtual hardware loading conditions. We see the files/hour performance is <400,000 files/hour.
Since the change rate is low, similar to the fast backup, the backup behaviors are comparable.
Below the log, a graphical representation shows that the backup performance begins quickly, then slows down (area highlighted within the red box).
2015-09-22 07:41:35 avtar Info <5157>: PARTIAL Backup #35 timestamp
2015-09-22 07:41:35, 9,190,850 files, 1,030,683 folders, 8,283 GB (22,390 files, 397.8 MB, 0.00% new)
2015-09-22 07:41:35 avtar Info <6083>: Backed-up 8,283 GB in 1440.19 minutes: 345 GB/hour (382,902 files/hour)
The slow files-per-hour performance correlates precisely with the time period 11am to 7pm.
Note how from 11am-7pm CPU utilization drops from well above 40% to around 5-10%.
After 7pm CPU utilization fluctuates from 15-20% and files per hour performance increases, although not as high as during the initial part of the backup.
Cause
The dataset for our 'example client' is predictable and the change rate is negligible. The client performs a similar amount of work each day,
The erratic performance observed is due to contention for shared hardware resources used by Avamar whilst scanning files.
In the example shown above, hardware resources are either in high demand or are throttled for the client during peak business hours.
Behavior depends on available resources. The most likely resource to be bottlenecked is random seek I/O requests on the storage device where the data is located.
If CPU usage spikes during peak hours, the bottleneck may be a shortage of CPU shares available to the virtual machine.
The erratic performance observed is due to contention for shared hardware resources used by Avamar whilst scanning files.
In the example shown above, hardware resources are either in high demand or are throttled for the client during peak business hours.
Behavior depends on available resources. The most likely resource to be bottlenecked is random seek I/O requests on the storage device where the data is located.
If CPU usage spikes during peak hours, the bottleneck may be a shortage of CPU shares available to the virtual machine.
Resolution
- Run the backup during off-peak hours and (if possible), when the client is not heavily occupied with its regular duties.
- Ensure that the virtual machine hypervisor is not enforcing CPU, memory, or I/O limitations at the time of the backup window
- Avoid overlapping Avamar backups with periods of high activity where contention for shared storage is high.
- For low change rate applications, consider using image-based backup. The image backup proxy appliance backs up only modified blocks. It does not scan every file in the dataset so has lower I/O requirements.
Additional Information
For more information about how to investigate Avamar backup performance issues, see;
For more information about analyzing and interpreting avtar status messages;
For more information about analyzing and interpreting avtar status messages;
Affected Products
AvamarProducts
AvamarArticle Properties
Article Number: 000164580
Article Type: Solution
Last Modified: 10 Feb 2025
Version: 4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.