Avamar: Backup is slow or fails with "Time out - end" due to a lot of new or changed data

This article is a solution document belonging to the Avamar client backup performance. How to identify bottlenecks (RESOLUTION PATH)
 

An Avamar backup appears to run more slowly than usual or is not able to complete within its regular backup window.

The client log shows that the backup contains a high percentage of modified files and/or a lot of new data.

Log extract showing many changed files.

2013-12-03 20:48:24 avtar Info <5156>: Backup #72 timestamp 2013-12-03 20:48:24, 2,890,814 files, 420,421 folders, 5,362 GB (2,743,151 files, 24.36 GB, 0.45% new)

This log shows that the backup scanned 2,890,814 files and that 2,743,151 of those files had been modified since the last backup. Despite this large percentage of files being modified, only 0.45% new data was sent to the Avamar server.

Log extract showing a lot of additional data.
 

2015-09-26 11:09:52 avtar Info <5157>: PARTIAL Backup #31 timestamp 2015-09-26 11:20:21, 4,591 files, 1,044 directories, 145.1 GB (2,082 files, 61.63 GB, 42.47% new)

This log extract shows that 60 GB of new data was added to the backup, compared with a total backup size of 145 GB.

The reason the backup takes longer in these situations is because avtar must scan each file in the dataset to check whether it was modified.

Modified files are broken into small chunks, assigned a SHA hash and then compared with the contents of the cache. Files not in the cache are checked with those held on the Avamar server and, if missing, they are sent across the network. This can take a long time for large files.

If a backup contains many files or large percentage of files which are frequently modified, the backup takes longer.

Avamar is behaving as expected, but the data may be unusual or some other process may be operating on the dataset to modify the files.

If a lot of new data has been added to an Avamar backup, the job takes longer to process and ingest the new files.
 

 

Avamar is behaving as designed, but the data on the client may be changing unexpectedly.

The customer should review the client and determine if the data change is intentional and due to normal activity.

Has the customer modified the dataset for the client or added a disk drive?

If the amount of new data or the number of files increases, even after the initial backup completes it may eventually no longer complete within the confines of a daily window.  

Review the backup log and compare it with a previous backup log where the backup completed within the regular amount of time.

If the amount of backup data has increased beyond the original design scope, seek assistance from Dell Professional Services or the Dell Backup and Recovery Design Centre.

The following DELL article is useful when investigating which files in a backup underwent modification and had to be processed by Avamar.

• How to identify which files took a long time to be processed during an Avamar backup

Avamar does not modify files or use the 'archive bit'.
Investigate what is causing the files or the files' metadata to be modified.

Some anti-virus software programs have been observed to modify a file's metadata attributes, either by design or accident.  

Examples of this behaviour have been observed with the following products.

• Sophos:
  • Poor backup performance for Avamar client running Sophos anti-virus  
  • 118630: Sophos Anti-Virus causing backup times to increase.
• Trend Micro:
  • See Hotfix 1383 in these release notes.