Avamar: VMware image backup poor performance (RESOLUTION PATH)

Step 1: Gather a full set of 'required' information from the customer.
It is critical to have a detailed understanding of the environment before starting to troubleshoot the situation. Collect the following:

• Avamar - Information to gather when investigating client slow backup performance issues 

Also, make a note of:

• The version of vCenter in use
• The version of the proxy client in use

Step 2: Gather logs for the image backup.

• Avamar: How to gather logs for troubleshooting Avamar VMware image backup issues 

Step 3: Review known issues.
Check the 'troubleshooting' section of the Avamar for VMware User Guide for common issues. Next, check the following;

• Check if CBT is enabled for the VM.

See the following VMware articles: 

• Broadcom VMware article 1020128 Changed Block Tracking (CBT) on virtual machines

• b) Check if CBT is enabled for the Avamar VMware Image Dataset
• c) Check if the VM is generating an excessive number of changed blocks.
  • Avamar VM image backup using CBT is slow due to large number of changed blocks in the virtual disks (Log in as registered Dell Support user may be required to view article.)
• d) Check whether the proxy transport mode is using "hotadd" rather than "nbd" or "nbdssl."
  • Avamar - Fix transport mode to hotadd only in VMware image backups (Log in as registered Dell Support user may be required to view article.)
  • Avamar VM Image backup occasionally takes longer to complete due to using nbd instead of hotadd 
• e) Check for any backend storage performance issues.
  • The customer may open a support case with their vendor if third party support is experiencing an issue.
• f) Are CBT image backups running as full backups due to the issue in VMware article Reverting to a preexisting snapshot can cause incremental backups based on Changed Block Tracking to become inconsistent

Step 4: If the issue remains after the above issues have been corrected, escalate to the next level of support.

VM image backups work differently to traditional Avamar backups where the Avamar agent is installed as 'guest' service within the operating system.

During a traditional file system backup, the 'avtar' process scans the dataset and processes any files detected as new or modified. The time taken to complete the backup is proportional to the number of files in the dataset PLUS the combined size of all modified files.

A detailed discussion on this behavior is available in:

• Avamar slow backup performance - how to troubleshoot and identify bottlenecks (RESOLUTION PATH)  - How to understand Avamar Client backup performance and identify performance bottlenecks

With VM image backups, the proxy mounts the client's virtual disks. Avtar processes storage blocks, rather than files

If VMware's "Changed Block Tracking" (CBT) feature is enabled for the VM, only blocks containing changed data is processed. This minimizes data to be processed and reduces resource consumption.  

This drastically improves performance for clients with millions of files, but a low change rate.

For more information see the Avamar for VMware User Guide:

• Avamar Client for VMware Dell Support product page