Avamar: Backup performance impacted by anti-virus software real-time file scan
Summary: Anti-virus software for real-time scanning of files is impacting Avamar client backup or restore performance.
Symptoms
The following points are true:
- Status messages in the client log show low CPU usage by
avtar.exe(see Dell article Avamar - How to interpret avtar backup log status lines) - The backup is not bottlenecked due to network congestion.
- Anti-virus software is installed on the client.
- The anti-virus software is set to scan files in real time "on access" (when an application accesses files)
- Processes tab > View > Select Columns
- Enable the I/O Read Bytes, and I/O Reads columns.
- While a backup is in progress, monitor the anti-virus process.
- If the I/O Read Bytes or I/O Reads numbers for the anti-virus application are heavily incremented during the backup, this may indicate an issue.
Cause
The anti-virus on access scanning feature contends with the avtar process for the storage I/O by intercepting files which avtar is trying to access.
The anti-virus may be scanning snapshot storage which contents with avtar process for access to the snapshot that was taken as part of the backup.
This extra load on the storage hardware reduces the potential file scan speed of the backup.
Resolution
Temporarily configure a test Virtual Machine (VM) to do client-level backup with no anti-virus installed record timing then install anti-virus and compare.
Compare the performance of the latest backup with earlier backups where real-time scanning was active. To do this, review a complete client log for Backed-up.
Manually add --status=120 in additional options to increase the frequency of status messages for testing.
avtar Info <6083>: Backed-up 344.3 GB in 862.43 minutes: 24 GB/hour (1,508,688 files/hour)
In addition to the
avtar.exe, the executables to trust or safe-list from anti-virus ON ACCESS SCANNING are below. The files are typically installed in the c:\program files\avs\bin folder:
avsql.exe Microsoft SQL Server avexchglr.exe Exchange Server GLR avexvss.exe Exchange Server axionfs.exe Exchange Server GLR avoracle.exe Oracle avlotus.exe Lotus Notes avscc.exe Desktop Icon avagent.exe Communication service between the client and Avamar server avvss.exe Disaster Recovery, BMR avmossvss.exe SharePoint Servers avupdate.exe Client update tool
- Excluding the Directory:
- Find the anti-virus Settings: Locate your anti-virus software's settings or configuration panel.
- Access Scan Exclusion List: Navigate to the section for add or manage scan exclusions, often labeled as "Exclusions," "Scan Exclusion List," or similar.
- Add the Directory: Type or paste the path
\Device\HardDiskVolumeShadowCopy*into the exclusion list and save the changes.
-
Excluding the Process:
- Find the anti-virus Settings: Locate your anti-virus software's settings or configuration panel.
- Access Process Exclusions: Find the section for add or manage process exclusions.
- Add the Process: Add
"vssvc.exe"to the excluded processes list.
To list the absolute path of the Shadow Volume, use an elevated Command Prompt, and run the "vssadmin List Shadows" command.
\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy*\
\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy*
After resolving this type of interference from anti-virus software, if backup performance is still low, see Dell article: Avamar slow backup performance - how to troubleshoot and identify bottlenecks (RESOLUTION PATH)
Additional Information
The behavior described occurs with any anti-virus program that performs "real-time" or "on-access" scanning.
Below are links to suggested articles of various anti-virus vendors' online documentation that discuss the behavior or how to mitigate it.
Some anti-virus programs keep hash or MD5 checksums of excluded or safe listed programs. When upgrading Avamar software, the hash or MD5 checksum must be recalculated per software vendor requirements.
Symantec EndPoint Protection
- Broadcom Backup Exec or Backup Exec System Recovery: Backup and Restore job rates are slower than expected with Symantec EndPoint Protection 12.1 installed (although the article mentions Symantec's own backup software, the issue is the same for any backup solution.)
- Broadcom EndPoint Protection exclusions for Backup Exec
Trend Micro Office scan
- Dell article How to check if TrendMicro anti-virus real-time scanning is impacting Avamar client performance (log in to Dell Support is required to view this article)
- Trend Recommended scan exclusion list for Trend Micro EndPoint products
- Trend Excluding Volume Shadow copies from the Real-Time Scan of APEX One
Sophos
- Sophos Process exclusion (Windows)
- Dell article Poor backup performance for Avamar client running Sophos anti-virus (deprecated)
- Sophos Central Server: Set scanning exclusion for Volume Shadow Copies
Quick Heal Endpoint Security
ESET Endpoint anti-virus
- ESET Endpoint 7 Processes exclusions
Panda Security
CrowdStrike
- CrowdStrike Whitelisting
Microsoft
- Microsoft anti-virus policy for EndPoint security in Intune
- Microsoft Defender for EndPoint
- Microsoft How to add a file type or process exclusion to Windows Security
Carbon Black
Fortinet
Trellix (FireEye)
Cisco AMP
Rapid7