Avamar: SQL Backup fails due to timeout from high resource utilization

• SQL backups fail or terminate abnormally in the Avamar Administrator UI or Avamar User Interface (AUI) Activity Monitor.
• SQL backup jobs appear to complete data transfer but fail during SQL processing.
• SQL Client logs report I/O and timeout-related errors.
• CPU and RAM usage on the SQL Client remains consistently high during the backup window.

The following errors appear in the SQL backup log (on the SQL Client):

C:\Program Files\avs\var\SxOMxxxxx_SQL1_hpxxn_uxxmdb_test_SQL_Schedule-test_SQL_Backups-1722556800070-3006-SQL.log

2024-08-02 08:20:38 avsql Info <6688>: Process 17036 (C:\Program Files\avs\bin\avtar) finished (code 0: success)
2024-08-02 08:20:38 avsql Info <6446>: SQL VDI transfer of 'SxOMxxxxx\SQL1/hpxxn_uxxmdb/f-0.hpxxn_uxxmdb' to 'SxOMxxxxx_SQL1_hpxxn_uxxmdb_test_SQL_Schedule-test_SQL_Backups-1722556800070-3006-SQL' successful
2024-08-02 08:20:38 avsql Error <40258>: sqlconnectimpl_smo::execute Microsoft.SqlServer.Management.Common.ExecutionFailureException: An exception occurred while executing a Transact-SQL statement or batch. ---> System.Data.SqlClient.SqlException: Write on "SxOMxxxxx_SQL1_hpxxn_uxxmdb_test_SQL_Schedule-test_SQL_Backups-1722556800070-3006-SQL" failed: 995(The I/O operation has been aborted because of either a thread exit or an application request.)
 
A nonrecoverable I/O error occurred on file "SxOMxxxxx_SQL1_hpxxn_uxxmdb_test_SQL_Schedule-test_SQL_Backups-1722556800070-3006-SQL:" 995(The I/O operation has been aborted because of either a thread exit or an application request.).
 
BACKUP DATABASE is terminating abnormally.

Upon reviewing the performance metrics in the Task Manager for the SQL Client, it is clear that there is significant CPU utilization:

Figure 1. Task Manager CPU Utilization

While examining the processes that consume a significant amount of CPU and RAM, it was identified that there were other processes using the maximum available resources:

  Figure 2. Task Manager Processes Running

The issue occurs when other server processes, such as SQL Server Windows NT services or third-party agents, consume most of the available CPU and RAM.

High resource utilization prevents Avamar backup processes from maintaining required I/O operations, which causes SQL backup timeouts and failures.

Engage the SQL/Wintel administration team to mitigate the usage of CPU and RAM by SQL Server and other Agent processes.

• There is Insufficient System Memory in Resource Pool 'Internal' to Run this Query (External Link - microsoft.com)
• There is insufficient system memory in resource pool 'default' to run this query. on sql (External Link - stackoverflow.com)
• Server memory configuration option (External Link - microsoft.com)