Avamar SQL Plugin support for multiple listeners in the same AlwaysOn cluster configuration

Customer configured multiple listeners as per the guidelines at:

1. How to create multiple listeners for the same availability group https://techcommunity.microsoft.com/ (External Link)
2. Create additional listeners for an availability group https://techcommunity.microsoft.com/ (External Link)

Avamar scheduled backups are targeted to run on the secondary replica.
The problem is that different listeners are scheduled to run their backups simultaneously.
For example:

Work Order ID : : 1724036400012 for listener#1 Plugin ID 1724036400012-3006 which is SQL has start time “23:00:11”
Work Order ID : : 1723950000025 for listener #2 Plugin ID 1723950000025-3001 which is Windows has start time “23:01:44”

As a result of schedule backup running simultaneously there is an overlap on backups, which is not a supported Avamar configuration.

And the end result is that a backup that started @ 23:00:11 (11pm) begins (Work Order ID: : 1724036400012) but then there is no data processed for a couple of hrs and fails as:

2024-08-18 23:01:22 avsql Info <18513>: Waiting for secondary replica to start backup sub-workorder ...
2024/08/19-03:01:22.88299 [avsql_assist]  ===> avsql_iac_manager::wait_for_wakeup
2024/08/19-03:06:22.88599 [avsql_assist]  Waiting duration: 300.0 seconds
2024/08/19-03:06:22.88599 [avsql_assist]  <=== avsql_iac_manager::wait_for_wakeup
2024-08-18 23:06:22 avsql Error <18517>: Timeout occurred after 5 min.
 

The other backup that started @ 23:01:44 with a different work order ID 1723950000025 starts before work order ID 1724036400012 is completed. This backup fails because two backups cannot run simultaneously on the same node (client). This is shown below, and backup fails as "Unable to start backup with job key 'SQL_Transaction_Log-1723950000025' because it has already started."

2024-08-17 22:01:16 avagent Info <5964>: Requesting work from 10.111.60.16
2024-08-17 22:01:16 avagent Info <5264>: Workorder received: sleep
2024-08-17 22:01:16 avagent Info <5996>: Sleeping 3612 seconds
2024-08-17 23:01:26 avagent Info <43214>: udpsockhandle::getipv6andipv4 get ip first, outlen 1840
2024-08-17 23:01:26 avagent Info <43215>: udpsockhandle::getipv6andipv4 get ip first after, outlen 6152, ret 111
2024-08-17 23:01:26 avagent Info <6624>: Attempted connection from non-localhost IP address - xx.xxx.xx.xx:59604 != '127.0.0.1'
2024-08-17 23:01:26 avagent Info <10684>: Setting ctl message version to 3 (from 1)
2024-08-17 23:01:26 avagent Info <16136>: Setting ctl max message size to 268435456
2024-08-17 23:01:26 avagent Info <18950>: Starting workorder
2024-08-17 23:01:26 avagent Info <6686>: Process 2896 (C:\Program Files\avs\bin\avsql) for workorder SQL_Transaction_Log-1723950000025 started
2024-08-17 23:01:26 avagent Info <10684>: Setting ctl message version to 3 (from 1)

	xxxxxxxxxxx

2024-08-17 23:01:30 avagent Info <10684>: Setting ctl message version to 3 (from 1)
2024-08-17 23:01:30 avagent Info <16136>: Setting ctl max message size to 268435456
2024-08-17 23:01:32 avagent Warning <14844>: CTL: Unable to start backup with job key 'SQL_Transaction_Log-1723950000025' because it has already started.
2024-08-17 23:01:32 avagent Warning <15188>: failed to process message type 11 snapup
 

The above setup does not work as MCS/client Avagent does not allow two policies running concurrently on the same client simultaneously.

Proposed solution

1. Schedule the policy to run at different times and avoid conflict
2. Create multiple Avagents with their own var folders and can then run parallel policy runs with each agent handling individual policies independently