Avamar: SQL multistreaming backup and restore may fail with "The api was waiting and the timeout interval had elapsed"
Summary: SQL multistreaming backup and restore may fail with "The api was waiting and the timeout interval had elapsed" affecting SQL 2017 and 2019 on large Databases.
Symptoms
SQL multistreaming backup and restore may fail with:
2021-09-16 15:01:54 avsql Error <6478>: VDS::Getconfig failed with 'The api was waiting and the timeout interval had elapsed' 2021-09-16 15:01:54 avsql Error <6478>: VDS::Getconfig failed with 'The api was waiting and the timeout interval had elapsed' 2021-09-16 15:01:54 avsql Error <6478>: VDS::Getconfig failed with 'The api was waiting and the timeout interval had elapsed' 2021-09-16 15:01:54 avsql Error <6479>: Timed out. Was Microsoft SQLServer running?
All spawned avtar threads for the "avsql stack execution" would be showing no progress bytes.
This issue happens when six or more streams are set for large databases of around one or more Tera Bytes in size.
This issue affects Microsoft SQL 2017 and 2019 environments.
Running the same backup in a single stream would complete but it would take a long time.
Cause
It was found that for the Microsoft SQL versions 2017 and 2019, the VDI behavior differs from the previous Microsoft SQL version 2016.
Essentially for SQL 2016 the parameter "SetEvent" is used when the backup configuration is completed, while for newer SQL versions a "semaphore count" is used instead.
This Microsoft VDI change affected the Avamar "avsql" component when multi-streaming was enabled for Avamar backups and restores, causing the failure described above.
To resolve this issue, the Avamar engineering team changed the SQL plug-in code in "avsql" and addressed the deadlock scenario.
Resolution
To resolve this issue, upgrade the backup client and SQL plug-in to the version 19.8 or above where the code fix is already applied.
Additionally, it is also recommended to apply the latest SQL plugin cumulative HF (based on versions) which includes other code fixes:
-
- For 19.8 => HF 338744
- For 19.9 => HF 338778
To download the hotfix, see the section "Looking for a Hotfix with its number" in the article Avamar: How to find and download a product hotfix, patch, install or upgrade package from the Dell Support website