Avamar: VSS Plugin Backups running slow troubleshooting guidelines

VSS plug-in backup running slow or both VSS plug-in backups and taking a snapshot of these critical volumes using Diskshadow was showing poor performance. 

In the first scenario, poor VSS Plugin backup performance was traced to the contents of subfolders in the critical C:\ drive, impacting overall backup efficiency.

Case #1:  Following folder content for Windows update history was slowing down Avamar VSS operations.

c:\Windows\servicing\LCU\Package_for_RollupFix~31bf3856ad364e35~amd64~~20348.3328.1.7
c:\Windows\servicing\LCU\Package_for_RollupFix~31bf3856ad364e35~amd64~~20348.3207.1.6

Use Deployment Image Servicing and Management (DISM) and the Disk Cleanup utility to initiate folder cleanup as per guidelines.

https://learn.microsoft.com/en-us/answers/questions/2191628/winsxs-occupying-more-space [learn.microsoft.com (External Link) 
 

After folder cleanup and exclusion, VSS backups that previously took nearly 2 hours 30 minutes now complete in 35-40 minutes.

Case #2: Folder content of "c:\Windows\System32\spool\PRINTERS\*" was causing VSS backups to run slow. 

The solution was to add a dataset exclusion for these folder entries: 

--exclude=c:\Windows\System32\spool\PRINTERS\
--exclude=c:\Windows\System32\spool\PRINTERS*.tmp

Case #3: Folder content of "C:\users*\appdata\*" is slowing down Avamar VSS backups. 

Exclude "AppData\Local", "AppData\LocalLow" and " AppData\Roaming" directories from VSS backups. 

Exclusion of the above folders does not impact the integrity of VSS backups as per Microsoft Guidelines at:

https://learn.microsoft.com/en-us/windows/apps/design/app-settings/store-and-retrieve-app-data  (External Link) 

Case #4: In this case, using either Avamar VSS backups or Diskshadow utility to snapshot the critical volumes was slow. To isolate the root cause, enable VSS performance trace with the following commands (use DOS admin Command Prompt):

i) logman create trace vss_trace -ow -o %temp%\%computername%_vss_trace.etl -p {9138500E-3648-4EDB-AA4C-859E9F7B7C38} 0xffffffffffffffff 0xff -nb 16 16 -bs 1024 -mode Circular -f bincirc -max 4096 –ets
ii) logman create counter PerfLog-1s -o "%temp%\%computername%_PerfLog-1.blg" -f bincirc -v mmddhhmm -max 300 -c "\LogicalDisk(*)\*" "\Memory\*" "\.NET CLR Memory(*)\*" "\Cache\*" "\Network Interface(*)\*" "\Netlogon(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Processor Information(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues(*)\*" "\Terminal Services\*" -si 00:00:01
iii) logman start Perflog-1s

iv) Now START THE VSS backup USING DISKSHADOW or Avamar VSS Plugin

v) stop the trace once backup completed with the following elevated commands:
 
logman stop vss_trace -ets
logman stop Perflog-1s
logman delete Perflog-1s

Trace logs captured indicate continuous execution on the "GetRootAndLogicalPrefixPaths" function almost in a loop.

Also in the trace, it shows a driver verifier that has been enabled on all drivers being the culprit and slowing down the operations. More details about the impact of driver verifier at:
https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/driver-verifier (External Link)

Case #1: Root cause identified due to tiny Windows update files in c:\Windows\servicing\LCU folder.

c:\Windows\servicing\LCU\Package_for_RollupFix~31bf3856ad364e35~amd64~~20348.3328.1.7
c:\Windows\servicing\LCU\Package_for_RollupFix~31bf3856ad364e35~amd64~~20348.3207.1.6

Resolution is to initiate a DISM/"disk cleanup utility to remove redundant entries and if that does not help. 
Add explicit exclusion of these subfolders in the Dataset.

Case #2: Root cause identified due to .tmp files in "c:\Windows\System32\spool\PRINTERS*.tmp"
The resolution is to exclude these files from the VSS backup by adding an exclusion in the Dataset:

--exclude=c:\Windows\System32\spool\PRINTERS*.tmp

Case #3: Root cause identified due to three subfolder entries in "C:\users*\appdata\*." The files in the subfolders store user-specific application data that is not deemed necessary for Bare Metal recovery (BMR) recovery. To resolve this issue, add an exclusion in the Dataset at the global level for this folder and the three subfolders:

"C:\users*\appdata\*"

Case #4: The resolution was to disable driver verifier execution and VSS backups completed in 15 minutes instead of taking more than 10 hrs. Also the Diskshadow snapshot took only a few secs instead of more than 25 minutes before the change.