Avamar: Windows server performance: Slow file system backup on servers with PGP drives

Avamar backup performance is slow.

Example: 

Backed-up 6.659 MB in 105.79 minutes: 3.8 MB/hour (1 files/hour)

In the backup logs with debugging enabled, the backup starts and finishes quickly as shown below.

The beginning of the backup job starts as follows:

2017-05-25 04:33:52 avtar Info : Logging to C:\Program Files\avs\var\Test05232017.log xxxxxxxxxxxxxxxxx

The backup job is done quickly as shown here:

2017/05/25-02:34:17.84500 [avtar] creating .system_info directory
2017-05-25 04:34:17 avtar Info : Backup complete, wrapping-up session with Server

The delay was in creating the user profiles ".system_info/userinfo.xml" as it starts reading the profiles:

2017-05-25 04:34:18 avtar Info : Reading 37 user profiles

More than two hours later, the enumeration of the user profiles completed:

2017-05-25 06:50:34 avtar Info : Done reading user profiles

The backup completed half a second later.

2017-05-25 06:50:35 avtar Info : Backup #10 timestamp 2017-05-25 06:50:34, 3 files, 3 folders, 4.578 KB (3 files, 1.576 KB, 34.43% new)
2017/05/25-04:50:35.04299 [avtar] backup expire prectime=131431854349654640
2017-05-25 06:50:35 avtar Info : Label "test w/new flags", scheduled to expire after 06/29/17 (2017-06-29 04:50:34 UTC)
2017-05-25 06:50:35 avtar Info : Backed-up 4.578 KB in 136.72 minutes: 2.0 KB/hour (1 files/hour)

The issue was isolated to some SIDs like SID "S-1-5-21-1761170581-294597109-3035037286-64925452" that had the corresponding username entry missing. The Avtar process was stuck processing these SIDs, causing this delay.

As a workaround, create an avtar.cmd file in the C:\Program Files\var directory or whatever folder in which the Avamar Windows client software is installed. Then add the following flag and save the file:

--x05=65536

This flag disables user profile gathering during the backup process. This flag has a significant performance impact. Long term, check the Active Directory domain trusts in the environment and clean domain accounts that have inconsistent SID entries. To do that, get more detail about these SIDs by using the procedure below.

1. Open Registry Editor by typing the following command in the Windows Search bar:

regedit

2. Go to the path below:

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList

All the SIDs are listed here, and the "ProfileImagePath" key is about the name of each SID.
3. Use the third-party application "PsGetSid.exe" to list the name in the command line.

Example:

PsGetsid S-1-5-21-854245398-1973579041-362214817-1745107

PsGetSid.exe is available from: https://learn.microsoft.com/en-us/sysinternals/downloads/psgetsid (External Link)

Once the problematic user accounts are removed, delete them from the registry using regedit.exe. The avtar flag is no longer needed once the cleanup is done.

--x05=65536