Avamar: Backup completes with exceptions due to avtar errors 5137, 5244 and 12516

Scheduled backups complete with exceptions and the end of the log reports: 

avtar Info <5314>: Command failed (1299 errors, exit code 10020: completed with errors, client log should be examined)

Note: the number of errors reported will vary.

In examining the log, a sequence of errors similar to the following are reported:
 

avtar Error <5137>: Unable to open "E:\Users\UserA\folder1\subfolderA\sub-subfolderA\FEBRUARY 2014\Thumbs.db" for backup (code 2: The system cannot find the file specified).
avtar Error <5244>: Unable to open directory "E:\Users\UserA\folder1\subfolderA\sub-subfolderA\FEBRUARY 2015" (code 2: The system cannot find the file specified).
avtar Error <5244>: Unable to open directory "E:\Users\UserA\folder1\subfolderA\sub-subfolderA\VERSION 2" (code 2: The system cannot find the file specified).
avtar Error <5244>: Unable to open directory "E:\Users\UserA\folder1\subfolderA\sub-subfolderB" (code 2: The system cannot find the file specified).
avtar Error <5244>: Unable to open directory "E:\Users\UserA\folder1\subfolderB" (code 2: The system cannot find the file specified).
avtar Error <5244>: Unable to open directory "E:\Users\UserA\folder2" (code 2: The system cannot find the file specified).
avtar Error <5244>: Unable to open directory "E:\Users\UserB" (code 2: The system cannot find the file specified).
avtar Error <5244>: Unable to open directory "E:\Users\UserC" (code 2: The system cannot find the file specified).

Notice that the directory tree is progressively getting shorter as avtar moves up the tree.

These errors may then be followed by:

avtar Error <12516>: Target of "E:\Users\UserD" not found. Verify that target exists on the system
avtar Error <5244>: Unable to open directory "E:\Users\UserD" (code 3: The system cannot find the path specified).
avtar Error <12516>: Target of "E:\Users\UserE" not found. Verify that target exists on the system
avtar Error <5244>: Unable to open directory "E:\Users\UserE" (code 3: The system cannot find the path specified).


Avtar Error <12516> is usually caused by a broken Junction (Reparse Point) on the machine, but in this case it is a false lead.

On the client, open a command prompt and navigate to the directory one level above the directory referenced.  In this example "E:\Users".

Issue the command dir /a

If "UserD" and "UserE" were Junctions, they would show as:
 

<JUNCTION>     UserD [TargetFolder]
<JUNCTION>     UserE [TargetFolder]

In this case, they show as:
 

<DIR>     UserD
<DIR>     UserE


This indicates that they are not Junctions, but rather normal Directories.

The fact that the errors show that avtar is walking backwards up the file tree is the key here.  This is indicating that avtar has the tree structure in memory but when it tries to access the files, they are no longer there.  This is not indicative of a file deletion, as that would be accounted for by the VSS snapshot.  Rather this is caused by the VSS snapshot itself being removed during the backup.

Make note of the timestamp of the very first error.  In our example it is 2016-02-23 19:00:27
 

2016-02-23 19:00:27 avtar Error <5137>: Unable to open "E:\Users\UserA\folder1\subfolderA\sub-subfolderA\FEBRUARY 2014\Thumbs.db" for backup (code 2: The system cannot find the file specified).

The backup showed these errors at the same time every night.  On-demand backups run at a different time completed normally.

We need to examine the System Log in Windows Event Viewer for events at or near the same time.

In our example we find that at the same time that we start seeing avtar errors we get 2 events:
 

Date:          2/23/2016 7:00:27 PM
Event ID:      7045
Description: A service was installed in the system.
Service Name:  BeVssProvider
Service File Name:  "C:\Program Files\Symantec\Backup Exec\RAWS\VSS Provider\bevssprovider.exe"
Service Type:  user mode service
Service Start Type:  demand start
Service Account:  LocalSystem


Date:          2/23/2016 7:00:27 PM
Event ID:      7036
Description: The Volume Shadow Copy service entered the stopped state.