Avamar System State Backups and Open Files

Generally speaking, you should not need to exclude files that are reporting open file errors.

The most common cause of this type of error is a failure while creating a VSS snapshot. There should be warning or error messages in the Avamar logs that show the reason for the failure (or at least point in the right direction). These will be near the top of the log where the Avamar client requested the creation of a VSS snapshot.

Can you post log snippets showing any warnings and errors that appear in the logs around the time of VSS snapshot creation, along with a few lines of context above and below? I would recommend scrubbing the log snips of any identifying information like hostnames, IP addresses or filenames before posting them.

Sure thing:

2015-01-23 03:01:05 avtar Info <5550>: Successfully logged into Avamar Server [7.0.2-47] (Compression enabled)

2015-01-23 03:01:05 avtar Info <7563>: Back up of "c:" on server "server_test" for /clients/server_test.mydomain.org

2015-01-23 03:01:05 avtar Info <5586>: Loading cache files from C:\Program Files\avs\var

2015-01-23 03:01:05 avtar Info <8650>: Opening cache file C:\Program Files\avs\var\f_cache.dat

2015-01-23 03:01:05 avtar Info <5573>: - Loaded cache file (11,534,880 bytes)

2015-01-23 03:01:05 avtar Info <8650>: Opening cache file C:\Program Files\avs\var\p_cache.dat

2015-01-23 03:01:05 avtar Info <5573>: - Loaded cache file (12,583,456 bytes)

2015-01-23 03:01:05 avtar Info <6426>: Done loading cache files

2015-01-23 03:01:05 avtar Info <16281>: Not traversing 'c:\Documents and Settings' since it's of type 'REPARSE_SYSTEM_DIR_JUNCTION'

2015-01-23 03:01:07 avtar Info <16281>: Not traversing 'c:\ProgramData\Application Data' since it's of type 'REPARSE_SYSTEM_DIR_JUNCTION'

2015-01-23 03:01:07 avtar Info <16281>: Not traversing 'c:\ProgramData\Desktop' since it's of type 'REPARSE_SYSTEM_DIR_JUNCTION'

2015-01-23 03:01:07 avtar Info <16281>: Not traversing 'c:\ProgramData\Documents' since it's of type 'REPARSE_SYSTEM_DIR_JUNCTION'

2015-01-23 03:01:07 avtar Info <16281>: Not traversing 'c:\ProgramData\Start Menu' since it's of type 'REPARSE_SYSTEM_DIR_JUNCTION'

2015-01-23 03:01:07 avtar Info <16281>: Not traversing 'c:\ProgramData\Templates' since it's of type 'REPARSE_SYSTEM_DIR_JUNCTION'

2015-01-23 03:01:07 avtar Warning <16218>: Couldn't open 'c:\ProgramData\Symantec\Symantec Endpoint Protection\CurrentVersion' - Skipping

2015-01-23 03:01:18 avtar Error <5139>: File access share error "c:\ProgramData\Symantec\Symantec Endpoint Protection\CurrentVersion\Data\BASH\BASHV4.DB" (code 32: The process cannot access the file because it is being used by another process). (Log #1)
2015-01-23 03:01:18 avtar Error <5139>: File access share error "c:\ProgramData\Symantec\Symantec Endpoint Protection\CurrentVersion\Data\CmnClnt\SBSDKGbl.dat.log" (code 32: The process cannot access the file because it is being used by another process). (Log #1)
2015-01-23 03:01:18 avtar Error <5139>: File access share error "c:\ProgramData\Symantec\Symantec Endpoint Protection\CurrentVersion\Data\CmnClnt\_lck\_RDRPluginG" (code 32: The process cannot access the file because it is being used by another process). (Log #1)
2015-01-23 03:01:18 avtar Error <5139>: File access share error "c:\ProgramData\Symantec\Symantec Endpoint Protection\CurrentVersion\Data\CmnClnt\_lck\_SNDPluginG" (code 32: The process cannot access the file because it is being used by another process). (Log #1)

The errors are all for files locked by our antivirus application.  This is a non-issue on for the Windows Filesystem backups, because the exclude list on the parent directory C:\ProgramData\Symantec works with no issues.

However, I see plenty of "system excludes" further up in the logs:

015-01-23 03:01:05 avtar Info <8940>: Starting back up at 2015-01-23 03:01:05 Eastern Standard Time as "NT AUTHORITY\SYSTEM" on "server_test" (24 CPUs) [7.0.102-47]

2015-01-23 03:01:05 avtar Info <15223>: Volume "\\?\Volume{aee4445abbd-3f96-11e4-80b5-806e6f6e666963}\" is not acessible via a path name, so Avtar will not get a list of its junctions and symlinks.

2015-01-23 03:01:05 avtar Info <5730>: Entering include/exclude rules.

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "c:\Windows\ntfrs\jet"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "C:\Windows\debug\NtFrs\NtFrs*"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "c:\Windows\SYSVOL\domain\DO_NOT_REMOVE_NtFrs_PreInstall_Directory"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "c:\windows\sysvol\domain\NtFrs_PreExisting___See_EventLog"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "c:\Windows\SYSVOL\staging\domain\NTFRS_*"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "C:\Windows\System32\dns\backup\dns.log"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "C:\Windows\System32\dns\dns.log"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "System Volume Information\DFSR"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "System Volume Information\EfaData"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "C:\Windows\System32\Bits.bak"

2015-01-23 03:01:05 avtar Info <7838>: - Adding system exclude "C:\Windows\System32\Bits.log"

Thank you for jumping in, Chetan. Unfortunately, if the software includes a filesystem filter driver, stopping the service is not sufficient to rule it out as the culprit. Filter drivers may still intercept filesystem calls even if the associated service is stopped or disabled.

Hi,

I am Chetan Savade from Symantec Technical Support Team.

To confirm Symantec Endpoint Protection (SEP) is not causing any issue, disable SEP service and verify the performance. 

To disable SEP service, go to

Start --> Run --> (Type) smc -stop , This command will stop SEP service.

Start --> Run --> (Type) smc -start , This command will start SEP service

SEP 12.1 RU5 is the latest version of SEP, would suggest to upgrade SEP client if it's on old version.

Best Regards,

Chetan

Hi Chetan -

We're on 12.1.4112.4156, which I don't know how to translate into which "RU" this version is.  SEP updates are handled by another team, so I will ask them if we're on the latest version.

SEP isn't causing an issue, per se, it's just that SEP has these files locked during a VSS System State backup, generating the exceptions you see above.  I will investigate some other options with this weekend's backups, and see if we have better luck.

Thanks!

One suggestion I saw on the Symantec forums while I was researching this issue was to disable tamper protection as that might be what's preventing access to the files. That might be worth a try.

Thanks, Ian - I'm sending that to the security team now.  I'll followup with their response as soon as I can.

You are on SEP 12.1 RU4 MP1B ( 12.1.4112.4156) version, SEP 12.1 RU5  (12.1.5000.5337) is the latest version.

See this article to have some more idea about SEP release details:https://www-secure.symantec.com/connect/blogs/symantec-endpoint-protection-release-details

One known issue has been fixed in the latest release, see if it's related:

Windows Backup of Hyper-V virtual machine fails

Fix ID: 3234743

Symptom: Windows Backup of a virtual machine fails on a Hyper-V host. The backup fails with the following error in event viewer Application logs:

Event ID 517: "The backup operation that started has failed with following error code '0x80070020' (The process cannot access the file because it is being used by another process.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved."

Solution: Modified the SymEFA.sys driver to handle a scenario where the Microsoft API CreateFile() could fail during backup.

Reference: http://www.symantec.com/business/support/index?page=content&id=TECH224706

Note: To complete SEP client upgrade reboot it necessary.

Thanks.  Our security team says we will upgrade to RU5 during next month's maintenance cycle (this weekend). I'll report back if that fixes the issue.

In case anyone finds this, EMC posted a KB about this exact issue, which resolved my problem:

File access share errors for Symantec Endpoint files during VSS backup