Avamar

Question

We are utilizing Avamar to backup Windows 2008 and 2012 servers in our environment, backing them up with the Windows File System and Windows VSS plugins. We frequently find exceptions on the file system backup for file locks on %windir%\system32\, NTUSER.dat files and other similar objects. It seems like we don't need to worry about backing some of these up with the file system plugin since we are also using the VSS plugin on these same servers. Am I correct in that assumption? If so, how can I determine what to safely exclude from the file system backups? Our dataset is only configured to exclude very few things from that, such as the Recycle Bin.

ionthegeek · Accepted Answer

VSS snapshots are all or nothing -- either the whole snap succeeds or the whole snap fails.

I think you're onto something with the D:\ volume. If this volume is not really a disk, we won't be able to take a snapshot of it (hence the VSS_E_OBJECT_NOT_FOUND message -- we're trying to snap something that isn't really there). It sounds like some kind of an edge case that was not considered. It makes sense that the VSS System State plug-in wouldn't be affected because the D:\ volume wouldn't be considered a critical volume so it wouldn't be included in the VSS snapshot for system state backups.

I'd recommend opening a service request for this issue. It sounds like there may be a problem in the Avamar filesystem client code that's causing it to add this volume to the snapshot list in error.

In the meantime, you can exclude the volume from VSS snapshots in the dataset advanced options (see Re: Re: Avamar and Windows mount points). Be sure to change the plug-in dropdown on the Options page to "Windows File System" before you try to make the change.

ionthegeek · Answer

If exceptions are being reported for system files, it sounds like the VSS snapshot for the filesystem backup may be failing. I would recommend investigating and resolving that issue before you consider excluding anything.

mmusser · Answer

Hi Ian,

Thanks for your reply. Having checked the Application events, I did not see any VSS errors being reported. I *DID* see events indicating that VSS was shutting down due to its idle timeout, which I did not think anything of. Then I noticed the Avamar logs for the Windows file system plugin showed the following:

2014-08-04 00:00:26 avtar Warning <6698>: VSS exception code 0x80042308 thrown freezing volumes - 'code 0x80042308: No error description string'

2014-08-04 00:00:26 avtar Info <10662>: Unable to create volume snapshot; backup proceeding with original (non-snapshot) volume content. Some file content may not be backed up.

This seems to support your suggestion that there are problems with the snapshot, despite Windows not logging any such problems in Application events. When I checked the Windows Shadow Copy service this afternoon, I saw that it was stopped. I manually started the service and ran only the file system backup - we run both the file system and VSS backup as part of the same policy and dataset; one runs after the other - and that completed without any exceptions. The Avamar log confirmed successful snapshot creation:

2014-08-05 12:08:00 avtar Info <8776>: Creating snapshot set...

2014-08-05 12:08:00 avtar Info <8780>: Creating the shadow (DoSnapshotSet) ...

2014-08-05 12:08:02 avtar Info <8781>: Shadow copy set successfully created.

So this leads me to wonder if the shadow copy service idling out is the problem. I can't definitely state that for sure, because Windows is not logging when the service starts; only when it stops due to timeout. Therefore, I cannot establish a timeline of when the service runs and correlate that to backups. There is a means to increase the timeout, but without knowing if that's the issue I'd rather not mess with the registry.

Have you observed this before? Is there a problem with running both the VSS and the file system plugins in the same dataset and group/policy?

Thanks.

ionthegeek · Answer

I would recommend examining the System event logs as well -- VSS can log events in both the System and Application event logs. Look for both "VSS" and "VolSnap" events. The VSS service is started on-demand so the behaviour you are seeing with the idle timeout is normal. I wouldn't recommend changing this timeout without running it by Microsoft first.

Code 0x80042308 maps to "VSS_E_OBJECT_NOT_FOUND" so it's strange that some backups would work and others not. What happens if you run "vssadmin list writers" around the time of the snapshot failure? Does this return correctly?

I haven't seen issues with combining multiple plug-ins in the same dataset for a very long time (not since the Avamar 4.x era). As far as I know, there shouldn't be any issues with running both plug-ins in the same dataset. That said, you could always try separating them temporarily to see if that helps.

mmusser · Answer

Thanks for the tip. The System events do not show any VSS errors. Strangely, this error is only being reported in the Avamar log.

The System events do show the start and stop times of the VSS service, though. This allows me to establish a timeline for the most recent file system backup (earlier today) where we had exceptions:

00:00:04 - VSS plugin backup starts

00:00:07 - Volume Shadow Copy service enters running state

00:00:36 - Shadow Copy Provider service enters running state

00:09:45 - VSS plugin backup completes

00:10:04 - file system plugin backup starts

00:10:06 - Avamar logs VSS exception code 0x80042308 for D:\ volume

00:13:06 - VSS idle timeout, the Volume Shadow Copy Service enters stopped state

00:16:06 - Shadow Copy Provider service enters stopped state

00:44:12 - file system plugin backup completes

Quick observation here: the VSS error is actually for the D:\ volume, not the C:\ volume (where we see the file lock exceptions). Does a snapshot failure for one volume result in using the non-snapshot/original volumes for all drives on the server? If yes, will adding an exclusion for D:\ within avtar.cmd circumvent this? That drive is not a native fixed disk; it's actually a folder within C:\ that's been mounted as its own volume, so there's no need to try backing it up separately (and create snapshots for it).

ionthegeek · Answer

I would recommend keeping the two issues separate. It sounds like the issues have similar symptoms but unrelated causes. I assume you want to back up the encrypted volumes? My recommendation would be to open up two service requests (one for each issue).

My pleasure! Looking forward to your update.

mmusser · Answer

That makes sense. During my earlier on-demand file system backup of this server, I specifically backed up only the C:\ drive, and not the other two (D: and I:). This attempt completed without exceptions.

I will try adding a volume exclusion to the dataset for this server as your instructions suggest. I will also be happy to have a service request opened up for this.

We have similar issues with another server on which there are encrypted volumes - this causes VSS errors for that drive and lots of locked file exceptions during backup. Should this issue be included in the service request, or is that already a known/documented issue?

Thanks a ton for your help. I'll update this thread tomorrow after the next scheduled backup for this server on the tweaked dataset.

mmusser · Answer

Good morning!

Today's scheduled backup of the server in question worked without any exceptions. From the logs, it looks like the freeze excludes took for those other two volumes while the snapshot for C:\ was successful. Those other volumes were still backed up in the file system backup - I'm assuming this means they were just backed up off the original/non-snapshot volume instead?

I'll work now on opening a service request with EMC. Thanks very much for your help!

ionthegeek · Answer

I'm glad to hear the workaround was effective.

I'd imagine if the volumes are pointing at locations on the C drive, the VSS snapshot for C would be used to back up open files. In addition to the VSS exclude, you could always exclude these volumes in the dataset if you know they are redundant data.

My pleasure!

Avamar - Windows - File-level exclusions and VSS

Was this post helpful?