Sharepoint Services Writer keeps failing during backup

Question

I have the Avamar 6.0 client installed on all three servers (see below) and the VSS 6.0 client installed on both the front-end and the back-end servers. Everything seems to be working, but anytime I start a backup (scheduled or on-demand), it fails. The error in the log is as follows:

2012-06-27 14:58:45 avmossvss Info <0000>: Completing VSS backup...

2012-06-27 14:58:45 avmossvss Info <8389>: Calling BackupComplete and retrieving BackupComponentsDocument.xml and writers metadata files...

2012-06-27 14:59:03 avmossvss Error <0000>: ERROR: Selected writer 'SharePoint Services Writer' is in failed state!

- Status: 12 (VSS_WS_FAILED_AT_BACKUP_COMPLETE)

- Writer Failure code: 0x00000000 (S_OK)

- Writer ID: {da452614-4858-5e53-a512-38aab25c61ad}

- Instance ID: {b1fea9ab-efe0-4c45-ae8d-cbb4ac14273c}

2012-06-27 14:59:03 avmossvss Error <0000>: VSS exception code 0x8000ffff has been thrown by BackupComplete() - 'code 0x8000ffff: Catastrophic failure'

2012-06-27 14:59:03 avmossvss Error <0000>: Error occurred during completing VSS backup!

Other errors I have found:

Avagent.log file:

Error <6677>: Error processing CTL message

Warning <6690>: CTL workorder non-zero exit status 'code 536870919: Miscellaneous error'

Restarting the services will get the VSS Writer back into a "Stable" state, but it always seems to fail between 33.3GB and 33.7GB backed up. When I do a vssadmin list writers on the WFE server, it should "State: [12] Failed." I've made permission changes in the registry to the HKLM\System\CurrentControlSet\Services\VSS\Diag and VssAccessControl keys. I've made access permissions changes to dcomcnfg. I have tried various Avamar hotfixes (I can dig up the nubmers if needed). I have done research on the various codes, but found nothing relevant. All other client and VMDK backups work just fine.

Any help anyone is able to provide would be greatly appreciated.

Chris

Servers: Windows 2008 R2

Setup: Web front-end server, Application back-end server, SQL back-end server

Sharepoint version: 2010

Avamar Version: 6.0.101.66

ionthegeek · Answer

Is there anything interesting reported about VolSnap in the Windows Event Logs? That may give some hints about the nature of the VSS failure.

In a previous case where I saw the "Catastrophic failure" error code, the back-end disk was a heavily loaded SAN volume and the failure was being caused by disk writes timing out after 5 minutes in the VSS writer itself. That's not a very common scenario but it couldn't hurt to check the disk performance stats.

Chris_J1 · Answer

Thanks for the response, Ian. Sorry it took so long to respond back. AC in the server room decided to take a break. I checked the event log, but saw no VolSnap entries in either Application log or the System log. I get the error regardless of load or time of day, so I don't think it's being caused by heavy load.

Let me know if you can think of anything else though. Thanks again.

ionthegeek · Answer

Could you try running a SharePoint backup using the native SharePoint backup tools? Does this backup complete successfully or does it fail as well? Knowing that will help narrow things down.

My pleasure!

bairdc03 · Answer

Did anyone find a solution to this problem?

Avamar

Sharepoint Services Writer keeps failing during backup

Was this post helpful?