SQL Backup Configured as NSRSQLSV suddenly failing - error attached

[zwoodcock]

Good day;

Beginning roughly two days ago, the backup for one of our SQL Servers started failing out of the blue - no configuration changes, no server-side changes, nothing.

The only errors I'm getting are:

1. Unsupported File System

          a. No change to the File System; don't know how this could be an issue

    2. Write-Protected Disc

          b. Nope; no changes made to disk permissions

    3. No space on Disc

          c. Plenty of space on all Drives on the server in question

    4. Drive not found

          d. No changes to Drive Naming, no drives added/removed.

I've been attempting to modify and troubleshoot the configuration for the past couple of days, but no dice. Any assistance is appreciated.

I have attached the full error text.

Thank you!

[zwoodcock]

Reply just to add - I already tried my generic Troubleshooting steps:

1. Stop Services on host, rename TMP folder, Start Services on host
2. Restart all NW Services on NW Server

Interestingly enough, the Filesystem backup for this exact same server is getting the same error message.

[ble1]

I see you use VSS for backup - did you try VDI to rule out any VSS related issue?

[wlee4]

If the file system backups are also failing, then this should be looked at first.

BTW... this backup is not using nsrsqlsv to backup SQL.  You are using NMM to backup SQL using the VSS backup method.

For now... I suggest focusing on the file system backup failure.  On this client:

1. what is the o/s version?
2. what is the n/w client version?
3. what is the NMM version?
4. What is the SQL version?
5. Is this a stand alone SQL server, or is this a cluster node?  If cluster, are the n/w and nmm versions the same on all nodes?
6. what errors are reported during file system backups?   Are all backups affected?  Does any saveset succeed?
7. Use NMM's config_checker.  are there any reported issues?

[zwoodcock]

Edit: I took a screenshot of the wrong error; the original screenshot included an error from when both the Filesystem and SQL Backups were occurring simultaneously.

Wallace,

First, that was my mistake. Through troubleshooting, I modified the configuration of the Client to match our Production Database Server's - which is using nsrsqlsv. You are correct in that I was using NMM when this error arose.

1. The O/S is Win Server 2012 Standard
2. The N/W Client version is not reporting; doesn't say (used print type: NSR Client;name:*client_name*)
3. The NMM Version is 2.4.0.375
4. The SQL Version is SQL Server 2012 SP1
5. Stand-alone
6. Filesystem Backup is reporting this error, and no savesets succeed.

         

    7. Config_Checker reports no issues.

[wlee4]

@zwoodcock...  Funny i thought you updated this discussion.  Based on what i thought I saw:

• NetWorker client: SQL 2012, Windows 2012, n/w client 8.0, NMM 2.4.0
• NetWorker server: Windows, n/w 8.0
• Error:  "There is another instance of NMM save program is already running"

This file system backup error only shows that NMM is already running. This is does does not itself show there is a technical problem, but a scheduling problem with the file system backups.  Do you see the same error all the time?  Is there a different error when backing up the file system?

You are using NMM 2.4, which is why NMM is used to backup the file system as well as SQL.  This was changed starting with NMM 3.0, where NMM was only used to backup the databases. The file system backups went back to using save.exe, just as if NMM was not installed and only NetWorker client was installed.   Note: NMM 2.4.x support was ended on March 31, 2016.  NMM 3 is still currently supported, and there are also more recent versions.

Since the issue only started 2+ days ago, the errors could be caused by a problem with Windows Volume Shadow Copy Services (VSS).  Please check the client's application and system even logs for events during the backup period.  Also run the following and look for any errors in the output.

1. vssadmin list writers
2. vssadmin list providers
3. vssadmin list shadows

If there is a problem with VSS, then try rebooting this SQL server, and check VSS again.

[zwoodcock]

Wallace

I noticed the NMM Save Already Running - that was a mistake screenshot. I edited the reply with the correct screenshot, but it's still being "Moderated" so I guess you can't see it:

1. vssadmin list writers shows all of the writers in a Stable state, with no error
2. vssadmin list providers shows Microsoft File Share Shadow Copy provider, and Microsoft Software Shadow Copy Provider 1.0
3. vssadmin list shadows shows "No items found that satisfy the query."

[wlee4]

please try

     vssadmin List Volumes

     vssadmin List ShadowStorage

On my desktop:

     vssadmin list ShadowStorage

Shadow Copy Storage association

   For volume: (C:)\\?\Volume{24c476ae-db08-11e5-93c5-806e6f6e5563}\

   Shadow Copy Storage volume: (C:)\\?\Volume{24c476ae-db08-11e5-93c5-806e6f6e5563}\

   Used Shadow Copy Storage space: 33.818 GB (3%)

   Allocated Shadow Copy Storage space: 34.883 GB (3%)

   Maximum Shadow Copy Storage space: 46.576 GB (5%)

So this shows that for my c:, it creates the shadow copy on c: and shows how much space it currently uses, and max it can use.  Please run on your system and see where are all these shadow copies are going to.

[zwoodcock]

Wallace

Well, according to vssadmin List ShadowStorage, they are disappearing into the Aether:

[wlee4 · Accepted solution]

Your output indicates that the shadow storage is not set up.

Did you look at the event logs, and there were no VSS issues?

On my system, I deleted all my shadows, bit it still shows the shadow storage association:

C:\windows\system32>vssadmin List volumes

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Volume path: C:\

    Volume name: \\?\Volume{1c425a42-db08-11e5-93c5-806e6f6e6963}\

C:\windows\system32>vssadmin List shadows

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

No items found that satisfy the query.

C:\windows\system32>vssadmin List Shadowstorage

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Shadow Copy Storage association

  For volume: (C:)\\?\Volume{1c425a42-db08-11e5-93c5-806e6f6e6963}\

  Shadow Copy Storage volume: (C:)\\?\Volume{1c425a42-db08-11e5-93c5-806e6f6e6963}\

  Used Shadow Copy Storage space: 0 B (0%)

  Allocated Shadow Copy Storage space: 0 B (0%)

  Maximum Shadow Copy Storage space: 46.576 GB (5%)

Refer to the following article on how to recreate the shadow storage area, and then see that changes the backup results.

https://www.petri.com/forums/forum/server-operating-systems/windows-server-2008-2008-r2/58571-vss-er...