Highlighted
2 Bronze

SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Good day;

Beginning roughly two days ago, the backup for one of our SQL Servers started failing out of the blue - no configuration changes, no server-side changes, nothing.

The only errors I'm getting are:

  1. Unsupported File System

          a. No change to the File System; don't know how this could be an issue

    2. Write-Protected Disc

          b. Nope; no changes made to disk permissions

    3. No space on Disc

          c. Plenty of space on all Drives on the server in question

    4. Drive not found

          d. No changes to Drive Naming, no drives added/removed.

I've been attempting to modify and troubleshoot the configuration for the past couple of days, but no dice. Any assistance is appreciated.

I have attached the full error text.

2017-05-30 09_44_31-NMM_Error.txt - Notepad.png

Thank you!

Labels (1)
0 Kudos
1 Solution

Accepted Solutions
Highlighted
3 Argentum

Re: Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Your output indicates that the shadow storage is not set up.

Did you look at the event logs, and there were no VSS issues?

On my system, I deleted all my shadows, bit it still shows the shadow storage association:

C:\windows\system32>vssadmin List volumes

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Volume path: C:\

    Volume name: \\?\Volume{1c425a42-db08-11e5-93c5-806e6f6e6963}\

C:\windows\system32>vssadmin List shadows

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

No items found that satisfy the query.

C:\windows\system32>vssadmin List Shadowstorage

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Shadow Copy Storage association

  For volume: (C:)\\?\Volume{1c425a42-db08-11e5-93c5-806e6f6e6963}\

  Shadow Copy Storage volume: (C:)\\?\Volume{1c425a42-db08-11e5-93c5-806e6f6e6963}\

  Used Shadow Copy Storage space: 0 B (0%)

  Allocated Shadow Copy Storage space: 0 B (0%)

  Maximum Shadow Copy Storage space: 46.576 GB (5%)

Refer to the following article on how to recreate the shadow storage area, and then see that changes the backup results.

https://www.petri.com/forums/forum/server-operating-systems/windows-server-2008-2008-r2/58571-vss-er...

View solution in original post

0 Kudos
19 Replies
Highlighted
2 Bronze

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Reply just to add - I already tried my generic Troubleshooting steps:

  1. Stop Services on host, rename TMP folder, Start Services on host
  2. Restart all NW Services on NW Server

Interestingly enough, the Filesystem backup for this exact same server is getting the same error message.

0 Kudos
Highlighted
7 Thorium

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

I see you use VSS for backup - did you try VDI to rule out any VSS related issue?

0 Kudos
Highlighted
3 Argentum

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

If the file system backups are also failing, then this should be looked at first.

BTW... this backup is not using nsrsqlsv to backup SQL.  You are using NMM to backup SQL using the VSS backup method.

For now... I suggest focusing on the file system backup failure.  On this client:

  1. what is the o/s version?
  2. what is the n/w client version?
  3. what is the NMM version?
  4. What is the SQL version?
  5. Is this a stand alone SQL server, or is this a cluster node?  If cluster, are the n/w and nmm versions the same on all nodes?
  6. what errors are reported during file system backups?   Are all backups affected?  Does any saveset succeed?
  7. Use NMM's config_checker.  are there any reported issues?
0 Kudos
Highlighted
2 Bronze

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Edit: I took a screenshot of the wrong error; the original screenshot included an error from when both the Filesystem and SQL Backups were occurring simultaneously.

Wallace,

First, that was my mistake. Through troubleshooting, I modified the configuration of the Client to match our Production Database Server's - which is using nsrsqlsv. You are correct in that I was using NMM when this error arose.

  1. The O/S is Win Server 2012 Standard
  2. The N/W Client version is not reporting; doesn't say (used print type: NSR Client;name:*client_name*)
  3. The NMM Version is 2.4.0.375
  4. The SQL Version is SQL Server 2012 SP1
  5. Stand-alone
  6. Filesystem Backup is reporting this error, and no savesets succeed.

         

    7. Config_Checker reports no issues.

0 Kudos
Highlighted
3 Argentum

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

@zwoodcock...  Funny i thought you updated this discussion.  Based on what i thought I saw:


  • NetWorker client: SQL 2012, Windows 2012, n/w client 8.0, NMM 2.4.0
  • NetWorker server: Windows, n/w 8.0
  • Error:  "There is another instance of NMM save program is already running"

This file system backup error only shows that NMM is already running. This is does does not itself show there is a technical problem, but a scheduling problem with the file system backups.  Do you see the same error all the time?  Is there a different error when backing up the file system?

You are using NMM 2.4, which is why NMM is used to backup the file system as well as SQL.  This was changed starting with NMM 3.0, where NMM was only used to backup the databases. The file system backups went back to using save.exe, just as if NMM was not installed and only NetWorker client was installed.   Note: NMM 2.4.x support was ended on March 31, 2016.  NMM 3 is still currently supported, and there are also more recent versions.

Since the issue only started 2+ days ago, the errors could be caused by a problem with Windows Volume Shadow Copy Services (VSS).  Please check the client's application and system even logs for events during the backup period.  Also run the following and look for any errors in the output.

  1. vssadmin list writers
  2. vssadmin list providers
  3. vssadmin list shadows

If there is a problem with VSS, then try rebooting this SQL server, and check VSS again.

0 Kudos
Highlighted
2 Bronze

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Wallace

I noticed the NMM Save Already Running - that was a mistake screenshot. I edited the reply with the correct screenshot, but it's still being "Moderated" so I guess you can't see it:

  1. vssadmin list writers shows all of the writers in a Stable state, with no error
  2. vssadmin list providers shows Microsoft File Share Shadow Copy provider, and Microsoft Software Shadow Copy Provider 1.0
  3. vssadmin list shadows shows "No items found that satisfy the query."
0 Kudos
Highlighted
3 Argentum

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

please try

     vssadmin List Volumes

     vssadmin List ShadowStorage


On my desktop:

     vssadmin list ShadowStorage

Shadow Copy Storage association

   For volume: (C:)\\?\Volume{24c476ae-db08-11e5-93c5-806e6f6e5563}\

   Shadow Copy Storage volume: (C:)\\?\Volume{24c476ae-db08-11e5-93c5-806e6f6e5563}\

   Used Shadow Copy Storage space: 33.818 GB (3%)

   Allocated Shadow Copy Storage space: 34.883 GB (3%)

   Maximum Shadow Copy Storage space: 46.576 GB (5%)

So this shows that for my c:, it creates the shadow copy on c: and shows how much space it currently uses, and max it can use.  Please run on your system and see where are all these shadow copies are going to.

Highlighted
2 Bronze

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Wallace

Well, according to vssadmin List ShadowStorage, they are disappearing into the Aether:

0 Kudos
Highlighted
3 Argentum

Re: Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Your output indicates that the shadow storage is not set up.

Did you look at the event logs, and there were no VSS issues?

On my system, I deleted all my shadows, bit it still shows the shadow storage association:

C:\windows\system32>vssadmin List volumes

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Volume path: C:\

    Volume name: \\?\Volume{1c425a42-db08-11e5-93c5-806e6f6e6963}\

C:\windows\system32>vssadmin List shadows

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

No items found that satisfy the query.

C:\windows\system32>vssadmin List Shadowstorage

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Shadow Copy Storage association

  For volume: (C:)\\?\Volume{1c425a42-db08-11e5-93c5-806e6f6e6963}\

  Shadow Copy Storage volume: (C:)\\?\Volume{1c425a42-db08-11e5-93c5-806e6f6e6963}\

  Used Shadow Copy Storage space: 0 B (0%)

  Allocated Shadow Copy Storage space: 0 B (0%)

  Maximum Shadow Copy Storage space: 46.576 GB (5%)

Refer to the following article on how to recreate the shadow storage area, and then see that changes the backup results.

https://www.petri.com/forums/forum/server-operating-systems/windows-server-2008-2008-r2/58571-vss-er...

View solution in original post

0 Kudos
Highlighted
2 Bronze

Re: Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Hello

I noticed initially it was stated that the SQL host is running 

NetWorker Client 8.0 and NMM 2.4.0.375 

Please NOTE:

NetWorker Client 8.0 and NMM 2.4.0, 2.4.1  are no longer supported. 

NetWorker Client 8.1 and NMM 3.0 , 3.0 SP1 are no longer supported. 

See Main NetWorker support page for PRODUCT SERVICE LIFE

https://support.emc.com/products/1095_NetWorker

Best ACTION:

You are running a current SQL 2012  or higher 

therefore,   best action is to remove NMM 2.4 and reboot

then remove the 8.0 client

Once removed,  Then remove or rename the old Networker Client installation path,

Then download the most current 8.2.4.x  NetWorker Client and NMM

Cumulative Product Downloads

CLIENT

ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes/8.2/8.2.4.6/

NMM

ftp://ftp.legato.com/pub/NetWorker/NMM/Cumulative_Hotfixes/8.2.4/8.2.4.6/

8.x product is similar to the 2.4  will allow you to run backups with a current 8.x product

before planning to move the MOST current product 9.1.1.x   

(There are some major differences in 9.x versus 8.x which is why first recommend upgrading to 8.x)

** If you are still having problems with the 8.2.4.x (Client and NMM)

then please open a service request ticket for support. 

Dell EMC Support

0 Kudos
Highlighted
2 Bronze

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Wallace

Followed the instructions, and my "vssadmin list shadowstorage" is now working correctly and displaying the shadow storage information for each drive... however, the backup is still failing with the same error message.

I just deleted the shadowcopy I had to create as part of the linked guide, and I'm running the backup again. I'll update when I see how that goes.

0 Kudos
Highlighted
3 Argentum

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Hope you create sufficient space for the shadow copies.  Check the nmm.raw for details on what happens during the backup.  Does the SQL backups also fail as well?

To simplify the matter, as well as to address @Mauro's concerns about compatibility, you should consider upgrading to a supported n/w version.  Since the n/w server is still 8.0, I would personally not upgrade higher than v8.2.2.6.  At least, after upgrading NMM, you only use it to backup SQL.  The file system backups are backed up using the default save.exe.

The NetWorker 8.2.2.6 client and NMM install kits can be downloaded from:


     ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes/8.2/8.2.2.6/nw822_win_x64.zip

     ftp://ftp.legato.com/pub/NetWorker/NMM/Cumulative_Hotfixes/8.2.2/8.2.2.6/nmm822_win_x64.zip

If you installed GLR (Granularity Level Recovery) option when you installed NMM, then you need to reboot during the NMM uninstall.  Installing with GLR will also require a reboot as well.

To determine if you select the GLR option, on the SQL server, open a command line in admin mode, and run:

     driverquery  | find /i "dos"

If it shows an eldos driver, then you will need to reboot during the uninstall and install process.

0 Kudos
Highlighted
2 Bronze

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Wallace

No need to update; I've got it working.

Looks like I just needed to delete the manually-created shadow-copies for each drive. Both Filesystem and SQL are working perfectly now.

Thanks so much for the help!

0 Kudos
Highlighted
3 Silver

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

SQL backups are performed based on SQL VDI technology.  Hence, checking VSS related errors are not going to help.

What is exact error reported in NSRSQLSV.RAW?  Make sure, you have not selected "Snapshot" for the group properties.  Hope you aware that, SQL VDI does not support snapshot method. 

Hope this helps!!

Thanks,

Elango C

0 Kudos
Highlighted
3 Argentum

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

zwoodcock has already told us that he is using VSS...   please look at the screenshots.

The title of this discussion is incorrect.

0 Kudos
Highlighted
3 Silver

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Just checked the attached screeshot.  I think NMM is trying to create a snapshot of Drive B:\.  Generally, A:\ and B:\ are assigned to Floppy drive.  Can you check whether B:\ drive is available or not?  Also, check the SQL Server writer components are listing B:\

You can, run a VSSToolBox v2 application (https://inside.dell.com/community/active/networker-support/blog/2016/09/29/vss-troubleshooting-appli...) and select the "SQL Server Writer" to see / check whether the drive mapping for the SQL components are good.

Thanks,

Elango C

0 Kudos
Highlighted
7 Thorium

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

I am getting old and I need glasses, but where do you see this B drive?

EDIT: Ah ok, now I see it... I missed that until now... strange indeed...

0 Kudos
Highlighted
2 Bronze

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Wallace

Hrvoje Crvelin

ElanC

You mentioned the B:\ Drive: We do - in fact - have a B:\ Drive on this server (God help me; managing servers that someone else imaged and configured is a damned nightmare), but it is labeled as "System Reserved". Older savesets show successful backups of this drive, so I don't know why that would suddenly become a problem.

0 Kudos
Highlighted
2 Bronze

Re: SQL Backup Configured as NSRSQLSV suddenly failing - error attached

Jump to solution

Replying to original question:

Hello

I noticed initially it was stated that the SQL host is running

NetWorker Client 8.0 and NMM 2.4.0.375

Please NOTE:

NetWorker Client 8.0 and NMM 2.4.0, 2.4.1  are no longer supported.

NetWorker Client 8.1 and NMM 3.0 , 3.0 SP1 are no longer supported.

See Main NetWorker support page for PRODUCT SERVICE LIFE

https://support.emc.com/products/1095_NetWorker

Best ACTION:

You are running a current SQL 2012  or higher

therefore,   best action is to remove NMM 2.4 and reboot

then remove the 8.0 client

Once removed,  Then remove or rename the old Networker Client installation path,

Then download the most current 8.2.4.x  NetWorker Client and NMM

Cumulative Product Downloads

CLIENT

ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes/8.2/8.2.4.6/

NMM

ftp://ftp.legato.com/pub/NetWorker/NMM/Cumulative_Hotfixes/8.2.4/8.2.4.6/

8.x product is similar to the 2.4  will allow you to run backups with a current 8.x product

before planning to move the MOST current product 9.1.1.x  

(There are some major differences in 9.x versus 8.x which is why first recommend upgrading to 8.x)

** If you are still having problems with the 8.2.4.x (Client and NMM)

then please open a service request ticket for support.

Dell EMC Support

0 Kudos