Start a Conversation

This post is more than 5 years old

Solved!

Go to Solution

10126

April 9th, 2013 01:00

Skipping "C:\" although Saveset is set to "All" and failing "Disaster_Recovery:\"

Networker Server ("backup1" or "rx600" [DNS Alias]):

  • Networker Server: "NetWorker 7.6.3.5.Build.891 Network Edition/219"
  • OS: Linux version 2.6.32.12-0.7-default
  • OS: SUSE Linux Enterprise Server 11 (x86_64) ; VERSION = 11 ; PATCHLEVEL = 1
  • same network as Datadomain

Networker Client ("samira"):

  • Networker Client: "8.0.1.3.Build.155"
  • new installation, not upgraded
  • OS: Windows Server 2008 R2 SP1 64 bit, German Version
  • Physical server: HP PL DL360 G7, 12 GB RAM
  • 1 Disk (600 GB): 3 volumes (system-reserved Z: 100MB, System C: 98GB, Data D: 460GB)
  • Every volume has at least 50 % of free space
  • roles: Domain Controller, Fileservices, DHCP-Server, DNS-Server
  • remote office
  • firewall allows any traffic between Client on the one side and Networker Server / Datadomain on the other side

Datadomain Device ("dd1" - name of device in NMC "dd1_LSU-Backup1"):

  • DD670
  • Networker Client does "Direct File Access" or "Client direct" to Datadomain

Problem:

  • "Saveset All" saves:
    • sucessfully:
      samira.sis.net Z:\ 14 MB full 00:00:10 09.04.13 08:36:03
      samira.sis.net Z:\ 14 MB full 00:00:25 09.04.13 07:56:47
      samira.sis.net D:\ 3521 MB incr 00:09:05 09.04.13 07:56:42
      samira.sis.net samira.sis.net:index 60 MB 9 00:00:00 09.04.13 09:03:03
      samira.sis.net VSS SYSTEM BOOT:\ 753 MB full 00:01:55 09.04.13 08:35:58
      samira.sis.net VSS SYSTEM BOOT:\ 743 MB full 00:03:15 09.04.13 07:56:37
      samira.sis.net VSS SYSTEM SERVICES:\ 33 MB full 00:00:05 09.04.13 08:37:53
      samira.sis.net VSS SYSTEM SERVICES:\ 33 MB full 00:00:15 09.04.13 07:59:52
    • but fails:
      samira.sis.net DISASTER_RECOVERY:\ incr 09.04.13 08:35:19 00:27:39 savegrp: suppressed 2 lines of verbose output System Writer - Error saving writer System Writer System Writer - ERROR: Aborting backup of saveset VSS SYSTEM FILESET: because of the error with writer System Writer. System Writer - Error saving 90110:save: save of 'VSS SYSTEM FILESET:\' to server 'rx600.swsn.net' failed: No such file or directory 90123:save: VSS save walkers: the backup of save set 'VSS SYSTEM FILESET:\' failed. 90123:save: VSS save walkers: the backup of save set 'VSS SYSTEM FILESET:\' failed. 86024:save: Error occured while saving disaster recovery save sets. : Failed with error(s)
      samira.sis.net VSS SYSTEM FILESET:\ full 09.04.13 08:37:58 00:25:00 9975 MB
      samira.sis.net VSS SYSTEM FILESET:\ full 09.04.13 08:00:07 00:35:11 9975 MB
  • summary:
    • C:\ is missing
    • all savesets but "D:\" try to save two times: "Z:\", "VSS SYSTEM BOOT:\", "VSS SYSTEM SERVICES", "VSS SYSTEM FILESET:\"
    • "DISASTER_RECOVERY:\" fails with error "VSS SYSTEM FILESET:\ [...] No such file or directory"
  • entries in Eventlog on Client OS:
    • Application Log: "Disaster Recovery: backup of critical volumes failed." (Source Networker)
    • System Log: No relevant entries
    • daemon.raw on NSR Client has no relevant entries
    • daemon.raw on NSR Server has no relevant entries as far as I can see it (attached daemon.raw parts from relevant time)
  • Saving "C:\" as single Saveset works just fine
  • vssadmin list shadowstorage duriung backup of Client shows that shadowstorage for all volumes is located on D:\:
      • C:\Users\jfuchs>vssadmin list shadowstorage

        vssadmin 1.1 - Verwaltungsbefehlszeilenprogramm des Volumeschattenkopie-Dienstes

        (C) Copyright 2001-2005 Microsoft Corp.

        Schattenkopie-Speicherassoziation

        Für Volume: (Z:)\\?\Volume{bab27ec0-82b8-11e0-8d43-806e6f6e6963}\

        Schattenkopie-Speichervolume: (D:)\\?\Volume{d33fda03-6439-11e1-ae86-e4115bedee8a}\

        Verwendeter Schattenkopie-Speicherbereich: 400 KB (0%)

        Zugewiesener Schattenkopie-Speicherbereich: 2.93 GB (0%)

        Max. Schattenkopie-Speicherbereich: UNEINGESCHRÄNKT (-568898606%)

        Schattenkopie-Speicherassoziation

        Für Volume: (C:)\\?\Volume{bab27ec1-82b8-11e0-8d43-806e6f6e6963}\

        Schattenkopie-Speichervolume: (D:)\\?\Volume{d33fda03-6439-11e1-ae86-e4115bedee8a}\

        Verwendeter Schattenkopie-Speicherbereich: 40.531 MB (0%)

        Zugewiesener Schattenkopie-Speicherbereich: 2.93 GB (0%)

        Max. Schattenkopie-Speicherbereich: UNEINGESCHRÄNKT (-568898606%)

        Schattenkopie-Speicherassoziation

        Für Volume: (D:)\\?\Volume{d33fda03-6439-11e1-ae86-e4115bedee8a}\

        Schattenkopie-Speichervolume: (D:)\\?\Volume{d33fda03-6439-11e1-ae86-e4115bedee8a}\

        Verwendeter Schattenkopie-Speicherbereich: 4.031 MB (0%)

        Zugewiesener Schattenkopie-Speicherbereich: 2.93 GB (0%)

        Max. Schattenkopie-Speicherbereich: UNEINGESCHRÄNKT (-568898606%)

    • is it normal that "system reserved" has the same volume-id (see abough) as drive "c:\" ?

I would like to:

  • save "C:\" properly with Saveset "All"
  • get rid of the "DISASTER_REECOVERY:\" failures (maybe the first comes together with the second problem?)

We have about ten other Networker Clients with an almost equal Client environment but only this is failing.

Anyone with ideas? Are there special VSS log files that I could analyse?

Many thanks in advance to all,

Jens Fuchs

SIS - Schweriner IT- und Servicegesellschaft mbH

1 Attachment

1 Message

April 10th, 2013 04:00

Hi, might be an issue with file(s) missing on disk or registry.

Please try another backup after changing Client properties for this client.

Client properties-->>Apps&Modules-->Backup Command, add this command:

save -a '"ignore-all-missing-system-files=yes"'

This will let networker ignore if any files are missing.

By default networker fails the saveset if a single file is missing on backup even if thosands have

gone OK.

I see this problem often on Windows there the registry often has a link to a file that does not exist

anymore on disk. Might be some problem with windows installer and 3rd party software vendors that

don't uninstall products/drivers well.

  //brgds Thomas

April 9th, 2013 11:00

Run 'vssadmin list writers' and see if system writer is being showed failed.

1.7K Posts

April 10th, 2013 02:00

Hi Jens,

I suspect there is some issue with the System Writer or with some VSS component.

if you run vssadin listw riters, as Mehrota suggested, you should see all the writers (especially SYSTEM WRITER) as status [1] stable and "no error".

Did you, by any chance tried to browse the content of those backups? Just open NW client on the client and find out if there is ny valid backup for C:\

Thank you,

Carlos.

1.7K Posts

April 10th, 2013 03:00

ok,

unless I-m missing something this is an odd behaviour.

Could you please run this command?

savegrp - D3vvv -c client_name group_name > C:\save.out 2>&1

then please attach the C:\save.out

This to be ran on the NW server

Thank you,

Carlos

34 Posts

April 10th, 2013 03:00

If I open Networker Client and browse for C:\ I just see the backups that I do for saveset C:\ . In NMC ->Media -> Indexes same: Whenever saveset is All , index for saveset C:\ shows no backup for that time...

output of vssadmin list writers shows no errors:

vssadmin 1.1 - Verwaltungsbefehlszeilenprogramm des Volumeschattenkopie-Dienstes

(C) Copyright 2001-2005 Microsoft Corp.

Verfassername: "Task Scheduler Writer"

   Verfasserkennung: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}

   Verfasserinstanzkennung: {1bddd48e-5052-49db-9b07-b96f96727e6b}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "VSS Metadata Store Writer"

   Verfasserkennung: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}

   Verfasserinstanzkennung: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "Performance Counters Writer"

   Verfasserkennung: {0bada1de-01a9-4625-8278-69e735f39dd2}

   Verfasserinstanzkennung: {f0086dda-9efc-47c5-8eb6-a944c3d09381}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "System Writer"

   Verfasserkennung: {e8132975-6f93-4464-a53e-1050253ae220}

   Verfasserinstanzkennung: {97573503-b312-4ca6-89d7-45727fd0b23b}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "SqlServerWriter"

   Verfasserkennung: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}

   Verfasserinstanzkennung: {359e60c5-720b-4c17-8cf9-224ae04b05e1}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "ASR Writer"

   Verfasserkennung: {be000cbe-11fe-4426-9c58-531aa6355fc4}

   Verfasserinstanzkennung: {7e40d56e-3031-429f-ae93-baed2daac238}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "Shadow Copy Optimization Writer"

   Verfasserkennung: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

   Verfasserinstanzkennung: {5c051298-fd53-4525-9468-5f6606994538}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "WMI Writer"

   Verfasserkennung: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}

   Verfasserinstanzkennung: {33128199-92e7-442c-9ba9-119aad9980fd}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "Registry Writer"

   Verfasserkennung: {afbab4a2-367d-4d15-a586-71dbb18f8485}

   Verfasserinstanzkennung: {cbec4606-b6c4-4b97-8997-ed35fde002f6}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "COM+ REGDB Writer"

   Verfasserkennung: {542da469-d3e1-473c-9f4f-7847f01fc64f}

   Verfasserinstanzkennung: {db895b68-06bb-406e-bfeb-a20af8c3b090}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "NTDS"

   Verfasserkennung: {b2014c9e-8711-4c5c-a5a9-3cf384484757}

   Verfasserinstanzkennung: {749a18c4-56fb-46a6-acc7-d2b947f11aeb}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "DFS Replication service writer"

   Verfasserkennung: {2707761b-2324-473d-88eb-eb007a359533}

   Verfasserinstanzkennung: {26bbd90a-fd42-46bf-9e52-0dc9947f73d4}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Verfassername: "Dhcp Jet Writer"

   Verfasserkennung: {be9ac81e-3619-421f-920f-4c6fea9e93ad}

   Verfasserinstanzkennung: {75111493-f3bd-47e8-b63e-6915ad5b71c5}

   Status: [1] Stabil

   Letzter Fehler: Kein Fehler

Networker support said that VSS should be the problem. Requested output /nsr/tmp/sg folder of group while saving only "DISASTER_RECOVERY:\" saveset with Debug-Level 9.

1.7K Posts

April 10th, 2013 05:00

Hi Jens, This is a very well known VSS issue. The root cause of this problem is when for instance you uninstall a piece of software (in this casa an anti virus) the registry doesn't get cleaned out so, when the registry writer tries to backup all the files pointint to that registry key fails to find it, as the file is no longer present,so the workaround EMC created is that flag for the save.exe command (save -a ignore...). I recommend you to use this approach rather than creating the dummy file as otherwise, there is a chance that there are multiple missing files and you will need to identify all of them by checking the logs and then create those dummy files. Let me know if you need any further informatiom. Thank you Carlos

34 Posts

April 10th, 2013 05:00

Hi Carlos,

as requested from EMC Support I ran the group only with "DISASTER_RECOVERY:\" saveset with Debug-Level 9. Finally we've got a real error message that sounds familiar:

System Writer - Error saving file path C:\program files (x86)\mcafee\virusscan enterprise\x64\engineserver.exe that is associated with Shadow Copy path \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy34\program files (x86)\mcafee\virusscan enterprise\x64\engineserver.exe. Das System kann die angegebene Datei nicht finden. C:\program files (x86)\mcafee\virusscan enterprise\x64\

I know now 2 work-arounds for that:

  1. create empty dummy-file for "C:\program files (x86)\mcafee\virusscan enterprise\x64\engineserver.exe"
  2. save -a '"ignore-all-missing-system-files=yes"'

Do you know how this problem derives? The file "virusscan enterprise\x64\engineserver.exe" was moved to other folder months ago to folder "virusscan enterprise_\x64\engineserver.exe". How comes that VSS thinks it's still at the old location? Are there known methods for "resetting" VSS "brain"?

Thanks so far,

Jens

34 Posts

April 10th, 2013 07:00

We removed McAfee package completely, deleted obsolete registry entries to missing file and reinstalled McAfee - Backup now works properly, no double backups anymore, C:\ backups with "All" again.

Registry keys (after uninstallation of McAfee) that affected the backup to fail:

HKLM\SYSTEM\ControlSet001\services\McAfeeEngineService

HKLM\SYSTEM\CurrentControlSet\services\McAfeeEngineService

That maybe came along with an unproper user attempt to upgrade McAfee. As we saw an additional folder

C:\Program Files (x86)\McAfee\VirusScan Enterprise_

(where the missing .exe was still to be found)

along with

C:\Program Files (x86)\McAfee\VirusScan Enterprise

So, maybe an OSI Layer 8 problem :-) instead of an unproper MSI upgrade process?

Thanks to all!

No Events found!

Top