EX2013 CU7 granular recovery NW 8.1.1 NMM 3.0.1 RDB empty, no browsing

Hello JoergS-AS 

Exchange 2013 CU7 is not yet qualified hence not "officially supported" yet, however this is targeted to be done soon for NMM 3.0.1 and NMM 8.2

Having this said let's try to move forward with it and try to solve it:

You said RDB, but there is not really any RDB created but a new DB called GLRxxxx where xxx is the date and time of the DB creation.

Please make sure all the requirements shown in the error pop-up are met (including the send as-receive as).

Also, have you checked Application and system event logs to see if there is any error that could lead you to the root cause of the problem?

Can you please let us know what is the NW client version and build as well as the build of NMM?

Have you ran NMM Config Checker? If so, did you get any error or warning?

Make sure you use the correct MAPI version.

Check the libmapibrowse log to retrieve more information.

Thank you,

Carlos

you may look at the libmapibrowse.raw first.

Is this a new install of NMM on this machine or only a CU Upgrade wich worked prior ?

If it is a new install of nmm , wich version of mapicdo did you use?

even all are called 1.2.1, you need the version updated in 2013:

Download Microsoft Exchange Server MAPI Client and Collaboration Data Objects 1.2.1 March 2014 Update from Official Micr…

Hi again,

Set Throttling policy to increase the RCA Burst.  Use Exchange Management Shell (PowerShell);

             
New-ThrottlingPolicy -Name -RcaMaxBurst:Unlimited -IsServiceAccount:$true

             
Set-ThrottlingPolicyAssociation -ThrottlingPolicy:

   
Install  latest MAPI/CDO1.2.1

-- For Exchange 2007/2010 MAP/CDO should be  6.5.8244

-- For Exchange 2013 MAPI/CDO shuold be 6.5.8320 or later

Old MAPI must be Uninstalled first

After the above perform an RDB restore. 

If after the above you are not able to browse the RDB, then close NMM Gui, open cmd prompt in Administrator mode, run  this command
                           
WINRM QUICKCONFIG

                           
then open NMM Gui and proceed.

And please, check Application and System event logs in case it fails.

Thank you,

Carlos

Hey guys,

mapi/CDO is 6.5.8320.0. NW and NMM is a new installation, nmm build is 3.0.1.2.Build.374.ntx64

Send-as/Receive-as Rights are set for our users.

libmapibrowse give us an info:

81528 1421926529 0 0 0 9416 10584 0 ex2013-01.domain.tld (pid10584) NSR info 28 %s(%d): Using mailbox [%s]:  3 23 7 profile 1 3 613 0 15 exbackup-admin

118219 1421926529 0 0 0 9416 10584 0 ex2013-01.domain.tld (pid10584) NSR info 58 %s(%d): Configuring message service - [%d] property tags:  3 23 7 profile 1 3 767 1 2 11

100264 1421926572 0 0 0 9416 10584 0 ex2013-01.domain.tld (pid10584) NSR info 135 The Exchange Information Store is busy. Please retry after 5 minutes. On Exchange 2013, check if the necessary ThrottlingPolicy is set. 0

52525 1421926572 2 0 0 9416 10584 0 ex2013-01.domain.tld (pid10584) NSR warning 25 %s(%d): %s() = 0x%.8x: %s 5 23 4 mapi 1 4 1526 0 12 OpenMsgStore 1 11 -2147221231 24 53 The information store could not be opened. (MAPI 1.0)

We check our throttling Policy.

Please check the following KB:

  191960 : Troubleshooting NMM Exchange RDB BRowsing and GLR Issues            

https://support.emc.com/kb/191960

Thank you,

Carlos

Hey carlos,

thanks for cmdlets. Throttling Policy is created and associated to the exchange backup user.

mapi cdo is alreay installed 1.2.1 6.5.8320

WINRM QUICKCONFIG is already set

NMM Gui is started as administrator.

Same "error" empty GLR

97690 1422018439 0 0 0 11780 22332 0 ex2013-domain.tld (pid22332) NSR info 45 %s(%d): Calling [create_ems_service_exch2013] 2 23 4 mapi 1 3 360

81528 1422018439 0 0 0 11780 22332 0 ex2013-domain.tld (pid22332) NSR info 28 %s(%d): Using mailbox [%s]:  3 23 7 profile 1 3 613 0 15 exbackup-admin

118219 1422018439 0 0 0 11780 22332 0 ex2013-01.domain.tld (pid22332) NSR info 58 %s(%d): Configuring message service - [%d] property tags:  3 23 7 profile 1 3 767 1 2 11

100264 1422018484 0 0 0 11780 22332 0 ex2013-01.domain.tld (pid22332) NSR info 135 The Exchange Information Store is busy. Please retry after 5 minutes. On Exchange 2013, check if the necessary ThrottlingPolicy is set. 0

52525 1422018484 2 0 0 11780 22332 0 ex2013-01.domain.tld (pid22332) NSR warning 25 %s(%d): %s() = 0x%.8x: %s 5 23 4 mapi 1 4 1526 0 12 OpenMsgStore 1 11 -2147221231 24 53 The information store could not be opened. (MAPI 1.0)

46116 1422018484 5 0 0 11780 22332 0 ex2013-01.domain.tld (pid22332) NSR critical 73 %s(%d): Could not open EMS Store for Exchange server: %s, status 0x%0.8x. 4 23 7 adsutil 1 4 4037 0 12 ex2013-01 1 11 -2147221231

81504 1422018484 0 0 0 11780 22332 0 ex2013-01.domain.tld (pid22332) NSR info 41 %s(%d): Exiting exch_get_RSG_capability() 2 23 17 libmapibrowse.cpp 1 4 1056

In Eventlog app log reports a warning:

The Microsoft Exchange Mailbox Replication service was unable to process jobs in a mailbox database.

Database: GLR20150123135616

Error: MapiExceptionMdbOffline: Unable to make connection to the server. (hr=0x80004005, ec=1142)

Diagnostic context:

    Lid: 41192   dwParam: 0x1

    Lid: 49384

    Lid: 51176   StoreEc: 0x476  

    Lid: 48104

    Lid: 39912   StoreEc: 0x476  

    Lid: 41192   dwParam: 0x2

    Lid: 49384

    Lid: 51176   StoreEc: 0x476  

    Lid: 48104

    Lid: 39912   StoreEc: 0x476  

    Lid: 41192   dwParam: 0x0

    Lid: 49064   dwParam: 0x1

    Lid: 38439   EMSMDBPOOL.EcPoolConnect called [length=48]

    Lid: 54823   EMSMDBPOOL.EcPoolConnect returned [ec=0x476][length=20][latency=0]

    Lid: 53361   StoreEc: 0x476  

    Lid: 51859

    Lid: 33649   StoreEc: 0x476  

    Lid: 43315

    Lid: 58225   StoreEc: 0x476  

    Lid: 39912   StoreEc: 0x476  

    Lid: 54129   StoreEc: 0x476  

    Lid: 50519

    Lid: 59735   StoreEc: 0x476  

    Lid: 59199

    Lid: 27356   StoreEc: 0x476  

    Lid: 65279

    Lid: 52465   StoreEc: 0x476  

    Lid: 60065

    Lid: 33777   StoreEc: 0x476  

    Lid: 59805

    Lid: 52487   StoreEc: 0x476  

    Lid: 19778

    Lid: 27970   StoreEc: 0x476  

    Lid: 17730

    Lid: 25922   StoreEc: 0x476

and this warning

The Microsoft Exchange Mailbox Replication service was unable to process jobs in a mailbox database.

Database: GLR20150123142302

Error: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80004005, ec=2423)

Diagnostic context:

    Lid: 47655   EMSMDBPOOL.EcPoolCreateSession called [length=375]

    Lid: 39463   EMSMDBPOOL.EcPoolCreateSession exception [rpc_status=0x6BA][latency=0]

    Lid: 62184 

    Lid: 16280   dwParam: 0x0 Msg: EEInfo: ComputerName: n/a

    Lid: 8600    dwParam: 0x0 Msg: EEInfo: ProcessID: 18436

    Lid: 12696   dwParam: 0x0 Msg: EEInfo: Generation Time: 0415-01-23T13:37:19.4920000Z

    Lid: 10648   dwParam: 0x0 Msg: EEInfo: Generating component: 2

    Lid: 14744   dwParam: 0x0 Msg: EEInfo: Status: 1722

    Lid: 9624    dwParam: 0x0 Msg: EEInfo: Detection location: 10

    Lid: 13720   dwParam: 0x0 Msg: EEInfo: Flags: 0

    Lid: 11672   dwParam: 0x0 Msg: EEInfo: NumberOfParameters: 2

    Lid: 12952   dwParam: 0x0 Msg: EEInfo: prm[0]: Long val: 3221225524

    Lid: 8856    dwParam: 0x0 Msg: EEInfo: prm[1]: Unicode  string: \RPC Control\LRPC-b290f29d28d00d9e09

    Lid: 62184 

    Lid: 16280   dwParam: 0x0 Msg: EEInfo: ComputerName: n/a

    Lid: 8600    dwParam: 0x0 Msg: EEInfo: ProcessID: 18436

    Lid: 12696   dwParam: 0x0 Msg: EEInfo: Generation Time: 0415-01-23T13:37:19.4920000Z

    Lid: 10648   dwParam: 0x0 Msg: EEInfo: Generating component: 2

    Lid: 14744   dwParam: 0x0 Msg: EEInfo: Status: 1727

    Lid: 9624    dwParam: 0x0 Msg: EEInfo: Detection location: 1010

    Lid: 13720   dwParam: 0x0 Msg: EEInfo: Flags: 0

    Lid: 11672   dwParam: 0x0 Msg: EEInfo: NumberOfParameters: 1

    Lid: 12952   dwParam: 0x0 Msg: EEInfo: prm[0]: Long val: 3221225527

    Lid: 55537   StoreEc: 0x6BA    

    Lid: 43249 

    Lid: 41073   StoreEc: 0x977    

    Lid: 48243 

    Lid: 50033   StoreEc: 0x977    

    Lid: 50544   ClientVersion: 15.0.1044.25

    Lid: 52080   StoreEc: 0x977    

    Lid: 51152 

    Lid: 52465   StoreEc: 0x977    

    Lid: 60065 

    Lid: 33777   StoreEc: 0x977    

    Lid: 59805 

    Lid: 52487   StoreEc: 0x977    

    Lid: 19778 

    Lid: 27970   StoreEc: 0x977    

    Lid: 17730 

    Lid: 25922   StoreEc: 0x977    

looks like the wrong mapi cdo ...

should be 8353

Download Microsoft Exchange Server MAPI Client and Collaboration Data Objects 1.2.1 March 2014 Update from Official Micr…

from a normal anatonmy, the browse would start AFTER the writer recovery ( this is a CU7 on nw 8.2.0.5 with nmm82 )

Get-EventLog -LogName Application -Newest 25 -Source Networker,NMM,VSS,msexchangerepl | Sort- Object Index -Descending | ft Message Message ------- The VSS service is shutting down due to idle timeout.... Database: labbuildrDAG_DB1\E2013N1... profile(608): Using mailbox [NMMBackupUser]: ... profile(608): Using mailbox [NMMBackupUser]: ... profile(608): Using mailbox [NMMBackupUser]: ... NMM .. restore completed and returned no error. ... NMM .. Mounted Exchange database [GLR20150123155309].... Exchange Replication Service VSS Writer will perform database recovery on database 'GLR20150123155309.edb' as part o... Exchange Replication Service VSS Writer successfully restored the backup set. In order to bring the restored databas... Exchange Replication Service VSS Writer will restore a backup set to database 'GLR20150123155309\E2013N1', which is ... Exchange Replication Service VSS Writer will restore a backup set using the following Restore Options string:... NMM .. Completing GLR restore processing.... The Microsoft Exchange VSS Writer has successfully collected the metadata document in preparation for backup. NMM .. Initialization success -- Exchange shell successfully initialized. Required for Exchange 2010 or Exchange 201... NMM .. Initialization success -- ExchEseHlpr successfully initialized. Required for GLR Restore.... NMM .. Initialization success --  Exchange shell successfully initialized. Required for Exchange 2010 or Exchange 20... NMM is starting the recover operations.... flag=? arg=labbuildrdag.labbuildr.local... Microsoft Exchange Server Locator Service failed to find active server for database '7f2d6df5-007e-4956-a5b4-fd552d7...

as i said, wrong mapi.....

you may want to install a testdrive ...

https://community.emc.com/blogs/bottk/2014/06/16/announcement-labbuildr-released

sources.vhd contains the needed cdo ...

hey,

thanks for infos.

Mapi CDO ist now updated to 1.2.1 (Update March 14, 8353)

With this CDO we have no errors in NMM Gui, but the nmm.raw says:

102436 1422265266 0 0 0 13704 24724 0 ex2013-01.domain.tld nsrsnap_vss_recover NSR info 26 vssClientRestore() failed. 0

102437 1422265266 0 0 0 13704 24724 0 ex2013-01.domain.tld nsrsnap_vss_recover NSR info 84 Recovery operation failed.  Refer to log file(s) and Windows event logs for details. 0

eventlogs vss:

Volume Shadow Copy Service error: Unexpected error An older writer session state is being removed. .  hr = 0x80042409, Writer status is not available for one or more writers.  A writer may have reached the limit to the number of available backup-restore session states.

.

Operation:

   PreRestore Event

Context:

   Old snapshot set: {0e8fe462-7ac5-40d1-9de9-10ce2d6a32ee}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Execution Context: Writer

   Writer Class Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}

   Writer Name: Microsoft Exchange Writer

   Writer Instance Name: Exchange Replication Service

   Writer Instance ID: {a805e73b-c1f4-4d6d-98d4-16191d9fa61d}

----------

Volume Shadow Copy Service error: Unexpected error An older writer session state is being removed. .  hr = 0x80042409, Writer status is not available for one or more writers.  A writer may have reached the limit to the number of available backup-restore session states.

.

Operation:

   PreRestore Event

Context:

   Old snapshot set: {0e8fe462-7ac5-40d1-9de9-10ce2d6a32ee}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Old snapshot set: {763d46ca-bddf-4dd8-8356-3d3e152416e6}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Execution Context: Writer

   Writer Class Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}

   Writer Name: Microsoft Exchange Writer

   Writer Instance Name: Exchange Replication Service

   Writer Instance ID: {a805e73b-c1f4-4d6d-98d4-16191d9fa61d}

-----------

Volume Shadow Copy Service error: Unexpected error An older writer session state is being removed. .  hr = 0x80042409, Writer status is not available for one or more writers.  A writer may have reached the limit to the number of available backup-restore session states.

.

Operation:

   PreRestore Event

Context:

   Old snapshot set: {0e8fe462-7ac5-40d1-9de9-10ce2d6a32ee}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Old snapshot set: {763d46ca-bddf-4dd8-8356-3d3e152416e6}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Old snapshot set: {0d345117-9e17-46d1-87d2-4bd54d47819b}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Execution Context: Writer

   Writer Class Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}

   Writer Name: Microsoft Exchange Writer

   Writer Instance Name: Exchange Replication Service

   Writer Instance ID: {a805e73b-c1f4-4d6d-98d4-16191d9fa61d}

----

Volume Shadow Copy Service error: Unexpected error An older writer session state is being removed. .  hr = 0x80042409, Writer status is not available for one or more writers.  A writer may have reached the limit to the number of available backup-restore session states.

.

Operation:

   PreRestore Event

Context:

   Old snapshot set: {0e8fe462-7ac5-40d1-9de9-10ce2d6a32ee}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Old snapshot set: {763d46ca-bddf-4dd8-8356-3d3e152416e6}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Old snapshot set: {0d345117-9e17-46d1-87d2-4bd54d47819b}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Old snapshot set: {4b09a3ac-a1b6-47f5-b7ed-8729d6239da0}

   Old operation: 1016

   Old state: 1

   Old failure: 0

   Old extended failure hr: 0x1

   Old extended failure message: 0

   Execution Context: Writer

   Writer Class Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}

   Writer Name: Microsoft Exchange Writer

   Writer Instance Name: Exchange Replication Service

   Writer Instance ID: {a805e73b-c1f4-4d6d-98d4-16191d9fa61d}

-------------

eventlogs nmm:

NMM .. gather writer status  after PreRestore returned status failure STABLE for writer 'Microsoft Exchange Writer'.  Writer failure 0x800423f3 -- VSS_E_WRITERERROR_RETRYABLE

----

NMM .. After PreRestore ... writer status indicates a failure -- 0x800423f3, check the Windows event logs for possible VSS related errors.

----

NMM .. Failure during VSS pre processing of Exchange GLR restore.  Error 0x800423f3.

----

Check auf VSSwriters:

All Writers are stable, with no error

no Shadows are listed

Hi JoergS-AS 

This could be a problem with the garbage collection system of the VSS writer.

Have you rebooted that box before performing the restore?

You can enable VSS traces to look deeper into the problem. To do it so please see below:

How to perform a VSS trace:

1. Create a file tracefile.reg using the contents shown below and change the TraceFile entry to point to a volume that is not going to be shadow copied. Note the double backslash delimiter usage-you need to enter "\\" as the delimiter for each backslash in the path you wish to specify.

2. Install tracing.reg.

3. Reproduce the problem.

4. Turn off tracing by deleting the key "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing"

Here are the contents of the tracefile.reg registry file:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing]

"TraceFile"="c:\\trace.txt"

"TraceLevel"=dword:ffffffff

"TraceEnterExit"=dword:00000001

"TraceToFile"=dword:00000001

"TraceToDebugger"=dword:00000000

"TraceFileLineInfo"=dword:00000001

"TraceForceFlush"=dword:00000000

P.S.: Note that depending on the OS this method to enable VSS traces may be different.

Before referring you to Microsoft, I would like to see the output of the reboot+attempt and also the traces of VSS.

You can first enable the VSS traces and then reboot and give it another try.

Also make sure there is nothing left from previous restore attempts (such as GLR, RDB mounted nor any folder7file associated to the GLR/RDB).

Thank you,

Carlos

yes we rebooted the server(s) and same errors in eventlog:

NMM .. gather writer status  after PreRestore returned status failure STABLE for writer 'Microsoft Exchange Writer'.  Writer failure 0x800423f3 -- VSS_E_WRITERERROR_RETRYABLE

----

NMM .. After PreRestore ... writer status indicates a failure -- 0x800423f3, check the Windows event logs for possible VSS related errors.

----

NMM .. Failure during VSS pre processing of Exchange GLR restore.  Error 0x800423f3.

----

SDK with VSS trace can´t be installed on 2012 R2 ...

Hi JoergS-AS 

There is no need to install SDK, just make changes in the registry to enable VSS traces. I'm sure your Windows admin will be able to do that.

Thank you,

Carlos

Can you please add your Errormessage. There where several Messages in this Thred ( i encourage open a new thread anyway )

Hello,

i've got the same error message and checked all (i hope ;-) ). Any news or suggestions around this?

Thank you!

F.Albers

Hello,

we're on Exchange 2013 CU7, Windows 2012 R2, NMM Version 8.2.1. and

latest MAPI Version 1.2.1 Version 6.5.8353.0. Backup the whole Database

into a Recovery DB is working but then choosing 'RDB Databaserecover'

the MAPI error comes.

Thank you!

Best Regards,

F. Albers

Florian Albers