Avamar: ADMe-Exchange BU recovery using -exchvssavtar (avtar)
Summary: ADMe can recover Exchange backups to a generic Windows staging server without the need for the Exchange S/W or the Avamar Exchange plug-in. Transaction logs in this case must be manually replayed against the edb files using Eseutil in order to get it into a Clean Shutdown state. The following procedures document the Eseutil steps required to replay the transaction logs. ...
Instructions
A typical ADMe job policy used to stage Exchange backup to a generic Windows staging server without the need for Microsoft Exchange S/W nor Avamar Exchange plug-in should look similar to the following example.
The ADMe option required to invoke this process is "-exchvssavtar" which must be assigned to the Migration Flags line or field as shown.
================< Job Policy Contents >================= Environment_Number:: 1 Base_Group_Name:: exch01 Sub_Group_Numbers:: 0 Select_Flags:: -d -last -exchvss View_Flags:: -d Migration_Flags:: -d -stageonly -D -m -exchvssavtar (include -stageonly only if you plan to confirm the eseutil steps) =====================================================================================================
TEST-1:
In this lab test, Exchange backup #293 was staged to a target folder named C:\ADME3\exch01 using ADMe option "-exchvssavtar."
Navigate down the path name to where the edb, log files, and control files exist which in this example sees highlighted path.
In this example, the edb and logs files were in a common directory but in a customer environment edb and logs are almost always in separate folders on separate drive letters.
C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox>cd DB1
From a command prompt with admin privilege, go to the directory containing the edb and log files.
List folder contents for both the edb and log file locations and copy and paste contents to notepad for reference, in particular for the timestamps and available log file names.
C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1> dir Directory of C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1
01/11/2023 05:22 PM 142,671,872 DB1.edb <========note the edb file timestamp is from January 11, 2023 the time of the VSS Exchange backup.
01/12/2023 03:22 AM 8,192 E01.chk 01/12/2023 03:13 AM 1,048,576 E0100016666.log 01/12/2023 03:22 AM 1,048,576 E0100016667.log 01/12/2023 03:22 AM 1,048,576 E0100016668.log 01/12/2023 05:23 PM 1,048,576 E0100016669.log 01/13/2023 03:01 AM 1,048,576 E010001666A.log 01/13/2023 03:02 AM 1,048,576 E010001666B.log 01/13/2023 03:12 AM 1,048,576 E010001666C.log 01/14/2023 03:01 AM 1,048,576 E010001666D.log 01/14/2023 03:01 AM 1,048,576 E010001666E.log 01/14/2023 03:16 AM 1,048,576 E010001666F.log 01/16/2023 03:01 AM 1,048,576 E0100016670.log 01/16/2023 03:01 AM 1,048,576 E0100016671.log 01/16/2023 03:27 AM 1,048,576 E0100016672.log 01/17/2023 03:02 AM 1,048,576 E0100016673.log 01/17/2023 03:10 AM 1,048,576 E0100016674.log 01/17/2023 03:11 AM 1,048,576 E0100016675.log 01/18/2023 03:01 AM 1,048,576 E0100016676.log 01/18/2023 03:02 AM 1,048,576 E0100016677.log 01/18/2023 03:18 AM 1,048,576 E0100016678.log 01/18/2023 08:52 AM 1,048,576 E0100016679.log 01/18/2023 08:53 AM 1,048,576 E010001667A.log 01/18/2023 08:55 AM 1,048,576 E010001667B.log 01/18/2023 09:23 AM 1,048,576 E010001667C.log 01/18/2023 09:23 AM 1,048,576 E01tmp.log
Confirm the shutdown state of the edb using "Eseutil" command.
C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1>eseutil /mh DB1.edb | findstr State
State: Dirty Shutdown <==== note the EDB shows as being in a dirty shutdown state because it was backed up using a VSS snapshot, this is fully expected unless the exchange database was idle at BU time with no outstanding logs to apply.
Change the directory to the log directory then using Eseutil to confirm that all required logs are available and in a good state prior to them being replayed against the edb file.
E01 in this example represents the log file prefix or base value, this varies especially when multiple Exchange edb DBs are contained within a given Exchange instance.
C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1>eseutil /ml E01
Verifying log files...
Base name: E01
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016666.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016667.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016668.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016669.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E010001666A.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E010001666B.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E010001666C.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E010001666D.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E010001666E.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E010001666F.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016670.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016671.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016672.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016673.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016674.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016675.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016676.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016677.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016678.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E0100016679.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E010001667A.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E010001667B.log - OK
Log file: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1\E010001667C.log - OK
No damaged log files were found. <============ verification step to ensure all logs are in a ready to use state
Operation completed successfully in 0.984 seconds.
Change the directory to the folder containing the edb file to be repaired.
Using "Eseutil," run the syntax as shown passed all on one line to replay the logs and update the edb file.
Options "/l <points to log path> /d <points to edb path>," enclose each path name within double quotes as various directory names may contain spaces.
If you specify an incorrect path names to Eseutil, it may result in the E01.chk file and possibly others being updated (compare to original file timestamps) preventing the logs from being successfully applied even after correcting the path names. It is advisable to make a copy of the folders involved to allow putting original files back in place quickly if this situation arises.
C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1> eseutil /r E01 /l "C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1" /d "C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1"
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 14.03 Initiating RECOVERY mode... Logfile base name: E01 Log files: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1 System files: <current directory> Database Directory: C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1 Performing soft recovery... <========= this is the step which will update the edb Restore Status (% complete) 0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ................................................... Operation completed successfully in 0.812 seconds.
After the logs have been successfully replayed, reverify the shutdown state of the edb, it should now be in a Clean Shutdown state.
C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1>eseutil /mh DB1.edb | findstr State
State: Clean Shutdown <========= note the edb now shows to be in a clean shutdown state after successfully replaying the logs.
C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1>dir Directory of C:\ADME3\exch01\BYDATE\srv2008-exch10.nicktest.local\2023-01-18-08-59-34-293\srv2008-exch10_00000000\C\Program Files\Microsoft\Exchange Server\V14\Mailbox\DB1
01/23/2023 04:03 PM <DIR> . 01/23/2023 04:03 PM <DIR> .. 01/23/2023 04:03 PM 142,671,872 DB1.edb <========note the edb & chk file timestamps have now been updated to Jan 23, 2023 from Jan 11, 2023 as documented at the start of this process ============================================================================================================================================
Test-2
This next example uses the Avamar UI to recover the Exchange backup using the Exchange plug-in but disabling replaying of the logs. It was recovered to folder name C:\ADAM4 and the same "Eseutil" command sequences used as described above in the Test-01 scenario.
c:\ADAM4>dir Directory of c:\ADAM4 01/23/2023 04:54 PM <DIR> . 01/23/2023 04:54 PM <DIR> .. 01/11/2023 05:22 PM 142,671,872 DB1.edb <============= note the edb file timestamp is from Jan 11, 2023 the time of the VSS Exch backup 01/23/2023 04:54 PM <DIR> _restoredLogs <============ The Avamar plugin recovers all logs to a common folder as shown
There has been at least one case where the consolidated common log folder for a backup containing x4 edb files resulting in the log prefix value overlapping and overwriting some log files which ultimately could not be applied successfully to the edb file. This was the fault of Avamar as Exchange maintains separate log directories for each edb and backup contents reflects the separate log folders. See Esc https://jira.cec.lab.emc.com/browse/AVAMARESC-271 for details and access to a plug-in binary update which recovers logs pertaining to each edb into their original separate folder names.
C:\ADAM4> eseutil -mh DB1.edb | findstr State
01/23/2023 04:03 PM 8,192 E01.chk 01/23/2023 04:03 PM 1,048,576 E01.log 01/12/2023 03:13 AM 1,048,576 E0100016666.log 01/12/2023 03:22 AM 1,048,576 E0100016667.log 01/12/2023 03:22 AM 1,048,576 E0100016668.log 01/12/2023 05:23 PM 1,048,576 E0100016669.log 01/13/2023 03:01 AM 1,048,576 E010001666A.log 01/13/2023 03:02 AM 1,048,576 E010001666B.log 01/13/2023 03:12 AM 1,048,576 E010001666C.log 01/14/2023 03:01 AM 1,048,576 E010001666D.log 01/14/2023 03:01 AM 1,048,576 E010001666E.log 01/14/2023 03:16 AM 1,048,576 E010001666F.log 01/16/2023 03:01 AM 1,048,576 E0100016670.log 01/16/2023 03:01 AM 1,048,576 E0100016671.log 01/16/2023 03:27 AM 1,048,576 E0100016672.log 01/17/2023 03:02 AM 1,048,576 E0100016673.log 01/17/2023 03:10 AM 1,048,576 E0100016674.log 01/17/2023 03:11 AM 1,048,576 E0100016675.log 01/18/2023 03:01 AM 1,048,576 E0100016676.log 01/18/2023 03:02 AM 1,048,576 E0100016677.log 01/18/2023 03:18 AM 1,048,576 E0100016678.log 01/18/2023 08:52 AM 1,048,576 E0100016679.log 01/18/2023 08:53 AM 1,048,576 E010001667A.log 01/18/2023 08:55 AM 1,048,576 E010001667B.log 01/18/2023 09:23 AM 1,048,576 E010001667C.log 01/23/2023 04:03 PM 1,048,576 E01res00001.jrs 01/23/2023 04:03 PM 1,048,576 E01res00002.jrs 01/23/2023 04:03 PM 1,048,576 E01tmp.log State: Dirty Shutdown <========= note the EDB shows as in a dirty shutdown state because it was backed up using a VSS snapshot, this is fully expected unless the exchg DB was totally idle at BU time.
C:\ADAM4\_restoredLogs>eseutil /ml E01
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Initiating FILE DUMP mode... Verifying log files... Base name: E01 Log file: c:\ADAM4\_restoredLogs\E0100016666.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016667.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016668.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016669.log - OK Log file: c:\ADAM4\_restoredLogs\E010001666A.log - OK Log file: c:\ADAM4\_restoredLogs\E010001666B.log - OK Log file: c:\ADAM4\_restoredLogs\E010001666C.log - OK Log file: c:\ADAM4\_restoredLogs\E010001666D.log - OK Log file: c:\ADAM4\_restoredLogs\E010001666E.log - OK Log file: c:\ADAM4\_restoredLogs\E010001666F.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016670.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016671.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016672.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016673.log - OK[1][2] Log file: c:\ADAM4\_restoredLogs\E0100016674.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016675.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016676.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016677.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016678.log - OK Log file: c:\ADAM4\_restoredLogs\E0100016679.log - OK Log file: c:\ADAM4\_restoredLogs\E010001667A.log - OK Log file: c:\ADAM4\_restoredLogs\E010001667B.log - OK Log file: c:\ADAM4\_restoredLogs\E010001667C.log - OK No damaged log files were found. <============ verification step to ensure all logs are in a ready to use state Operation completed successfully in 1.125 seconds.
Return to the folder containing the edb file.
Using "Eseutil" run the following syntax all on one line to replay the logs and update the edb file where option "/l <points to log path> /d <points to edb path>."
Enclose each path name within double quotes as various directory names may contain spaces.
c:\ADAM4>eseutil /r E01 /l "c:\ADAM4\_restoredLogs" /d "c:\ADAM4"
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 14.03 Initiating RECOVERY mode... Logfile base name: E01 Log files: c:\ADAM4\_restoredLogs System files: <current directory> Database Directory: c:\ADAM4 Performing soft recovery... <========= this is the step which will update the edb Restore Status (% complete) 0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ................................................... Operation completed successfully in 0.515 seconds.
c:\ADAM4>dir Directory of c:\ADAM4 01/23/2023 05:35 PM <DIR> . 01/23/2023 05:35 PM <DIR> .. 01/23/2023 05:35 PM 142,671,872 DB1.edb <==note the edb & chk file timestamps have now been updated to Jan 23, 2023 from Jan 11, 2023 as documented at the start of this process 01/23/2023 05:35 PM 8,192 E01.chk
C:\ADAM4>eseutil /mh DB1.edb | findstr State
State: Clean Shutdown <========= note the edb now shows to be in a clean shutdown.
Additional Information
Non-Dell Links
The following links are provided solely for reference purposes. There are numerous links available on the Internet discussing the use of Eseutil to repair an edb file.
A link documenting the use of the /p option which is a hard repair is available as a last resort to repair an edb.
https://social.technet.microsoft.com/wiki/contents/articles/52087.exchange-extensible-storage-engine-utility-eseutil.aspx
There are also professional tools available to repair edb files similar to the one at this lin.
https://www.systoolsgroup.com/updates/exchange-dirty-shutdown-state/