Exchange 2007 DBs inconsistent in Recoverpoint

Can you tell me a bit more about your environment?

What version of RecoverPoint are you using?

What Splitter type are you using?

What array is in use?
Are you attempting to mount a crash consistent image or an application consistent image (i.e one you quited the system)?

Hi Rick, I have a CX4-480 at the production site using the array splitter and a CX700 using the kdriver at the DR site. We are using stretch CDP as the sites are connected thru dark fiber. Recoverpoint version 3.2.0. I need an application consistant image as they DBs are coming up dirty shutdown when I try to mount any images.I did notice I didn't install KVSS on the production cluster so I want ahead and did that on the passive side. I am waiting for a failover to patch the other side.

Thanks

The problem with all that is we have Recoverpoint. I would like the ability to recover any image in the journal by enabling image access on the DR side. This is as advertised by EMC and worked just fine in 2003.  I may have to open a ticket but you learn more when things are broken

That is correct. You can also run an eseutil.exe to determine if the Transaction logs and databases have torn pages, etc. In most cases the Exchange databases will mount even though they are marked “Dirty Shutdown”. That really means that the checkpoints were not written and the transactions in the Transaction logs have not been flushed to the database. If the database mounts, the transactions should either roll forward or back during the database mount.

By running the eseutil.exe you can find out if there are unrecoverable errors. However, on a large Exchange database, running eseutil.exe can take hours. You might want to run that once while the production node is up and healthy just to assure yourself that the mirrored copy is recoverable. The best thing is to also do VSS snapshots which place the database and logs into a consistent state and then make the snapshot. That gives you a MS approved point in time restore point. This protects against regular failures but also you can roll back to a consistent state if there is email corruption. Depending on how many and how often you create snapshots you can take up some additional drive space but it would help ensure a short RTO.

Just my 2 cents worth…js

We did not have this issue with the Exchange 2003 system it replaced. I was able to mount any image from any time stamp in the journal and it came up clean and worked fine. That's why I went back to check the install and noticed the KVSS wasn't installed on the production cluster nodes. That might have something to do with it

Does esutils run clean against the source site.. i.e are you sure your source is ok? 

Yes, we are in full production with no problems. Well as far as the Exchange Team knows I should add

drozz,

When you were running RP with Exchange 2003 did the Exchange server at the replica site see the DB as clean? I would have expected Exchange 2003 to have to recover the DB as well as Exchange 2010 since using RP without any host side application quiesce you will get write order consistency on the LUNs and not a clean application consistent data set.

I use RP to replicate about 8TB of SQL databases and the data is crash consistent as expected since RP is just ensuring write order consistency, but that has never been a problem for SQL to recover the database using the log files.

Aran

Yes, they came up clean every time no matter how far back or how current the image was. The only difference between the systems is the Exchange team decided to use mount points in 2007 instead of drive letters.

The problem with running eseutil.exe against the production node is that it would have to be done with Exchange offline, or at least the database dismounted. If you take the database offline that implies that the database and transaction logs are shut down normally and the checkpoints would be consistent. That itself would make eseutil indicate no “Dirty Shutdown” had occurred. To simulate what a crash consistent state (a Dirty Shutdown) would show on the Source would require crashing the Information Store process. That is not a good idea and I would never temp fate by suggesting that.

There is really no simple way to guarantee the replica is error free which is “not” the same a Dirty Shutdown state. If that is the end goal then a planned outage would be required. Basically you would need to stop the replication by doing something like pulling the plug on the splitter and then dismounting the database on the Source node. That would preserve the data on the Source node. Then mount the replica database on the Target node. At that point you would see if 1) the database mounts successfully and 2) how many messages were lost due to rollbacks of transactions in the transaction log.

This is more of a brute force test of things and not for the faint of heart. It really only answers one question; does the replica mount? There will almost certainly be some lost email because some transactions were in an uncommitted state and would be considered incomplete and are discarded, or rolled back. Of course the email will still be preserved on the Source machine and when you dismount the database from the replica server and mount it on the Source server, they will be there. How many might be lost? That is entirely a matter of chance and depends on how many uncommitted transactions were in the transaction log when the connection was severed. On a slow email day maybe none; on a really busy day, well…probably some would be lost.

I stand corrected. J

I shouldn't need to take down the system to test this. I have a DR site that I mount the replicas to. When I enable image access in Recoverpoint, all volumes mount with no problem and have good time stamps. The error hits when we try to mount the exchange DBs. There are 13 volumes, each with Dbs on them. SG1 is the only one that mounts everytime. The other 12 have not been consistant. I'm sure I could run esutil against them and get them to come up. This would defeat the purpose of Recoverpoint all together and I could go back to Replication Manager which has a "freeze" and "thaw" while it commits to the target LUNs. I have never needed anything further in Recoverpoint other than to "mount and go" which is the beauty of the appliance.

Yes that's correct, all 26 DB and Log LUNs plus 2 mount point LUNs

You know I am surprised that when replicating with Exchange 2003 you were able to bring the databases online without it having to perform recovery on the database. Just replicating the LUNs via RP the data will always be crash consistent not application consistent. Maybe Exchange 2010 is more sensitive?