This post is more than 5 years old
2 Posts
0
2217
VNX Checkpoint inactive
We are trying to recover critical data from a checkpoint. The problem is that the checkpoint we need is marked as inactive for some reason. Checkpoint space is not full, the checkpoint is less than a week old, I cannot figure out why it was marked as state inactive. More than that I cannot figure out how to reactivate it. If we cannot reactivate this checkpoint we are looking at elected official data loss (not good). I've paused the checkpoint schedule to keep it from being overwritten but it's no good sitting there inactive. Support said they couldn't do it but we've had very hit and miss results through support.
SO... does anyone know how to take a checkpoint marked inactive and make it active?
Why not restore from backup you ask? We tried and discovered that the NDMP backup in Avamar was skipping almost half of our home drives with no notice whatsoever (including the one we need of course). Normally we recover from checkpoints when a user screws up a document (which I guess is why we've not noticed). Since Avamar was a complete fail - can't restore what was never backed up - we're back to trying to get at the inactive checkpoint.
Thanks,
Chris (desperately seeking data)
umichklewis
1.2K Posts
0
October 18th, 2015 07:00
If a checkpoint fills the SavVol, the VNX will attempt to expire old checkpoints to keep the newer checkpoints active. It's possible you ran out of SavVol space, and this checkpoint was expired. This is probably why you see your SavVol is not full, but the checkpoint is inactive. Like Rainer said, though, inactive checkpoints are history.
Sorry to hear about Avamar skipping vital data. Avamar's NDMP restore interface is slow and time-consuming (not to mention, not searchable), but have checked every possible backup? We've found Java errors preventing the Avamar GUI from giving us the full file list. Also, don't forget the GUI will only show 10000 entries: your home directories might actually be getting backed up, but the GUI is showing them. Have you tried listing entries via the GUI?
dynamox
2 Intern
2 Intern
•
20.4K Posts
0
October 16th, 2015 19:00
how far up the chain in EMC support did you go ? Since this is a potential data loss situation, you should be able to escalate to engineering.
Rainer_EMC
8.6K Posts
1
October 18th, 2015 04:00
I think you are out of luck – an inactive checkpoint is gone.
You must have been running out of checkpoint space.
The data mover logs should indicate that – unless they have rolled over.
ChrisFricke
2 Posts
0
October 19th, 2015 10:00
Thanks for the input guys... I've adjusted the checkpoint schedule to reduce the likelihood of the checkpoint space getting filled (assuming that's what happened). In the meantime we found that the home drives were being backed up but not displaying correctly in the GUI as Karl suggested. Data recovered... YAY for that! But now there's 1 more irritating issue with Avamar (in a looong line). Oh if we'd only known how bad it really was... testament to the strength of the EMC sales team, I guess.
Anyways - crisis averted and I learned something. THANKS!
Rainer_EMC
8.6K Posts
0
October 19th, 2015 14:00
Nice work Karl!
In regards to checkpoints/snapshots that’s the way the technology works – only the up-to-date combination of the live data and the deltas together provide a consistent image.
Once a storage system gets a new write and doesn’t have space to store the old value it only has two choices – deny the write or invalidate the snapshot.
Denying the write will make all the live users unhappy.
umichklewis
1.2K Posts
0
October 21st, 2015 07:00
Glad to hear it! Check with your EMC data protection sales team, but there are some product announcements coming for Avamar 7.2 + BRM 1.x that should give all Java issues a final fix. Supposedly, there's a version of BRM coming that can let you select files for restoration from NDMP backups on Avamar, using the native interface and not the Java Avamar GUI. It's supposed to be a lot faster and more reliable!