Solved!
Go to SolutionAvamar/Data Domain storage utilization
Hi All,
We have two Avamar nodes (each with an attached Data Domain). Each node performs backups, and each node replicates all data to the other data domain (bi-directional replication). We've also configured all replicated data to keep its existing expiration date. In this scenario, I would expect the data domain storage utilization to be nearly the same on each side since each DD will be storing the same backup data and replicated data. For quite a while - about a year and a half - this was true. Each data domain held steady and was within a percentage or two from the other. Just recently though, this has not been true. One of the data domains is at about 50% utilization while the other has rapidly increased to over 70% utlized.
How can we identify the disparity between the two data domains? Am I flawed to think that the utilization of each should be about the same? From what I can see, all data is being replicated without any problems..
Any insight is much appreciated!
Avamar version 7.0.2-47
Data Domain 5.4.0.4
Thanks,
Josh
1935
1
Posted March 14th, 2015 06:00
We're doing uni-directional replication (PROD -> DR only), but we have a 5% to 10% delta during the week, until the weekly Data Domain cleaning operation runs on both sides, getting them back to within 5% of each other. In your case, the 20% delta doesn't sound too out-of-line, unless the utilization on each DataDomain doesn't get closer after each cleaning.
Can you check the GUI to see if you've missed a cleaning? Back on 5.1, we had an issue where one side would "miss" the cleaning operation start. The cleaning start message appeared in the log, but the filesystem never performed the operation and it was not visible in the GUI.
If you haven't missed a cleaning, I would open a ticket to investigate. There's a small chance you have a data set that has poor commonality on one side, but I find it hard to believe it would suddenly "happen" - you should have seen that behavior all along.
Let us know what you find!
Karl
1935
0
Posted March 16th, 2015 08:00
Hi Karl,
Thanks for your insight. After checking things in the DD GUI, it appears that cleaning at least started, but seems stuck at 'Phase 1 of 10 (pre-enumeration)'. Total time is over 1 week.
Have you, or anyone, heard of the pre-enumeration phase taking such a long time? I haven't been regularly checking the cleaning cycle on the DD's, so I'm not sure if this is to be unexpected, or not.
Thanks,
Josh
1935
0
Posted March 16th, 2015 09:00
Well, it would be good to check when was the last time, that particular Data Domain (having cleaning issue) was last rebooted.
Also, recommend you to check the autosupport emails from the same system for any issues.
If still you have trouble figuring it out, better open up a support case with EMC than to take chances.
jpanei
10 Posts
1935
2
Posted March 16th, 2015 11:00
I just worked with EMC support on the stuck cleaning cycle.. Turns out that the version DDOS we're using (5.4.0.4) has a known bug that causes cleaning to get stuck on the first phase.
As a temporary work around, we disabled the 'virtual-synthetics' option from the CLI and we immediately saw cleaning get past the phase which it was previously stuck at.
To check the virtual-synthetics option:
To disable the virtual-synthetics option:
Long term and permanent fix is to upgrade to the most recent version of DDOS.
Thanks for everyone's help!
Josh.