This post is more than 5 years old
5 Posts
0
14465
Garbage collection does not reclaim expected amount of space
Hi all,
I need help with space management on the Avamar grid. I added a number of servers without the proper filters and the grid filled up. I corrected the filters, and last night I deleted 3.5 TB (yes, TB) of backups, but this morning the garbage collection (GC) only recovered 300 GB! Why didn't GC do a better job and how can I reclaim that space?
Thanks,
Beth
J_H_
498 Posts
1
August 6th, 2013 10:00
I recently learned about this so will share what helped me.
it seemed my GC was just not getting much in its run. So to see your history of GC run this command at cli
dumpmaintlogs --types=gc --days=120 |grep passes| cut -d ' ' -f1,12,13
change the # of days to something where you can get a good look at trending - you will get something like this
2013/07/26-18:39:08.78474 megabytes-recovered="210029" passes="29"
2013/07/27-19:14:01.78604 megabytes-recovered="247776" passes="36"
2013/07/29-18:27:55.22241 megabytes-recovered="208956" passes="28"
2013/07/30-19:34:16.81329 megabytes-recovered="245157" passes="35"
2013/07/31-18:44:48.81560 megabytes-recovered="301888" passes="31"
2013/08/01-19:06:22.58182 megabytes-recovered="246840" passes="37"
2013/08/02-18:13:29.97334 megabytes-recovered="136612" passes="29"
2013/08/03-19:09:45.53327 megabytes-recovered="325953" passes="31"
from here you can see just how much it is getting and how many passes -
my issue was that I could go back and see that I use to get 50 or more passes - but was down to only getting 1 pass a day (and not even a full pass)
the way it was explained to me is that of the GC window it will use 2/3eds getting info if it cannot complete a pass in 2/3eds fo the time it will then use the remaing 1/3 to clean what it did find. If it can complete a pass it will then clean for that pass and start another pass. So I had to increase my window for GC to run. And it slowly got more and more passes in and more and more space recovered. you also see when it finished (stopped).
You can see that I am now up to around 30 a day and getting lots of space back (when I first creased my time I was getting 1 tb a day cleaned out)
so as suggeted by others increase your time and use the command to see if it is doing better then it was before.
christopher_ime
2K Posts
1
July 19th, 2013 22:00
Beth,
Please consider moving this question as-is (no need to recreate) to the proper forum for maximum visibility. Questions written to the users' own "Discussions" space don't get the same amount of attention and questions can go unanswered for a long time.
You can do so by selecting "Move" under ACTIONS along the upper-right. Then search for and select: "Avamar Support Forum" which would be the most relevant for this question.
Avamar Exorcist
462 Posts
1
July 23rd, 2013 07:00
Garbage collection may take a number of days to completely remove a large amount (Terabytes) of data. You can help it along by temporarily increasing the length of the blackout maintenance window.
You should expect to see utilisation trending down over the coming week whilst it chips away at the expired data chunks.
You can read more about how GC works in the following KB article
https://support.emc.com/kb/esg129640 - Avamar space reclamation processes - Part 1: Garbage Collection
Nayak2010
50 Posts
1
July 23rd, 2013 08:00
Since Avamar is a deduplication solution it would be unfair to expect to see a difference of 3.5 TB difference in usage when you delete backup worth that much. Also! if you've deleted file system backups then the change will be even low. Also! as nr suggested this might take a few days to really show the difference. You might want to check if GC is utilizing the time alotted to it and if there is any need to increase the GC window.
Beth_
5 Posts
0
July 23rd, 2013 10:00
Thanks nr & Nayak2010 for your replies. I also found this and I learned that partial backups are automatically deleted after 7 days, so now I will increase the GC window and be patient.
fdxpilot
143 Posts
1
July 23rd, 2013 21:00
FYI -- that link appears to be internal only, like all too many of the useful Avamar KB articles.
ionthegeek
2K Posts
1
August 6th, 2013 13:00
Just a heads up that the partials behaviour changed in Avamar 6.1 so that all partial backups in these versions are assigned the retention value that would have been assigned to the completed backup. This change was made due to an issue where applications that manage their on backup catalog (e.g. Oracle RMAN) were having issues restoring expired backup pieces (since they are stored as partials) even though the full backup was still available on the system. I was not aware of this change during the Ask the Expert session as the escalation where it came to light was filed almost a year later.
Unfortunately, this means that if there are filesystem backups that were canceled or plug-in backups that have been deleted, the partials may still be hanging around. If you suspect this is the case, I would recommend working with support to clean up these stale partials. If you know which clients might be affected and you're comfortable at the Avamar command line, you can get a complete list of backups (including partial backups) from the command line using syntax like the following:
As always, this command should be run from the utility node while logged in as the admin user. Please note that the avmgr command is incredibly powerful and potentially very dangerous so you should only run it on advice from support. The various "get" commands are safe but anything else could cause serious problems on the system.
Handling for these "catalog" partials was improved in the Avamar 7 client so partial backup retention was returned to the previous default of 7 days for the Avamar 7 release. Note that it is the client software that is responsible for setting expiration so the behaviour is dependent on the client version, not the server version.