Avamar: Avamar Garbage Collection reports "skipped-hashes" that cannot be cleaned up.

Avamar Garbage Collection reports several skipped hashes:

dumpmaintlogs --types=gc --days=5 | grep passes | cut -d ' ' -f1,10.14,15,17  

2021/02/11-11:03:23.01310 skipped-hashes="3118" megabytes-recovered="120" passes="10" elapsed-time="134"
2021/02/12-23:30:13.07016 skipped-hashes="96510" megabytes-recovered="604" passes="21" elapsed-time="210"
2021/02/12-23:34:15.11324 skipped-hashes="96511" megabytes-recovered="1" passes="3" elapsed-time="99"
2021/02/13-13:41:02.20624 skipped-hashes="97301" megabytes-recovered="0" passes="1" elapsed-time="81"
2021/02/14-23:58:47.70116 skipped-hashes="102294" megabytes-recovered="1" passes="4" elapsed-time="110"
2021/02/15-11:02:29.45054 skipped-hashes="102538" megabytes-recovered="0" passes="1" elapsed-time="82"

History:

The Hash Referenced Bit Maps feature was that introduced with Avamar v7.x enabling backups to occur while the garbage collection (GC) maintenance activity was in progress. 

Prior to Avamar v7.x, GC and backups could not run simultaneously due to the possibility of data conflicts. 

Overview:

During the garbage collection phase, this feature maintains information in memory about data which is added or changed (a "map of referenced hashes"). Garbage collection checks this information to know which hashes (and the data they reference) should not be removed.  

This feature requires that these "maps" have a minimum of 5 minutes "quiet" time, during while no backups are run so that they can reset. Once this reset occurs, the data that was locked can then be scanned during the subsequent garbage collection cycle if they remain unchanged.

This quiet time should be considered when designing the daily Avamar backup and maintenance schedule. 

Cause:

Although backups have expired, garbage collection cannot recover all the data eligible for deletion if some of the data was in use at the time. 

Short-term solution:

1. Ensure that the maps can reset and allow garbage collection to re-run.

2. Ensure that there are no running avtar sessions adding data to the system (backup or incoming replication data). Use the UI Activity Monitor and check for hung sessions seen only with the "avmaint sessions --full" command.

3. Stop all backups and incoming replication sessions.

4. Wait at least five minutes to allow the Hash Referenced Bit Maps enough time to reset.  

5. When GC runs again, check the GC maintenance log to confirm that the skipped-hashes is now zero.

If the skipped-hashes have not reset to zero, work with the Dell Technologies Avamar Support team to assist.

Long-term solution:

Design the backup and maintenance schedules so that there is a regular "idle time" available for the hash referenced bit map to be reset.

In other words, the backups should not be running 24/7. Build schedules that give a short amount of time when no backups or incoming replication data is being written to the system. 

• The only time the hash-referenced bit maps will check if they can be reset is right after an avtar session terminates.
• When no avtar sessions are in progress, the map would only reset:
  • If Index stripes are not in the process of splitting
  • If Garbage collection is not running, as the map cannot be locked while GC is running.
  • If no other avtar (backup, restore, or replication) sessions are running on the Avamar server. 
•  One reason that the skipped hashes count can be temporarily high is due to index stripe splitting.
  • This occurs because the hashes that are moved to the split target are protected.
  • Index stripe splitting occurs on systems that are still growing or being filled with data. 
  • Garbage collection may also fail with MSG_ERR_TRYAGAINLATER when index stripes are splitting.
• The Management Console Server (MCS) may not necessarily be aware of all avtar sessions that occur on the Avamar Server.