Avamar: How to Apply the "Avamar troubleshooting hierarchy" Approach Correctly

• While reviewing the proactive check results and when troubleshooting multiple Avamar issues, understanding how the various issues and failures affect one another.
• The resolution to many issues depends on underlying operations completing successfully.

• If checkpoint validation (hfscheck) is failing, the checkpoint overhead building up on the system increases the operating system capacity utilization.
• This increase in operating system capacity utilization causes garbage collection to fail once the operating system capacity utilization reaches the disknogc limit.
• The garbage collection failure, in turn, leads to high GSAN capacity utilization and sooner or later, the system goes into admin mode.
• The underlying hfscheck issue must be resolved first to free up enough operating system capacity to allow garbage collection to run before resolving the issue with GSAN capacity utilization.

• If there are multiple issues, they must be worked through and resolved in the order listed below in order (and return the Avamar grid back into a healthy state).
• Correcting each issue in this list requires that all the issues above it be resolved first.

• Critical hardware failures
• Stripes or nodes offline or suspended
• Checkpoint failures
• hfscheck failures
• Operating system capacity issues (high fs-percent-full, freespaceunbalance issues or stripe pool exhaustion)
• Garbage collection failing or failing to run
• High GSAN capacity utilization
• Capacity-related backup failures (For example, where the server has reached the diskreadonly threshold)

Once the above hierarchy has been satisfied, additional considerations may be present (for example, a long-running hfscheck may cause operational issues on the system) but it is always critical to move down through this hierarchy first in order to ensure that a sick grid becomes a healthy grid as quickly as possible.

Here is a graphical representation of which type of issue must be addressed first based on the Avamar priority, starting from the bottom and working upwards:

Related articles:

• Avamar: How to Log in to an Avamar Server and Load Various Keys.
• Avamar: How to Run the proactive_check.pl health check script on an Avamar Server