Avamar: Symptom Code 22409: A checkpoint validation (hfscheck) of server checkpoint data is overdue (Resolution Path)
Summary: This resolution path article addresses the following: Symptom Code 22409 - A checkpoint validation (hfscheck) of server checkpoint data is overdue.
Symptoms
This event may be generated even when a checkpoint validation (hfscheck) has been completing successfully every day.
The error can be seen in the logs (gsan.log, err.log), a dialhome, or in the Avamar UI:
22409 ERROR A checkpoint validation (hfscheck) of Avamar Server checkpoint data is overdue.
summary="A checkpoint validation (hfscheck) of Avamar Server checkpoint data is overdue."
type="ERROR"
code="22409"
Symptom: 22409, Desc: A checkpoint validation (hfscheck) of server checkpoint data is overdue
Cause
There are many factors to consider when troubleshooting checkpoint validation (hfscheck) overdue failures on an Avamar grid.
An overdue hfscheck is usually a symptom of another problem (such as configuration files).
This article helps to lead the reader through some of the most common hfscheck problems, and how to determine when (and why) the failure occurred.
Resolution
Validate that each troubleshooting step below is true for the environment.
Each step provides instructions or a link to a document, to eliminate possible causes and take corrective action as necessary.
The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution.
Do not skip a step.
Step 1: Log in to the Avamar Utility Node as admin, and load the admin keys. See Avamar: How to Log in to an Avamar Server and Load Various Keys for instructions on loading keys.
Note: The rest of this KB article assumes that the admin keys are loaded.
Step 2: Before taking any corrective action, run the health check script (proactive_check.pl) on the grid.
Make note of any errors detected as they may be directly related to the hfscheck failure.
Instructions for downloading and running this script can be found here: Avamar: How to run the proactive_check.pl health check script on an Avamar Server.
Note: Scheduled events (such as hotfix and patch updates) can be addressed later.
Step 3: Check if an incorrect MCUser password in one or more of the mccli configuration files has been used per Avamar: Symptom Code 22409 - A checkpoint validation (hfscheck) overdue Due to MCUser Password Inconsistency in mccli Files
Step 4: Check to see if maintenance is disabled on the Avamar Server per Avamar: Symptom Code 22409 - A checkpoint validation (hfscheck) is overdue due to a disabled maintenance scheduler.
Step 5: Check the permissions of mcserver.xml files per Avamar: Symptom Code 22409 - A checkpoint validation (hfscheck) is overdue due to Incorrect mcserver.xml permissions
Step 6: Check to see if there is an unresponsive, or problem avtar process on the utility node per Avamar: Error Code 22409: A Checkpoint Validation (hfscheck) is Overdue
Step 7: Check if the Management Console Server (MCS) is processing too many files in the error log per Avamar: Symptom Code 22409 - A checkpoint validation (hfscheck) is overdue due to too many messages in err.log
Note: The MCS can be delayed in parsing hfscheck completion messages causing an overdue message.
Step 8: Verify that the waittime flag is not set to 60 per Avamar: Checkpoints and HFScheck are skipped due to the "waittime" not being set
If the problem still exists after trying the steps in this article, contact Create a Service Request for further assistance.