Avamar: Checkpoints and HFScheck are skipped due to the waittime not being set

22409 ERROR A checkpoint validation (hfscheck) of Avamar Server checkpoint data is overdue.

summary="A checkpoint validation (hfscheck) of Avamar Server checkpoint data is overdue."
type="ERROR"
code="22409"

Symptom: 22409, Desc: A checkpoint validation (hfscheck) of server checkpoint data is overdue

• Checkpoints and Checkpoint validation (HFSCheck) do not run intermittently.

This can be verified by running the following from the Avamar Utility Node:

dumpmaintlogs --types=hfscheck,cp,gc --days=30 | grep '<4' | sort

Sample output:

2014/09/30-15:01:02.71102 {0.0} <4200> starting scheduled garbage collection
2014/09/30-18:58:36.88121 {0.0} <4201> completed garbage collection
2014/10/01-15:01:00.64655 {0.0} <4200> starting scheduled garbage collection
<...>
2014/10/12-15:00:58.45799 {0.0} <4200> starting scheduled garbage collection
2014/10/12-15:02:27.62869 {0.0} <4201> completed garbage collection
2014/10/13-15:01:04.91870 {0.0} <4200> starting scheduled garbage collection
<...>
2014/10/16-15:01:00.49086 {0.0} <4200> starting scheduled garbage collection
2014/10/16-15:02:28.64102 {0.0} <4201> completed garbage collection
2014/10/17-15:01:00.71545 {0.0} <4200> starting scheduled garbage collection
<...>

• The Waittime parameter is not set to the default time of 1 hour:

This can be verified by running the following from the Avamar Utility Node, as admin with admin keys loaded. (See Avamar: How to Log in to an Avamar Server and Load Various Keys for instructions on loading keys).

mapall --noerror --parallel 'grep -h "crunchwaittask::body waittime=" /data01/cur/gsan.log*' | sort

Sample output:

2014/10/12-15:02:27.63377 {0.0} [sched.crunchwait:4512]  crunchwaittask::body waittime=11:57:33.000000
2014/10/13-15:02:23.17292 {0.0} [sched.crunchwait:3093]  crunchwaittask::body waittime=00:00:00.000000
2014/10/14-15:02:27.37016 {0.0} [sched.crunchwait:5625]  crunchwaittask::body waittime=00:00:00.000000

• The Waittime parameter is not set properly on crunch.wait:

This can be verified by running the following from the Avamar Utility Node:

avmaint sched crunchwait --ava

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <task-param-request   reset="false"   task-type="crunchwait">   <crunchwait permanent="true"/>

The "crunchwait" parameter "waittime" is not set to the default of 60 minutes. 

1. Log in to the Avamar Utility Node as admin and load the admin keys. See Avamar: How to Log in to an Avamar Server and Load Various Keys for instructions on loading keys.

2. Set the "waittime" to 60: 

avmaint sched crunchwait --waittime=60 --ava --permanent

3. Verify that the waittime is now present:

avmaint sched crunchwait --ava

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<task-params>
  <crunchwait permanent="true">
    <param
      name="waittime"
      value="60"/>
  </crunchwait>
</task-params>

4. Within the next day or two, after maintenance has finished, run the following command again:

mapall --noerror --parallel 'grep -h "crunchwaittask::body waittime=" /data01/cur/gsan.log*' | sort

The sample output should be similar to the following:

2014/10/14-15:02:27.37016 {0.0} [sched.crunchwait:5625]  crunchwaittask::body waittime=00:00:00.000000
2014/10/15-15:02:24.63377 {0.0} [sched.crunchwait:4512]  crunchwaittask::body waittime=01:00:00.000000
2014/10/16-15:03:01.17292 {0.0} [sched.crunchwait:3093]  crunchwaittask::body waittime=01:00:00.000000

5. Monitor the grid over the coming days and verify that checkpoints and hfscheck are no longer being skipped.