Avamar: Failed garbage collection with error MSG_ERR_BADTIMESYNC

The Avamar garbage collection (GC) process fails with: 

ERROR: <4202> failed garbage collection with error MSG_ERR_BADTIMESYNC

Time is not synchronized between one or more Avamar data nodes and the Avamar Utility Node.

This causes Avamar maintenance activities to fail.

1. Log in to Avamar Utility Node.

2. Load the admin keys. See Avamar: How to Log in to an Avamar Server and Load Various Keys for more information about loading keys.

3. Check that the time is synchronized across all Avamar nodes: 

mapall --all --parallel 'date'

Sample output where all node times are synchronized:

Using /usr/local/avamar/var/probe.xml
(0.s) ssh  -x  -o GSSAPIAuthentication=no admin@192.168.255.1 'date'
(0.0) ssh  -x  -o GSSAPIAuthentication=no admin@192.168.255.2 'date'
(0.1) ssh  -x  -o GSSAPIAuthentication=no admin@192.168.255.3 'date'
(0.2) ssh  -x  -o GSSAPIAuthentication=no admin@192.168.255.4 'date'
Sun May 12 20:26:25 UTC 2024
Sun May 12 20:26:25 UTC 2024
Sun May 12 20:26:25 UTC 2024
Sun May 12 20:26:25 UTC 2024

4. If the time is not synchronized, follow the Avamar: How to Change NTP on an Avamar Server Using asktime to synchronize time across all nodes.

5. Once the time is synchronized, verify that the garbage collection completes successfully.