Avamar: VMware Image Backups Stuck in Waiting Client or Queued State

Observed Backup State Issues

All VMware image backups on Avamar remain in the following states:

• Waiting – Client
• Waiting – Queued

The backups do not progress and eventually terminate with a Timed Out – End error.

Relevant Log Entries

avagent Info <5964>: Requesting work from [Avamar Server IP]
avagent Info <5264>: Workorder received: sleep
avagent Info <5996>: Sleeping 15 seconds
avagent Info <40019>: Checking certificate status.
avagent Error <5365>: Cannot connect to [Avamar Server IP]:28001.
avagent Info <5059>: unable to connect, sleep(60) then retrying
avagent Error <5365>: Cannot connect to [Avamar Server IP]:28001.
avagent Info <5059>: unable to connect, sleep(60) then retrying
avagent Error <5365>: Cannot connect to [Avamar Server IP]:28001.
avagent Info <5059>: unable to connect, sleep(60) then retrying
avagent Error <5365>: Cannot connect to [Avamar Server IP]:28001.
avagent Info <5059>: unable to connect, sleep(60) then retrying
avagent Info <40022>: The certificates for '[proxy hostname]' have become invalid and need to be updated.
avagent Info <40023>: '[proxy hostname]' will reregister with the Management Console to update certificates.
avagent Info <5435>: Exiting work request session
avagent Info <19036>: Client will reregister with Management Console (Avamar Server IP), in 39 minutes, 45 seconds
avagent Info <6626>: Agent Main: Client ID (cid) = [Proxy CID]
avagent Info <18918>: Registration: Processing secure registration with the MCS.
avagent Info <18921>: Registration: Requesting root CA from the MCS.
avagent Info <18926>: Registration: Saving root CA.
avagent Info <18928>: Registration: Creating certificate signing request.
avagent Info <18930>: Registration: Sending the certificate signing request to the MCS.
avagent Error <40012>: Avamar server certificate verification error 7 at depth 0: certificate signature failure
avagent Error <5365>: Cannot connect to [Avamar Server IP]:30003.
avagent Info <5059>: unable to connect, sleep(60) then retrying

Invalid SSL/TLS Certificates After Regeneration

The Avamar server regenerated its SSL/TLS certificates. Client proxies retained the previous certificates, causing a mismatch that prevented secure communication.

Key error messages observed in avagent.log 

avagent Info <40022>: The certificates for '[proxy hostname]' have become invalid and need to be updated.
avagent Info <40023>: '[proxy hostname]' will reregister with the Management Console to update certificates.
avagent Info <5435>: Exiting work request session
avagent Info <19036>: Client will reregister with Management Console (Avamar Server IP), in 39 minutes, 45 seconds
avagent Info <6626>: Agent Main: Client ID (cid) = [Proxy CID]
avagent Info <18918>: Registration: Processing secure registration with the MCS.
avagent Info <18921>: Registration: Requesting root CA from the MCS.
avagent Info <18926>: Registration: Saving root CA.
avagent Info <18928>: Registration: Creating certificate signing request.
avagent Info <18930>: Registration: Sending the certificate signing request to the MCS.
avagent Error <40012>: Avamar server certificate verification error 7 at depth 0: certificate signature failure
avagent Error <5365>: Cannot connect to [Avamar Server IP]:30003.
avagent Info <5059>: unable to connect, sleep(60) then retrying

Restart the Avamar Agent Service on Affected Proxy Appliances

Procedure

1. Log in to the proxy appliance. Use SSH or direct console access with a user that has root privileges.

   $ ssh root@[PROXY_HOSTNAME_OR_IP]
   Password:

2. Restart the Avamar agent service. This forces the client to reregister with the Management Console and obtain fresh SSL/TLS certificates.

   $ service avagent restart

3. Confirm the service restarted successfully.

   $ service avagent status