Avamar: Backup Fails Unable To Load Hash Cache

An Avamar backup fails immediately after launching with the following error messages reported in the logs.
 

2017-03-18 18:11:06 avtar Error <5064>: Cannot open file '/usr/local/avamar/var/EMC/root_vdm_1_EMC_p_cache.dat'
2017-03-18 18:11:06 avtar Error <18907>: Unable to load hash cache file '/usr/local/avamar/var/EMC/root_vdm_1_EMC_p_cache.dat' (Log #1)
2017-03-18 18:11:06 avtar FATAL <5224>: Cannot run multiple backups with the same hash cache name at the same time. Consider changing VARDIR or using --cache-prefix option.

Root Cause of the Issue

The Avamar backup failure can be attributed to two primary causes:

• A misconfiguration of the 'vardir' or '--cache-prefix' parameters, resulting in multiple avtar instances attempting to access the same cache file simultaneously.
• If the avtar process stops responding, which can occur due to an old backup or avtar process being stuck on the Linux client or Network Data Management Protocol (NDMP) accelerator node, causing the cache to be locked.

The error messages in the logs below show the cause of the failures.

2017-03-18 18:11:06 avtar Error <5064>: Cannot open file '/usr/local/avamar/var/EMC/root_vdm_1_EMC_p_cache.dat'
2017-03-18 18:11:06 avtar Error <18907>: Unable to load hash cache file '/usr/local/avamar/var/EMC/root_vdm_1_EMC_p_cache.dat'
2017-03-18 18:11:06 avtar FATAL <5224>: Cannot run multiple backups with the same hash cache name at the same time. Consider changing VARDIR or using --cache-prefix option.

Resolving the Avamar Backup Failure

To resolve the Avamar backup failure, follow these steps:

For Cause #1,

• A misconfiguration of the 'vardir' or '--cache-prefix' parameters
  • Check if the client is pointing at the same cache file as another in progress client backup. If true
  • Check the configuration of the backup:  VARDIR or cache-prefix parameters
• Check in the advanced options of the dataset and the client avtar.cmd file to ensure that the client is not configured so that multiple avtar processes do not point to the same vardir/cache files.

For Cause #2, an old backup/avtar process stopped responding on the Linux client or NDMP accelerator node, causing the cache to be locked. To check this, follow these steps:

• Log in into the system as root.
• Stop the avagent service to ensure no further backups attempt to launch

  service avagent stop
      

• Check if any avtar processes are running:

  ps -ef | grep av

  • If there are backups running, stop them from the Avamar Administrator UI. Alternatively, use

    kill 

  • Or

    kill -9 

  • To force, close the process. 
• When all old processes are cleared, restart avagent:

  service avagent start

• Test the backup again.