Avamar: NDMP Backups Fail With Comment File Not Found Error

NDMP backups fail when the dataset contains a high number of volumes (for example, approximately 300 volumes).

The following errors appear in the avtar logs:

2016/04/11-08:44:07.52813 [avtar]  sysfile::open open64 failed: code 2: No such file or directory
2016/04/11-08:44:07.52814 [avtar]  filedisk::doopen, fdesc == -1 after open attempt
2016/04/11-08:44:07.52814 [avtar]    Problem reading file (attempt#0), sleeping 30 sec and trying again
2016/04/11-08:44:25.01582 [ade-async]  servercontext::sendPingToAll(): dpninx=0, connections=1
....
2016/04/11-08:58:40.28278 [avtar]  sysfile::open open64 failed: code 2: No such file or directory
2016/04/11-08:58:40.28279 [avtar]  filedisk::doopen, fdesc == -1 after open attempt
2016/04/11-08:58:40.28280 [avtar]    Problem reading file (attempt#29), sleeping 30 sec and trying again
.....

2016/04/11-08:59:10.38269 [avtar]  Could not open file '<vadir>/<clientname>/<volume_name>.comment' (waited 30 secs)
2016-04-11 04:59:10 avtar Error <40091>: Comment entry "comment" file "<vadir>/<clientname>/<volume_name>.comment" not found.

This issue occurs when the dataset contains a large number of volumes.

During the backup:

• Avamar builds a composite tree of directory elements for the top-level backup structure.
• Avamar searches this composite tree to locate a valid base backup for each volume.
• Avamar performs this search up to 30 times for each volume across recent backups.

As the backup progresses:

• The time required to find a valid base increases.
• Volumes processed later in the dataset can take 3 to 4 minutes each.

During this extended search:

• Other avndmp threads cannot run.
• Threads responsible for writing .comment files for completed volumes become blocked.

This delay results in the .comment file not being created or available when required, which triggers the failure.

Apply the following workaround to limit the incremental search depth:

1. Open the avndmp.cmd configuration file.
2. Add the following parameter:

   --limit-incremental-search=10

3. Save the file.
4. Restart the Avamar NDMP backup process or rerun the backup job.

This parameter reduces the number of previous backups that Avamar searches for a valid base. The change improves performance and prevents blocking of threads responsible for creating comment files.

• This issue is more likely in environments with large NDMP datasets containing hundreds of volumes.
• Reducing the incremental search depth balances performance and backup efficiency.
• Monitor backup performance after applying the workaround to confirm improvement.