Avamar - Isilon - NDMP Incremental backups completed with exceptions Error 9982

Avamar - Isilon - NDMP Incremental Backups Completed with Exceptions.

Avamar backups are missing files during incremental backups. This condition could cause data to not be backed up and not be available for restore.

During NDMP incremental backup with Isilon OneFs, the backup completed with exceptions or failed with the following Avtar error:

avtar error: 2015-03-12 03:02:53 avtar Error <9982>: While processing _file_name_X or directory_name_X (inode XXXX713107), NDMP file history messages indicate the presence of a directory entry (inode:XXXX870544 name: Y) that doesn't appear in the data stream or in our previous backup of this directory.

• Severity Rating: High Impact
• Description: Avamar backups are missing files during incremental backups due to Isilon OneFs not detecting file case changes and not sending data stream for files or directories with case changes.

Root Cause of the Issue

The root cause of the issue is related to how file system scanning is performed by the NAS for changes, and not the NDMP client. It is up to the NAS to determine what has changed, and the NDMP client only receives what the NAS tells it has changed.

Avamar is sensitive during the incremental process and tries to find the previous data stream from the "file_name_X or directory_name_X" but instead finds a file or directory change case "File_Name_x or directory_name_x" and results in no valid backups.

This condition is caused because Isilon OneFs is not detecting file case changes and not sending a data stream for files or directories with case changes.

Specific Error Messages

The issue is associated with the following Avtar error:

2015-03-12 03:02:53 avtar Error <9982>: While processing _file_name_X or directory_name_X (inode XXXX713107), NDMP file history messages indicate the presence of a directory entry (inode:XXXX870544 name: Y) that doesn't appear in the data stream or in our previous backup of this directory.
📋

Permanent Fix and Workarounds

The final resolution is underway by both teams Avamar/Isilon to resolve this from the root cause.

Permanent fix In Isilon OneFS v7.2.0.3 as listed in "Isilon OneFS Version 7.2.0.0 - 7.2.0.5 Release Notes" ID 147606.

In Isilon OneFS v7.1.1.5 as listed in "Isilon OneFS Version 7.1.1.0 - 7.1.1.8 Release Notes" ID 131843.
 

Workaround Options

• Option 1: Perform Level 0 backups
• Option 2: Customers to keep track of incremental backups that completed with exceptions

Option 1. Perform Level 0 backups.
               

1. Some errors or exceptions must be tracked by customer to maintain data integrity.
2. To avoid performance issues split the Isilon NDMP backup into multiple target directories.
   Directory "/ifs" should not be a target. If it is, only a single stream will be used to backup up all data.
    

Option 2: Track Incremental Backups

Examine the logs to find the renamed files, then touch the files so that they appear changed in the next incremental backup.

Do not allow more than 30 consecutive backups to "complete with exceptions", otherwise additional manual intervention is required for the software to find a valid base for the next incremental.

As longer it is since the last valid "successful" backup, the more data that is sent by NAS during the next incremental.