Avamar: NDMP Backup Of NetApp Fails With High Memory Usage Fatal Signal 11
Summary: When backing up NetApp volumes, the backup fails in the initialization phase. When avtar tries to read the inode.dir.map from previous backup, it terminates with Fatal signal 11 due to high memory usage. ...
Symptoms
Backup Failure Symptoms
The backup of NetApp volumes fails in the initialization phase when avtar tries to read the inode.dir.map from the previous backup. The backup terminates with a Fatal signal 11.
NetApp uses inodes instead of file names to document the files in the data stream. With large file counts, the initial management processes can run out of memory as this is where the inode maps are stored, by default.
The failure typically occurs during scheduled backups. A manual backup usually succeeds.
Example: Two volumes are backed up. The larger volume (>200 TB) succeeds during a scheduled backup. The smaller volume (~21 TB) fails scheduled backups. It succeeds when run manually.
Avamar documentation mentions a limit of 200M files. This example had 480M files.
Relevant Log Messages
2022-04-20 13:00:36 avndmp Info : [snapup-/plz/Images] NDMP: DUMP: Using inowalk incremental dump for Full Volume
2022-04-20 13:01:14 avndmp Info : [ctl_spawn] Process 5718 (/usr/local/avamar/bin/avtar) finished (code 176: fatal signal)
2022-04-20 13:01:14 avndmp Warning : [ctl_spawn] CTL workorder "9am-3pm-Images-165044985231#1" non-zero exit status 'code 176: fatal signal'
2022-04-20 13:01:16 avndmp Error : [snapup-/plz/Images] Premature end of child. Changing exitcode from 176 to 157
The is no indication of a cause.
Avtar Log Messages
log 2022-04-20 13:00:43 avtar Info : Using backup #49 timestamp 2022-04-19 18:08:46 UTC label "mages-165044985231#0" as base for incremental backup
2022-04-20 13:00:43 avtar Info : - Establishing a connection via token to the Data Domain system with encryption (Connection mode: A:3 E:2).
2022-04-20 13:00:43 avtar Info : CDSF backup does not use asynchronous reading of NetApp stream. Disabling that feature.
2022-04-20 13:00:43 avtar Info : Reading inode.dir.map from previous backup
2022-04-20 13:01:14 avtar FATAL : Fatal signal 11 in pid 5718
2022/04/20-13:01:14.76035 [avtar] FATAL ERROR: uapp::handlefatal: Fatal signal 11
2022/04/20-13:01:14.76064 [avtar] ERROR: uapp::handlefatal: aborting program pid=5718, sig=11
2022-04-20 13:01:14 avtar FATAL : handlefatal: Aborting program with code 176, pid=5718, sig=11
Cause
Root Cause of the Issue
The issue is caused because the inode map is larger than the memory allocated for the map.
When it is read into memory, it writes past the allocated memory boundary causing the Signal 11 fatal error message.
This occurs due to the high inode count for the backup, which overruns the memory segment reserved for the inode map.
The error messages in the logs, such as "Fatal signal 11" and "uapp::handlefatal: Fatal signal 11", indicate that the program has encountered a fatal error due to the memory overrun.
2022-04-20 13:01:14 avtar FATAL : Fatal signal 11 in pid 5718 2022/04/20-13:01:14.76035 [avtar] FATAL ERROR: uapp::handlefatal: Fatal signal 11 2022/04/20-13:01:14.76064 [avtar] ERROR: uapp::handlefatal: aborting program pid=5718, sig=11
Resolution
Configuring Avtar to Use Disk for Inode Map Storage.
The workaround is to store the inode maps to a disk file rather than memory. This tells avtar that when reading the inodes maps to store them on disk rather than in memory. This eliminates the SIG-11 memory issue.
There are two methods to set the stream-disk-buffer flag to configure avtar to use disk for inode map storage.
Method 1: Using the Avamar Administrator Interface
Go to the client dataset, open the On-Demand Backup Options dialog box, click More Options. The Backup Command-Line Options dialog box is displayed. Click More. Type [avtar]stream-disk-buffer in the Enter Attribute, select true in the Enter Attribute Value fields. Click Add icon to add the attribute.
Method 2: Using avtar.cmd
Using a UNIX text editor, in the NAS client var directory in /usr/local/avamar/var/CLIENT_NAME, create a file avtar.cmd. If the file exists, edit the existing file. In the file, on its own line enter the following flag:
--stream-disk-buffer
Verifying the Resolution
After setting the stream-disk-buffer flag, verify that the issue has been successfully resolved by checking the avndmp.log and avtar log files for any errors related to inode map storage.