Avamar: NDMP Level 0 Backup Performance Issues, Causes, and Best Practices

Observed Issues with NDMP Level 0 Backups.

Performance‑related symptoms

The following conditions are commonly reported when performing an initial (Level 0) NDMP backup to Avamar:

• Backup duration extends to many hours or days (for example, 7 TB taking ≈ 121 hours, 40 TB taking ≈ 300 hours).
• Subsequent Level 1 or incremental backups complete in a fraction of the time (typically 45 minutes to a few hours).
• Avamar appears to remain in a "full backup" state and does not transition to incremental backups.
• Backup jobs are terminated after 24 hours when overtime is not enabled.
• When the file count exceeds roughly 10 million, the backup time increases markedly.
• The NAS spends most of the backup window scanning directory trees rather than transferring data.
• On Isilon systems, backing up the single

  /ifs

  
  Filesystem uses only one stream, resulting in slower completion. This is not supported.

Factors Contributing to Prolonged Level‑0 NDMP Backups

Level‑0 (full) NDMP backups require the NAS to enumerate and transmit every file and directory on the protected volume. The following conditions can cause the backup to run for many hours or days:

• **File count exceeds optimal limits** - Backups with more than 10,000,000 files significantly increase the time the NAS spends scanning the directory tree.
• **Deep or highly nested directory structures** - A deep tree forces the NAS to traverse many levels, slowing the scan compared with a wide, shallow hierarchy.
• **Large volumes of data** - Very large volumes (for example, 7 TB, 40 TB) naturally require more time for the initial full scan and data transmission.
• **High directory‑to‑file ratio** - Numerous directories with few files each increase metadata processing overhead.
• **File characteristics** - Small file sizes, varied file types, and frequent file changes can degrade scanning efficiency.
• **NAS disk activity** - When the NAS disks are heavily utilized by other workloads, the backup scan competes for I/O resources, extending runtime.
• **Insufficient NDMP streams** - By default an NDMP backup can run up to eight parallel streams; using fewer streams reduces parallelism and lengthens the backup.
• **Re‑assigning a volume to a different client** - Moving a volume triggers a new Level‑0 backup because the NAS must resend every byte to verify that data already exists on Avamar.
• **Isilon single‑filesystem behavior** - Backing up the sole

  /ifs

  
  Filesystem uses only one stream, limiting parallelism and increasing scan time. This is not supported by Isilon
• **Schedule limits without overtime** - If a backup exceeds the scheduled window and overtime is not enabled, the job may be terminated, causing repeated full backups.

These conditions collectively cause the NAS to spend most of the backup window scanning metadata rather than transmitting data, resulting in extremely long Level‑0 backup durations.

Recommended actions for NDMP Level 0 backups

1. Ensure the Level 0 backup can run to completion.

Critical Warning: Killing a Level 0 (full) backup prevents Avamar from moving to incremental backups, causing the backup job to repeat indefinitely.

• Do not cancel the Level 0 backup unless absolutely necessary.
• Verify that the backup schedule allows overtime so the job is not terminated when the scheduled window ends.

2. Configure overtime for the NDMP backup.

Set the overtime option in the Avamar client policy to allow the backup to run beyond the scheduled window.

# Example: enable overtime for a client (replace [CLIENT] with the actual client name)
avmaint setclientpolicy -c [CLIENT] --overtime true

 If you start a manual backup, the schedule limits do not apply, but the job will still be killed after 24 hours unless overtime is enabled.

3. Adjust the maximum number of streams.

Increasing the number of parallel streams speeds up the backup by allowing multiple directory trees to be processed simultaneously.

# Example: set max‑streams for a dataset (replace [DATASET] and [STREAM_COUNT])
avmaint setdataset -d [DATASET] --max-streams [STREAM_COUNT]

• Use up to eight streams per dataset (the default maximum for NDMP).
• Start with the largest volume first; subsequent volumes will follow automatically.

4. Verify file and directory count limits.

Best practice recommends backing up no more than 10 000 000 files per Level 0 backup.

• Run a file count on the source NAS and consider splitting the backup into multiple datasets if the limit is exceeded.

5. Optimize the directory tree structure.

• Prefer shallow, wide directory trees over deep, nested structures; shallow trees reduce scan time.
• When using Isilon, do not back up at the /ifs root level. This will limit the backup to a single stream.
• Avoid moving large directory subtrees between different paths; such moves cause the NAS to treat the data as new and perform a full scan.

6. Monitor NAS load during the backup.

• Check NAS disk I/O and CPU utilization; high load may indicate that the NAS is spending most of its time scanning directories.
• If the NAS becomes a bottleneck, consider reducing the number of streams or scheduling the backup during off‑peak hours.

7. Use Avamar tools for verification.

• Open Avamar Administrator and navigate to Backup Jobs → History to confirm the Level 0 job completes successfully.
• After the Level 0 finishes, run a Level 1 (incremental) backup and verify it completes within the expected time (typically < 1 hour for the examples given).
• Review the NDMP Accelerator logs for entries such as "Backup completed successfully" and ensure no "killed due to overtime" messages appear.

8. Contact Dell Support if issues persist.

If the Level 0 backup continues to exceed reasonable timeframes despite applying the above steps, open a support case. Provide the following information:

• Avamar version and patch level
• NAS model and firmware version
• Backup job ID and timestamps
• Current max‑streams setting and file count