Avamar: Isilon NDMP Backup Hung Caused by Wrong DMA Setting and Resolution Steps

Isilon NDMP Backup Appears Hung

Indicators observed in the log file

• Repeated deadlock messages indicating no work to do and nothing pending on the server.
• History processing reports zero NDMPHISTLISTLINE messages.
• Mark EOF history shows inode:0.
• Avtar errors reporting inability to locate history for specific inodes.
• NDMP history thread reports exhaustion of file‑history data and inability to answer pending GETNDMPISILONHISTLIST messages.
• Debug information showing handling of a GETNDMPISILONHISTLIST request for a directory inode.
• Response indicating inability to find history for inode 4371567677.

[avtar] context::loop: Probable deadlock: No work to do and nothing pending on server.
[avtar] context::loop: Probable deadlock: No work to do and nothing pending on server.
[avtar] context::loop: Probable deadlock: No work to do and nothing pending on server.

[phist] isilontar_history::processhist::body handled 0 NDMPHISTLISTLINE messages, which is all that readhist sent

[phist] isilontar_history::processhist::mark_eof_history lastparent_done inode:0

Avtar Errors relating unable to find history for inode ####

avtar Error <8667>: NDMP history processing thread has read all file history data and is unable to answer remaining 3 GETNDMPISILONHISTLIST messages from the data processing thread

avtar Info <10765>: DEBUG: answering GETNDMPISILONHISTLIST message:CLIENTONLY_GETNDMPHISTLIST64=412 serial=0 seq=0 flags=:N:0 kind=0 for dir inode:4371567677 with done=false and best effort data histlist:(nil)

[avtar] backisilontardir::response unable to find history for inode '4371567677'

Incorrect DMA Setting on Isilon

The Data Migration Accelerator(DMA) type on the Isilon cluster is configured with a value other than generic. This misconfiguration prevents Avamar from correctly processing NDMP history data, leading to deadlock conditions and history‑lookup failures.

• DMA type set to a non‑generic value (such as, vendor‑specific) blocks the NDMP history processing thread from answering GETNDMPISILONHISTLIST messages.
• Avamar logs repeatedly display "Probable deadlock: No work to do and nothing pending on the server."
• Avamar reports errors such as avtar Error<8667> and "unable to find history for inode….”

Log Evidence of the Misconfiguration

[avtar] context::loop: Probable deadlock: No work to do and nothing pending on server.
[phist] isilontar_history::processhist::body handled 0 NDMPHISTLISTLINE messages, which is all that readhist sent
avtar Error <8667>: NDMP history processing thread has read all file history data and is unable to answer remaining 3 GETNDMPISILONHISTLIST messages from the data processing thread
[avtar] backisilontardir::response unable to find history for inode '4371567677'

Fix Isilon NDMP backup deadlock.

Recommended solution - Set Isilon DMA mode to "generic"

Changing the Data Migration Accelerator (DMA) mode on the Isilon cluster to generic resolves the deadlock condition that prevents NDMP history processing.

• Log in to the Isilon cluster with an account that has admin privileges.
• Verify the current DMA setting.
• Modify the DMA mode to generic.
• Restart the NDMP service to apply the change.
• Confirm that Avamar backup jobs complete without hanging.

Example commands (run on the Isilon CLI):

# Show current DMA mode
isi storagepool settings view | grep dma

# Change DMA mode to generic
isi storagepool settings modify --dma-mode generic

# Restart NDMP service
svc_isilon restart ndmp

 After the change, monitor the avtar log for the removal of deadlock messages such as:

[avtar] context::loop: Probable deadlock: No work to do and nothing pending on server.

If changing the Isilon DMA setting is not feasible, apply the Avamar client flag as a temporary workaround.

• Edit the Avamar client configuration file ( /etc/avamar/avtar.conf or the appropriate location).
• Add the flag --disable-ndmp-history to the Avamar NDMP backup command line.
• Save the file and restart the Avamar client service.
• Run a test backup to ensure that the deadlock no longer occurs.

Example edit (replace the existing NDMP command line):

# Original command
avtar --ndmp ...

# Modified command with workaround flag
avtar --ndmp --disable-ndmp-history ...

 Restart the Avamar client service:

systemctl restart avamar-client

Caution: This flag disables NDMP history processing, which may limit reporting and audit capabilities. Use only as a temporary measure until the DMA setting can be corrected.

Verification steps

• Run a new Avamar NDMP backup job.
• Check the avtar log for the absence of deadlock messages and "unable to find history for inode" errors.
• Confirm that the backup completes successfully and that the data appears in the Avamar console.
• If issues persist, collect the latest avtar log and open a case with Dell Support.