Avamar: Exchange backups to Data Domain failing with "Dddrmgr write failure result code: 5040"

Summary: Avamar Exchange backups to Data Domain failing with "Dddrmgr write failure result code: 5040"

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Exchange backup fail with DDR errors similar to the following below:

2019-04-02 07:31:26 avtar Warning <18133>: Calling DDR_WRITE returned result code:(5040) calling system(), returns nonzero message:DDRIO_Write::WriteToDDR:  ddp_write failed
[ 1072] [5668] Tue Apr 02 07:31:26 2019 ddp_write() failed Offset 0, BytesToWrite 600, BytesWritten 0 Err: 5040-DDBoost reauthentication failure, NFS err = 1
[ 1072] [20940] Mon Apr 01 21:01:28 2019 ddp_stat() failed, Path avamar-123456789//STAGING/123456abcde/BACKUP-98765xxWXYZ/container.2.cdsf.trace, Err: 5004-nfs lookup failed (nfs: No such file or directory)
[ 1072] [20940] Mon Apr 01 21:00:43 2019
...
2019-04-02 07:31:26 avtar Error <16709>: DDRInstance::Invoke - ddrmgr write failure result code: 5040
2019-04-02 07:31:26 avtar Error <0000>: endDataSlices2: encountered an error with flushing buffered writes

This issue may occur even after increasing the OST ABANDON TIMEOUT value to a higher number than the default 10800 seconds (3 hours) in the Avamar grid.

Cause

This issue more likely to be seen if the DB backup size is very large (multiple terabytes).

From the ddrmaint log, we could see token authentication failure due to token expiration:

>04/02 07:31:26.325 (tid 0x7f5f3541d9xx): ost_validate_token: Invalid Token - Token End time 1554202326 < time now 1554204686
>04/02 07:31:26.325 (tid 0x7f5f3541d9xx): DDBOOST verify user Token failed: ost_validate_token: Invalid Token -  Token Expired
>04/02 07:31:26.325 (tid 0x7f5f3541d9xx): nfsprocx_ost_mnt_3_svc: connection failed permission from plugin 192.168.x.x version 3.4

By default, the token expiration time is 36000 seconds (10 hours). When it exceeds the expiration time and backup is still active, request is sent to Data Domain to extend the token.

In this case, the token extend request failed.

Resolution

As a workaround, split the backup jobs into separate datasets and policies, each backing up a subset of the databases so that the backup size will be smaller.  The smaller backup jobs then will complete in a time frame before the DDR token expires.

For a more permanent solution, open new service request to upgrade Data Domain operating system from v6.0.x to at least 6.1.1.x or higher.

Affected Products

Avamar Server

Products

Avamar Plug-in for Exchange VSS, Avamar Server, DD OS 6.0
Article Properties
Article Number: 000055440
Article Type: Solution
Last Modified: 03 Mar 2025
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.