Avamar jobs fail with error "user has insufficient privileges" due to DD Boost Issue 209416

Summary: Backups and restores fail with "user has insufficient privileges" in Avamar server running DD Boost library 3.4.0 due to Data Domain Issue #209416. This issue can cause your DD Boost user to get locked as the client authentication fails on Data Domain for the backup/restore process. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

This issue can cause the DD Boost user to get locked because of which the Avamar maintenance tasks may fail.

The error "user has insufficient privileges" is a generic error. Therefore, review the ddrmaint.log for details on the issue. 

On the backup or restore log, the following error is seen:      
avtar Error <10512>: Problem logging into the DDR server index:1 
avtar Error <10542>: Data Domain server "DD.X.X.com" open failed DDR result code: 5075, desc: the user has insufficient access rights
avtar Error <10512>: Problem logging into the DDR server index:1
To Verify the ddboost plug-in version on Avamar, run the following command: 
strings /usr/local/avamar/lib/libDDBoost.so | grep "[0-9]\.[0-9]\.[0-9]\.[0-9]"​
Or  
grep -i engine ddrmaint.log | tail -1
​​​​​Search the log with Avamar client's IP or the client ID. In DDFS log (/ddr/var/log/debug/ddfs.info) on Data Domain, the following errors are seen: 
06/11 09:18:31.347 : WARNING: Failed to verify the password for user ddboost.  Error is 7:Authentication failure
06/11 09:18:31.348 : nfs_rpc_svc_idx0 accepted 2e000001ba 201 from X.X.X.X:54896
06/11 09:18:31.354 : ost_decrypt_mnt_sec_request(): EVP_DecryptFinal_ex failed
06/11 09:18:31.354 : nfsproc3_ostmntsec_3_svc: connection failed permission (corrupted credentials) from host X.X.X.X - ost_decrypt_mnt_sec_request(): failed to finalize plain text (101077092, error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt)
This confirms that Data Domain was unable to decrypt the password provided by backups and restores.
 
Note: This issue can also occur if the client backing up is running v7.5.0, even if the server has been upgraded to v7.5.1+. This is due to the client making a direct connection to the Data Domain during the backup, the ddboost plug-in version from the client is used.

In this case, the client avtar log shows the following errors during the backup:      
2018-10-11 12:05:09 avtar Info <19155>: - Establishing a connection to the Data Domain system with certificate authentication (Connection mode: A:2 E:2).
2018-10-11 12:05:09 avtar Info <18120>: DDR trace is enabled.
2018-10-11 12:05:26 avtar Warning <18133>: Calling DDR_WRITE returned result code:(5075) the user has insufficient access rights message:DDRIO_Write::WriteToDDR: ddp_write failed
[ 6148] [3492] Thu Oct 11 12:05:26 2018
    ddp_write() failed Offset 0, BytesToWrite 16144, BytesWritten 0 Err: 5075-nfsproc3_ostmntsec_3 failed (nfs: Operation not permitted)
 [ 6148] [4932] Thu Oct 11 12:04:58 2018
    ddp_access() failed, Path avamar-1537798766/STAGING/77e75ce4a380b20c7572c3053e8409520d37c852/BACKUP-914CF43D1553E172BA1E7177D890CA1C77xxxxxx, mode 0 Err: 5004-nfs lookup failed (nfs: No such file or directory)
2018-10-11 12:05:26 avtar Error <16709>: DDRInstance::Invoke - ddrmgr write failure result code: 5075
And the ddfs.info log again shows "corrupted credentials" errors from around the time of the backup: 
10/11 12:05:10.544 (tid 0x7f7b90854xxx): nfsproc3_ostmntsec_3_svc: connection failed permission (corrupted credentials) from host 192.168.xxx.xx - ost_decrypt_mnt_sec_request(): failed to finalize plain text (101077092, error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt)

Cause

Avamar server 7.5.0 uses DD Boost library version: 3.4.0.2, this issue exists on DD Boost library 3.4.0 and is fixed on 3.4.1.

Avamar server version 7.5.1 has DD Boost library version: 3.4.1.1. embedded in it. 

Starting from the 3.4 plug-in, Data Domain does authentication using pre-shared-keys (PSKs) generated from the password. The client generates a PSK from the application-supplied password, encrypts some authentication information, and sends it to DDR. The DDR tries to generate the same PSK (since it already knows the password for a user) and decrypt the contents the client sent and validate.

With DD Boost ifgroup enabled, the Data Domain falls into the reconnected path, which calls the PSK connection path and relies on the password crypt_hash stored in the nfs_conn structure. On DD Boost plug-in 3.4.0.2 though, Data Domain does not generate the password crypt_hash again during reconnection. The PSK key is not the same on the client and DDR, hence the decryption failure.

The fix is available in DD Boost library 3.4.1.0-574461.

Resolution

Permanent Fix:
Upgrade the Avamar server and all clients running v7.5.0, to v7.5.1-101 or later.

Affected Products

Avamar

Products

Avamar, Avamar Server, Avamar Virtual Edition, Data Domain, Data Domain Boost - Open Storage
Article Properties
Article Number: 000080009
Article Type: Solution
Last Modified: 20 May 2024
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.