Avamar: Failed to Update or Connect to a Data Domain System due to Locked DD Boost User
Summary: The DD Boost user account is locked due to the several attempts of logging in with the wrong password.
Symptoms
All backups from Avamar to Data Domain fail with:
2020-02-09 10:05:39 avtar Error <10542>: Data Domain server "datadomain.local" open failed DDR result code: 5075, desc: the user has insufficient access rights (Log #1)
2020-02-09 10:05:39 avtar Error <10509>: Problem logging into the DDR server:'', only GSAN communication was enabled. (Log #1)
2020-02-09 10:05:39 avtar FATAL <17964>: Backup is incomplete because file "/ddr_files.xml" is missing (Log #1)
2020-02-09 10:05:39 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials. (Log #1)
All Maintenance jobs fail with:
MSG_ERR_DDR_ERROR
The below command reports the checkpoints as invalid, and the error:
admin@avamar :~/>: cplist --full
cplist: ERROR: ddrmaint: <4750>Datadomain get checkpoint list operation failed.
2020/02/09-12:08:56.31504 [cplist] ERROR: <0001> ddrmaint: <4750>Datadomain get checkpoint list operation failed.
cp.20200122200414 Thu Jan 23 00:04:14 2020 invalid --- --- nodes 1/1 stripes 1875
cp.20200122201338 Thu Jan 23 00:13:38 2020 invalid --- --- nodes 1/1 stripes 1875
cp.20200123043629 Thu Jan 23 08:36:29 2020 invalid --- --- nodes 1/1 stripes 1876
The command below reports the same error:
admin@avamar:~/>: ddrmaint cplist
<4750> Datadomain get checkpoint list operation failed.
The "DDR Authentication flag" is disabled in mcserver.xml:
root@avamar:~/#: grep -i "use_ddr_auth_token"
/usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
<entry key="use_ddr_auth_token" value="false" />
Cause
Due to several and frequent attempts from Avamar with an incorrect DD Boost password, the account gets locked, closing all the connections to the Avamar.
Attempting an SSH connection to the Data Domain as the DD Boost user shows that the account is locked due to X failed logins:
admin@avamar:~/>: ssh ddboost@datadomain
EMC Data Domain Virtual Edition
Account locked due to X failed logins
Password:
From a Data Domain account, the output of the below command shows the DD Boost user as enabled:
sysadmin@ddve# user show list
User list from node "localhost".
Name Uid Role Last Login From Last Login Time Status Disable Date
-------- --- ----- --------------- ------------------------ -------- -----------
sysadmin 100 admin xx.xxx.xx.x Mon Feb 10 11:57:06 2020 enabled never
ddboost 500 admin xx.xxx.xx.x Mon Feb 10 12:07:07 2020 enabled never
-------- --- ----- --------------- ------------------------ -------------------
2 users found.
The ddfs.info file is showing authentication failures from the Avamar IP Address.
Resolution
Contact Dell Support to work with the Avamar and Data Domain Support teams to apply the resolution.
This resolution must be followed on each Avamar using the same DD Boost user:
ddboost@ddve# ddboost storage-unit show
Name Pre-Comp (GiB) Status User Report Physical
Size (MiB)
----------------- -------------- ------ ------- ---------------
avamar-xxx 515401.8 RW ddboost -
----------------- -------------- ------ ------- ---------------
D : Deleted
Q : Quota Defined
RO : Read Only
RW : Read Write
RD : Replication Destination