Avamar: Avamar backups integrated with Data Domain fail with a token encryption failed with error
Résumé: Avamar backups integrated with Data Domain (DD) fail. The ddrmaing.log reports "Token encryption failed with error", and the ddfs.info file reports "This key has an invalid key magic".
Symptômes
Avamar Backups integrated with Data Domain are failing.
Reviewing the avtar logs indicates that the error is an invalid token:
<snip>
2017-02-21 11:35:50 avtar Info <40058>: - Client connecting to the Avamar Server using authentication, client connecting to the Data Domain system using two-way authentication
2017-02-21 11:35:51 avtar Info <10540>: - Resolved Data Domain Server name "testdd.emc.com" to the IP address "10.241.170.73"
2017-02-21 11:35:51 avtar Info <41236>: - Connecting to Data Domain Server name "testdd.emc.com" with token:5ba93c9db0cff93f52b521d7420e43f6eda2784f
2017-02-21 11:35:51 avtar Error <41439>: Using invalid token:5ba93c9db0cff93f52b521d7420e43f6eda2784f
2017-02-21 11:35:51 avtar Error <10542>: Data Domain server "testdd.emc.com" open failed DDR result code: 4904, desc: Invalid API argument.
2017-02-21 11:35:51 avtar Error <10509>: Problem logging into the DDR server:'', only GSAN communication was enabled.
2017-02-21 11:35:51 avtar FATAL <17964>: Backup is incomplete because file "/ddr_files.xml" is missing
<snip>
Reviewing /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log on the Avamar grid shows that the token request failed:
<snip>
Feb 21 11:35:23 av-un-ge06-adm0 ddrmaint.bin[47285]: Info: request-token:open_ddr:service handle:1 index:1 server:testdd.emc.com user:ddboost duration=1800 expires=2017-02-21 13:05:23
Feb 21 11:35:23 av-un-ge06-adm0 ddrmaint.bin[47285]: Error: request-token::body - Failed to get token. Use token=1:00. Error:5008 Reason:invalid argument
Feb 21 11:35:23 av-un-ge06-adm0 ddrmaint.bin[47285]: Error: <xxxx>Datadomain request token operation failed.
<snip>Reviewing the /ddr/var/log/debug/ddfs.info log on the Data Domain reveals that token encryption failed due to invalid key magic:
1. Log in to the Data Domain.
2. Review the ddfs.info log by using the following command:
log view debug/ddfs.info
<snip>
02/21 11:35:23.543 (tid 0x7f53bd82d990): ddboost-<testavamar.emc.com-59512>: testavamar.emc.com Local Time: Tue Feb 21 12:35:23 2017
02/21 11:35:23.553 (tid 0x7f53bd8078e0): nfsproc3_ost_get_token_3_svc: ost_build_token
02/21 11:35:23.553 (tid 0x7f53bd8078e0): ost_build_token()
02/21 11:35:23.553 (tid 0x7f53bd8078e0): ost_encrypt_string_with_key() This key has an invalid key magic 9df5c181
02/21 11:35:23.553 (tid 0x7f53bd8078e0): ost_build_token: Token encryption failed with error 5008
02/21 11:35:23.553 (tid 0x7f53bd8078e0): nfsproc3_ost_get_token_3_svc: ost_build_token failed [DDErrno = 5008 (ost_encrypt_string_with_key() Invalid User Key, Cannot be used to decrypt string)]
<snip>If the invalid key magic messages are not present in the ddfs.info log, stop referencing this article.
Cause
The access token must be refreshed.
Résolution
1. Revoke token access from the ddboost account on Data Domain:
ddboost user revoke token-access <ddboost username>
2. Within the Avamar UI, Edit the Data Domain server:
Management Console Server (MCS) UI:
a. In the Avamar Administrator, click the Server launcher button.
The Server window appears.
b. Click the Server Management tab.
c. Select the Data Domain to edit.
d. Select Actions > Edit Data Domain System.
The Edit Data Domain System dialog box appears.
e. Click OK. (No changes to the DD configuration are required)
-- or --
Avamar User Interface (AUI)
a. In the UI, Select System (Under Administration)
b. Select Data Domain.
c. Highlight the Data Domain to edit, and click edit.
d. Enter the ddboost username and password (twice), and click validate.
The following message should be displayed: Successfully retrieved Data Domain system information
If the credentials are correct, but the following is received: "Failed to get Data Domain system information", clear "Use certificate authentication for REST communication" and try again.
e. Click Next, Next, and finish (No changes to the DD configuration are required)
3. Restart DD Boost on the Data Domain:
a. Log in to an SSH session on the Data Domain.
b. Enter the following commands in the Data Domain CLI:
ddboost disable
ddboost enable
4. Perform a test backup to ensure this has resolved the invalid token issue.