Avamar - Backup Failure to DataDomain due to DDR_GET_AUTH_TOKEN due to too many IP addresses

Avtar logs report the backup failing due to an invalid token:

<snip>
2017-05-26 14:04:56 avtar Info <10539>: Connecting to Data Domain Server "testdd.dell.com"(1) (LSU: avamar-1393882510) with auth token
2017-05-26 14:04:56 avtar Info <10540>: - Resolved Data Domain Server name "testdd.dell.com" to the IP address "10.241.170.73"
2017-05-26 14:04:56 avtar Info <41236>: - Connecting to Data Domain Server name "testdd.dell.com" with token:5ba93c9db0cff93f52b521d7420e43f6edxxxxxx
2017-05-26 14:04:56 avtar Error <41439>: Using invalid token:5ba93c9db0cff93f52b521d7420e43f6edxxxxxx
2017-05-26 14:04:56 avtar Error <10542>: Data Domain server "testdd.dell.com" open failed DDR result code: 4904, desc: Invalid API argument.
2017-05-26 14:04:56 avtar Error <10509>: Problem logging into the DDR server:'', only GSAN communication was enabled.
2017-05-26 14:04:56 avtar FATAL <17964>: Backup is incomplete because file "/ddr_files.xml" is missing
2017-05-26 14:04:56 avtar Info <10642>: DDR errors caused the backup to not be posted, errors=0, fatals=0
2017-05-26 14:04:56 avtar Info <12530>: Backup was not committed to the DDR.
2017-05-26 14:04:56 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
2017-05-26 14:04:56 avtar Info <6149>: Error summary: 5 errors: 41439, 10542, 8941, 10509, 17964
2017-05-26 14:04:56 avtar Info <8468>: Sending wrapup message to parent
2017-05-26 14:04:56 avtar Info <5314>: Command failed (5 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
<snip>

Review of /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log shows shows an error message that there appears to be a large number of IPs associated with the single token request:

<snip>
May 26 14:17:27 testava ddrmaint.bin[107666]: Info: request-token:open_ddr:service handle:1 index:1 server:testdd.dell.com user:ddboost duration=1800 expires=2017-05-26 14:47:27
May 26 14:17:27 testava ddrmaint.bin[107666]: Info: ddrmaint Info <41440>: Data Domain handle:1 capabilities:0x0020023B
May 26 14:17:27 testava ddrmaint.bin[107666]: Warning: Calling DDR_GET_AUTH_TOKEN returned result code:(4904) Invalid API argument. message:Maximum number of client hostnames reached. 52 > 16
May 26 14:17:27 testava ddrmaint.bin[107666]: Error: request-token::body - Failed to get token. Use token=1:00. Error:4904 Reason:Invalid API argument.
May 26 14:17:27 testava ddrmaint.bin[107666]: Error: <xxxx>Datadomain request token operation failed.
May 26 14:17:27 testava ddrmaint.bin[107666]: Info: ============================= request-token finished in 0 seconds
May 26 14:17:27 testava ddrmaint.bin[107666]: Info: ============================= request-token cmd finished =============================
<snip>

At this time due to a limitation within the DDboost API the maximum number of IP addresses and hostnames together possible for a single token request is set at 16.

An RFE with the Data Domain team will be required to increase the maximum number of IPs accepted with a single token request for a particular client.

Ensure that the problematic client actually has that many IP addresses to verify this particular issue.  Use ifconfig (Linux) ipconfig (Windows) commands to verify that. 

As a temporary work around it is possible to disable token based authentication.

1. Verify the current token settings in the mcserver.xml file:
 

grep -i use_ddr_auth_token /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

<entry key="use_ddr_auth_token" value="true" />
            
2. Create a backup of the mcserver.xml:

cp -p /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml /usr/local/avamar/var/mc/server_data/prefs/x-mcserver.xml-backup-with-token

3. Dsable token based authentication. 

Note: It is not possible to disable token authentication only for affected clients. But it is possible to disable token based authentication per plugin.

Permanent fix:
Currently, there is no permanent fix available.  A RFE has been opened with Data Domain Engineering to increase the limit on the number of hostames and IPs when requesting a token with ddboost.  This knowledge article will be updated as a result of the RFE's resolution.