Avamar: Data Domain Integration: How to clear token cache in MCS and Data Domain

The following errors may be seen during failed Avamar backups to Data Domain in the backup logs (extracted from the Avamar Administrator UI):

avtar Warning <18125>: Calling DDR_OPEN_VIA_TOKEN returned result code:5341 message:SSL library error 

-- Or --

avtar Error <41439>: Using invalid token:<token>
avtar Error <10542>: Data Domain server "<ddhostname>" open failed DDR result code: 4904, desc: Invalid API argument. 

Tokens that are stored in the token cache in MCS or on Data Domain may have become stale and must be cleared.

This process clears any token caching on both the Management Console Server (MCS) and Data Domain:
(It may resolve Avamar client to Data Domain connection failures with token errors)

1. Log in to the Avamar Utility Node as admin.

2. Review the Data Domain (DD) information from Avamar:

ddrmaint read-ddr-info

Example output (truncated)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<avamar dpnid="1643691765" type="default" version="5">
  <datadomain count="1">
    <ddrconfig client-map-default="true" cloud_enabled="true" cloud_unit_name="ECS_Unit1" dd-cert-chain="-----BEGIN CERTIFICATE-----&#xA;MMME4zCCA0ugAwIBAgIJAJSeE....Gu+1P2v&#xA;Hj1Hje5Rsg==&#xA;-----END CERTIFICATE-----" 
    ddos-version="7.7.5.25-1078970" ddrcreatetime="1685672811" ddrid="F5F5F3A1A218C7E2D75F20E0A1596E66AA2EE764E9" gsan-backup-target-default="true" hostname="dd-prod.company.com" index="1" instant-access-limit="32" ...
    ipv4-hostname="dd-prod.company.com" ipv6-hostname="" max-streams="50" max-streams-for-cp-backup="2" modelno="DD VE" mtree_name="/data/col1/avamar-12345678... username="ddboost-prod">
      <snmp community="noSejOLJ1lRsg1kEHjWdiA==" version="2">
        <ports getter-setter="161" trap="163">
        </ports>
      </snmp>
      <client-map>
      </client-map>
      <auth-type>credential</auth-type>
    </ddrconfig>
  </datadomain>
</avamar> 

The Data Domain hostname and ddboost user (highlighted in green and yellow above) are required later in this procedure.

3. Confirm that there are no maintenance tasks (Checkpoint, Checkpoint validation (aka hfscheck), or Garbage Collection) running:

status.dpn 

...
Last checkpoint: cp.20250821130414 finished Thu Aug 21 09:04:38 2025 after 00m 24s (OK)
Last GC: finished Thu Aug 21 08:00:28 2025 after 00m 09s >> recovered 1.27 MB (OK)
Last hfscheck: finished Thu Aug 21 09:03:57 2025 after 02m 35s >> checked 1523 of 1523 stripes (OK)
...

4. Confirm from the status.dpn output that the previous maintenance tasks completed without error. 

Do not continue if maintenance tasks are failing.

5. Put the grid into a good controlled state per Avamar: How to Set Avamar into a Known Controlled State.

6. Verify that there are no backups running:

Avamar:

avmaint sessions | grep sessionid

If there are any backups running on Avamar, refer back to Avamar: How to Set the Avamar Server into a Known Controlled State.

Data Domain:

ddboost streams show active all

7. Once there are no backups running, stop the MCS and ddrmaint-service services on Avamar:

dpnctl stop ddrmaint-service 

dpnctl stop mcs

8. Verify that the services have stopped:

dpnctl status ddrmaint-service 

dpnctl status mcs

9. From a Data Domain session, revoke the token access for the ddboost user:

ddboost user revoke token-access <ddboost_username>

The value of <ddboost_username> was captured earlier in step 2.

Example:

ddboost user revoke token-access ddboost-prod

10. Stop and start ddboost service:

ddboost disable

ddboost enable

11. Return to the Avamar session and restart Avamar services:

dpnctl start mcs,sched,maint,ddrmaint-service 

12. Verify that the Avamar services are running:

dpnctl status

Expected output:

dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ddrmaint-service status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

13. From the Avamar command line, update the Data Domain configuration:

mccli dd edit --name=<dd_hostname> 

The value of <dd_hostname> was captured earlier in step 2.

Example:

mccli dd edit --name=dd-prod.company.com 

14. Perform a test backup and confirm that it is successful.

If backups are still failing after clearing the token cache, Create a new Service Request with the Dell Technologies Avamar Support team to assist.