Avamar: Backup Fails with Invalid Token Error Due to Ddrmaint Service being down

Observed Backup Failure Indicators.

The following symptoms are recorded when the backup process fails due to the Data Domain Replicator (DDR) token issue.

• Avamar logs report usage of an invalid token and multiple error codes.
• Backup fails with a missing ddr_files.xml file.
• A fatal server connection problem prevents initialization.
• Command execution ends with exit code 10008, indicating inability to establish a connection with the server.

  019-05-27 20:09:30 avtar Error <41439>: Using invalid token:5ba93c9db0cff93f52b521d7420e43f6eda2784f
  2019-05-27 20:09:30 avtar Error <10542>: Data Domain server "datadomain_123.com" open failed DDR result code: 4904, desc: Invalid API argument.
  2019-05-27 20:09:30 avtar Error <10509>: Problem logging into the DDR server:'', only GSAN communication was enabled.
  2019-05-27 20:09:30 avtar FATAL <17964>: Backup is incomplete because file "/ddr_files.xml" is missing
  2019-05-27 20:09:30 avtar Info <10642>: DDR errors caused the backup to not be posted, errors=0, fatals=0
  2019-05-27 20:09:30 avtar Info <12530>: Backup was not committed to the DDR.
  2019-05-27 20:09:30 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
  2019-05-27 20:09:30 avtar Info <6149>: Error summary: 5 errors: 41439, 10542, 8941, 10509, 17964
  2019-05-27 20:09:30 avtar Info <8468>: Sending wrapup message to parent
  2019-05-27 20:09:31 avtar Info <5314>: Command failed (5 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))

• Search for DDR_GET_AUTH_TOKEN in ddrmaint.log returns no output.

  grep -i  DDR_GET_AUTH_TOKEN  /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log*

• dpnctl status shows the ddrmaint-service is down. 

  dpnctl status output showing ddrmaint-service down
  admin@avamar123:~/>: dpnctl status
  Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
  dpnctl: INFO: Avamar server status: up
  dpnctl: INFO: MCS status: up.
  dpnctl: INFO: emt status: up.
  dpnctl: INFO: Backup scheduler status: up.
  dpnctl: INFO: Maintenance windows scheduler status: enabled.
  dpnctl: INFO: Unattended startup status: disabled.
  dpnctl: INFO: avinstaller status: up.
  dpnctl: INFO: ConnectEMC status: up.
  dpnctl: INFO: ddrmaint-service status: down.
  dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

The backup operation fails because the ddrmaint-service on the Avamar server is not running. When this service is down, the Avamar client cannot obtain a valid DDR authentication token, leading to a cascade of errors that fails the backup.

Key Contributing Factors

• The dpnctl output indicates that the ddrmaint-service is down.
• Authentication to the Data Domain server fails, producing an "invalid token" error.
• The required file /ddr_files.xml is not generated, causing the backup to be incomplete.
• Subsequent DDR errors prevent the backup from being posted or committed.

Relevant Log Evidence

2019-05-27 20:09:30 avtar Error <41439>: Using invalid token:5ba93c9db0cff93f52b521d7420e43f6eda2784f
2019-05-27 20:09:30 avtar Error <10542>: Data Domain server "datadomain_123.com" open failed DDR result code: 4904, desc: Invalid API argument.
2019-05-27 20:09:30 avtar Error <10509>: Problem logging into the DDR server:'', only GSAN communication was enabled.
2019-05-27 20:09:30 avtar FATAL <17964>: Backup is incomplete because file "/ddr_files.xml" is missing
2019-05-27 20:09:30 avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.

Service Status Confirmation

admin@avamar123:~/> dpnctl status
dpnctl: INFO: Avamar server status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: disabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: down.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

 Restart the DDR Maintenance Service.

Step‑by‑step procedure

1. Log in to the Avamar server with an account that has administrative privileges.
2. Check the current status of the ddrmaint-service to confirm it is down.

   # Verify service status
   $ dpnctl status | grep ddrmaint-service
   ddrmaint-service status: down

3. Start the service.

   # Start the DDR maintenance service
   $ dpnctl start ddrmaint-service

4. Verify that the service is now running.

   # Re‑check status to ensure it is up
   $ dpnctl status | grep ddrmaint-service
   ddrmaint-service status: up

5. Confirm that backups are completing without the "invalid token" errors.
   1. Run a test backup or wait for the next scheduled backup to finish.
   2. Inspect the Avamar log for the absence of errors such as 41439 , 10542 , 10509 , 17964 , or 8941 .

      # Example of a successful backup log snippet
      2024-01-30 02:15:12 avtar Info <10642>: Backup completed successfully.
      2024-01-30 02:15:12 avtar Info <12530>: Backup was committed to the DDR.

   3. Confirm that the file /ddr_files.xml is present after the backup completes.