Avamar: Data Domain Integration: How to recover from duplicate Data Domain entries

Overview:

When a Data Domain (DD or DDR) is added to an Avamar, it gets an index (starting from 1). It is set up this way so that clients can identify where to send the data to (Avamar, DD1, DD2, and so on).

Issue:

A Data Domain has been added twice with backups referencing both of the DD indexes:

For example, TESTDDVE-01 and the 10.10.10.10 are the same Data Domain:

ddrmaint read-ddr-info

<?xml version="1.0" encoding="UTF-8" standalone="yes"?
<avamar dpnid="1486428213" type="default" version="5">
   <datadomain count="2">
     <ddrconfig client-map-default="false" dd-cert-chain="-----BEGIN CERTIFICATE-----REMOVED;-----END CERTIFICATE-----" 
     ddos-version="6.0.0.9-544198" ddrcreatetime="1486470029" ddrid="F253B9041809D9C860C7622D246069471EF85C6B" 
     gsan-backup-target-default="false" hostname="TESTDDVE-01" index="1" instant-access-limit="1" ipv4-hostname="TESTDDVE-01" 
     ipv6-hostname="" max-streams="60" max-streams-for-cp-backup="2" modelno="DD VE Version 3" password="removed=="
     serialno="AUDVSJELBMSEVU" token="AQDh4CO2fhrIWwpxIDsf88yPNZeBtzCq6H+HwyXvc+i9RA==" username="ddboost">
       <snmp community="ahbPIvR+ACDJo5pSbCRwYA==">
         <ports getter-setter="161" trap="163">
         </ports>
       </snmp>
       <client-map>
       </client-map>
     </ddrconfig>
     <ddrconfig client-map-default="true" dd-cert-chain="" ddos-version="6.0.0.9-544198" 
     ddrcreatetime="1500084752" ddrid="8C04A202788C3018759DF31554789B396A44C36D" gsan-backup-target-default="true" 
     hostname="10.10.10.10" index="2" instant-access-limit="1" ipv4-hostname="10.10.10.10" ipv6-hostname="" max-streams="144"
     max-streams-for-cp-backup="3" modelno="DD VE Version 3" password="removed==" serialno="AUDVSJELBMSEVU" 
     token="AQDJj7hmwPER663UXdOEll37Jgc1ap0QksIniD/oLcgh3g==" username="ddboost">
       <snmp community="ahbPIvR+ACDJo5pSbCRwYA==">
         <ports getter-setter="161" trap="163">
         </ports>
       </snmp>
       <client-map>
       </client-map>
     </ddrconfig>
   </datadomain>
 </avamar> 

This results in one or more of the following:

1. The logs, or the output of the status.dpn command, report maintenance errors:

Checkpoint failed with result MSG_ERR_DDR_ERROR : cp.20170929084339 started Fri Sep 29 09:43:39 2017 ended Fri Sep 29 09:44:59 2017, completed 6236 of 6236 stripes
Last GC: finished Fri Sep 29 09:09:16 2017 after 08m 55s >> recovered 509.31 MB (MSG_ERR_DDR_ERROR)
Last hfscheck: finished Fri Sep 29 09:39:23 2017 after 28m 03s >> checked 2610 of 2610 stripes (MSG_ERR_DDR_ERROR)

2. The ddrmaint logs (/usr/local/avamar/var/ddrmaintlogs/ddrmaint.log*) show the following I/O errors: 

Sep 28 09:07:33 AVE-01 ddrmaint.bin[29899]: Warning: Calling DDR_OPEN_MSTREAM returned result code:(5009) I/O error message:DDRIO_Write::Initialize: ddp_ftruncate failed [29899] [140391700600576] 
Sep 28 09:07:33 2017 ddp_ftruncate() failed, Err: 5009-nfs setattr failed (nfs: I/O error)  [29899] [140391759820544] 
Sep 28 09:07:33 2017 ddp_rename() failed, FromPath avamar-1486428213/GSAN/cp.20170919080934, ToPath avamar-1486428213/DELETED/OOB/GSAN-cp.20170919080934_1506586053,  Err: 5004-nfs lookup failed (nfs: No such file or directory)  [29899] [140391759820544] 
Sep 28 09:07:33 2017 ddp_readdir() failed, Err: 5006-end of directory  [29899] [140391759820544] 
Sep 28 09:07:33 AVE-01 ddrmaint.bin[29899]: Warning: Calling DDR_OPEN_MSTREAM returned result code:5009 message:I/O error
Sep 28 09:07:33 AVE-01 ddrmaint.bin[29899]: Error: gc-finish::createFileOnDDR - failed opening ddr stream to write file GSAN/cp.20170920080908.expired, ddr: DDVE-01, lsu: avamar-1486428213, DDR result code: 5009, desc: I/O error 

3. When attempting backups to the Data Domain, they can fail with miscellaneous errors. 

Causes:

1. This rare scenario (which happens only if all the following are actions are performed): 

a. A Data Domain was added using the hostname.

b. Avamar backups were performed to the Data Domain.

c. The file system on the Data Domain was destroyed or removed before the Data Domain was removed from Avamar. 

d. The Data Domain is re-added using an IP address.

2. Where a Data Domain has not been cleanly removed, and then the same Data Domain is added again with a different IP Address or hostname. 

3. Where a Data Domain is initially added using the IP address, and later added again using a hostname. The issue here is that network name resolution is not performed before writing to the persistent store on Avamar.

Resolution:

Create a Service Request with the Dell Technologies Avamar Support team to resolve this issue, referencing this article number.