Data Domain: From Some Newer DD OS Versions, CA Certificate Errors May be Reported When Creating Replication Contexts From DD CLI

The Data Domain CLI replication add command may report CA certificate errors at both Source and Destination Data Domains. Example:

SE@ddve01## replication add source mtree://ddve01.lab/data/col1/test_boost_11 destination mtree://ddve02.lab/data/col1/test_boost_11

**** Error getting CA certificate for ddve01.lab (**** Error communicating with host ddve01.lab: could not resolve host.).

The DD debug/sms.info log also reports the error:

11/26 10:54:54.980028 [7f9eb99013e0] CURL error: curl_easy_perform() returned 6 [Could not resolve: ddve01.lab (Domain name not found)]
11/26 10:54:54.980091 [7f9eb99013e0] sms_trust_get_cert_targeted_do:(ddr/sm/sms/gen/ddr/sms_trust_data.c:527): **** Error communicating with host ddve01.lab: could not resolve host.
11/26 10:54:55.032059 [7f9eb99013e0] completed job: 24337 for operation: sms_replication_add, duration: 62 msec, status: **** Error getting CA certificate for ddve01.lab (**** Error communicating with host ddve01.lab: could not resolve host.).

This error can be reported for any of the Source or Destination Data Domain hosts being part of the replication context. At the example below we can also see that the error reported for the Destination:

SE@oscarddve01## replication add source mtree://ddve01.lab/data/col1/test_boost_11 destination mtree://ddve02.lab/data/col1/test_boost_11

**** Error getting CA certificate for ddve02.lab (**** Error communicating with host ddve02.lab: could not resolve host.).

Again, the Data Domain debug/sms.info log is showing the error too:

11/26 10:59:10.332885 [7f9eb9904a20] CURL error: curl_easy_perform() returned 6 [Could not resolve: ddve02.lab (Domain name not found)]
11/26 10:59:10.332935 [7f9eb9904a20] sms_trust_get_cert_targeted_do:(ddr/sm/sms/gen/ddr/sms_trust_data.c:527): **** Error communicating with host ddve02.lab: could not resolve host.
11/26 10:59:10.372900 [7f9eb9904a20] completed job: 24398 for operation: sms_replication_add, duration: 97 msec, status: **** Error getting CA certificate for ddve02.lab (**** Error communicating with host ddve02.lab: could not resolve host.).

Other similar issues may happen for port 3009 not being open between the Data Domains. Example below,

SE@oscarddve01## replication add source mtree://ddve01.lab/data/col1/test_boost_11 destination mtree://ddve02.lab/data/col1/test_boost_11

**** Error getting CA certificate for ddve02.lab (**** Error communicating with host ddve02.lab: the operation timed out.).

That example is shown at the sms.info log as:

11/26 11:33:23.403681 [1254ec80] CURL error: curl_easy_perform() returned 28 [Connection timed out after 30001 milliseconds]
11/26 11:33:23.403927 [1254ec80] sms_trust_get_cert_targeted_do:(ddr/sm/sms/gen/ddr/sms_trust_data.c:527): **** Error communicating with host ddve02.lab: the operation timed out.
11/26 11:33:23.474988 [1254ec80] completed job: 24741 for operation: sms_replication_add, duration: 30122 msec, status: **** Error getting CA certificate for ddve02.lab (**** Error communicating with host ddve02.lab: the operation timed out.).

From some newer DD OS versions (7.7.4.0 > 7.7.5.11, 7.10.0.0 > 7.10.1.1) the DD CLI "replication add" command is trying to get the Source and Destination Data Domain CA certificates. If this fails, it reports the issue and the replication context is not created.

As the first step, when getting the Source or Destination Data Domain certificates, it resolves the Source or Destination hostnames, the operation fails if the Data Domains are not able to be resolved the Source or Destination hostnames.

Other similar issues may happen too if the Source or Destination Data Domains are not able to communicate through TCP port 3009. (This is the port used for Data Domain CA certificate confirmation.)

Resolution:
Upgrade the DD OS version to 7.7.5.20 or later, 7.10.1.10 or later, or 7.12 or later after checking compatibility for your environment.

Workaround: 
Ensure that both Source and Destination Data Domains can resolve the partner Data Domain hostnames and their own hostnames (use the # net lookup command) through DNS or through Hostname local resolution (by adding them manually). 

Each Data Domain should have two host entries, an entry for its own hostname and one for the replication partner. 

Check the hosts mapping with the command: 

# net hosts show
# net hosts add <target IP> <target FQDN> <target hostname>

Example: 

# net hosts add 192.168.3.3 bkup20.yourcompany.com bkup20

Ensure that both Source and Destination Data Domains can reach the remote peer Data Domain through tcp port 3009 as this is the port used to get the remote peer CA certificates. You check in SE mode with the command:

# se telnet <IP> 3009

Alternatively, if port 3009 cannot be opened between Data Domains and a DD OS version upgrade is not possible for a long time, contact your support team to help in configuring a replication.