Data Domain: DDMC: Unable to add Managed Systems to Management Center
Summary: Troubleshooting Steps to add Data Domain (DD) to PowerProtect DD Management Center (DDMC).
Symptoms
Unable to add a specific Data Domain (DD) to DDMC. Example:
SE@DDMC01## managed-system add abc.com force The SHA1 fingerprint for the remote host's CA certificate is F1:D2:22:95:7C:45:C9:69:CB:76:25:18:C7:33:30:43:7A:CA:98:B9 Do you want to trust this certificate? Are you sure? (yes|no) [no]: yes ** Once added, all "admin" role users on this DD Management Center will operate on "abc.com" system with "admin" role. To allow "abc.com" to be managed by this DD Management Center, Enter "abc.com" sysadmin password: ok, proceeding. *** Add abc.com failed: System "abc.com" is in the "unknown" state. Data collection is disabled
Another possible error message when trying to add a new managed system to DDMC is as follows:
**** managed-dd.example.com: Error communicating with host ddmc.example.com: error occurred in the SSL/TLS handshake.
Cause
It can be due to various reasons like:
- Connectivity Issue
- Invalid entries in DD
- Invalid entries on DDMC
- Required port is not open
- SSL/TLS protocol version mismatch between the DDMC and the DD
Logs:
DDMC:
Messages.engineering:
Jul 29 19:04:36 MSPjDDMC01 sms: NOTICE: Trust with host aaa.com has been added Jul 29 19:09:42 MSPjDDMC01 -ddsh: NOTICE: MSG-DDSH-00017: (tty=pts/0, session=8899) tassos1: command "managed-system add abc.com force" exited with code: 95 Jul 29 20:58:37 MSPjDDMC01 -ddsh: NOTICE: MSG-DDSH-00009: (tty=pts/0, session=8899) tassos1: command "managed-system add abc.com force" Jul 29 21:04:36 MSPjDDMC01 sms: WARNING: ems_post_event: Failed to initialize event: Incompatible managed system version. EVT-OBJ::SystemName=abc.com EVT-INFO::DetectedVersion= Jul 29 21:23:32 MSPjDDMC01 sms: NOTICE: Trust with host aaa.com has been added Jul 29 21:47:24 MSPjDDMC01 -ddsh: NOTICE: MSG-DDSH-00017: (tty=pts/0, session=8899) tassos1: command "managed-system add abc.com force" exited with code: 245
sms.info
07/29 21:04:36.487 (tid 0x6ffbca0): **** Error communicating with host abc.com: Error communicating with host abc.com: error occurred in the SSL/TLS handshake. 07/29 21:04:36.509 (tid 0x6ffbca0): Workflow Getting system data (ID 1434912) starts child workflow (ID 1434913) to get current node config & status info for host "abc.com" 07/29 21:04:36.521 (tid 0x70005a0): Workflow (ID 1434913) begin to get_node_info for host "abc.com" 07/29 21:04:36.716 (tid 0x70005a0): **** Error communicating with host abc.com: error occurred in the SSL/TLS handshake. 07/29 21:04:36.723 (tid 0x70005a0): Workflow (ID 1434913) detected host "abc.com" is unreachable. No data collection is performed. 07/29 21:04:36.733 (tid 0x70005a0): WARNING: ems_post_event: Failed to initialize event: Incompatible managed system version. EVT- OBJ::SystemName=abc.com EVT-INFO::DetectedVersion=
Resolution
Below are the troubleshooting steps that can be followed to resolve the issue. Error "error occurred in the SSL/TLS handshake" is the result of security hardening for later DDMC releases. The DDMC/DDOS combination may be a supported per the matrix, but does not work due to the security change. KB article "Does DDOS/DDMC support TLS versions 1.1 and 1.2?" has all the technical details (A Dell Support account is required to view this article). The problem occurs when using DDMC 6.1 to manage DDs in versions older than DDOS 5.7.4.0. It is resolved by upgrading the managed DD to DDOS 5.7.4.0 or later.
For other possible causes of problems, follow the troubleshooting steps below:
-
Check the connectivity between DD and DDMC using "ping" and "net lookup" commands both ways.
-
Add appropriate host entries if required to make ping and lookup successful.
-
From the DDMC, also check connection to DD by running the below command:
#managed-system check-connection <DD Hostname>
-
Access SE mode and Check 3009 port is open both ways by using telnet:
On both DDR and DDMC:
Access SE mode by opening an SSH command-line connection [with putty for example]NOTE: "SE" commands have been deprecated in DDOS versions 7.7.5.25, 7.10.1.15, 7.13.0.15, 6.2.1.110 and above and are accessible only by Dell employees.- On DD:
# se telnet <DDMC IP> 3009
- On DDMC:
# se telnet <DD IP> 3009
Example of telnet connecting. Connection closed by foreign host is expected, since DD OS does not allow telnet.
se telnet 172.18.50.132 3009 Trying 172.18.50.132... Connected to 172.18.50.132. Escape character is '^]'. Connection closed by foreign host.
- On DD:
-
Compare the fingerprint that DDMC is fetching while adding DD to DDMC with that of CA certificate of DD.
DDMC should pick up the correct DD fingerprint.
SE@phxdd01#adminaccess certificate show detailed Type: host Cert Type: Host Certificate Application: https Subject/Issued To: abc.com Issued By: abc.com Valid From: Sat Aug 1 01:30:36 2015 Valid Until: Wed Jul 25 08:30:36 2046 Fingerprint: 7F:81:11:BC:F5:10:40:83:68:87:81:F5:97:77:EF:6C:EF:02:74:82 Type: ca Cert Type: Root CA Application: trusted-ca Subject/Issued To: abc.com Issued By: abc.com Valid From: Sun Aug 2 08:30:36 2015 Valid Until: Wed Jul 25 08:30:36 2046 Fingerprint: F1:D2:22:95:7C:45:C9:69:CB:76:25:18:C7:33:30:43:7A:CA:98:B9 SE@DDMC01## managed-system add abc.com force The SHA1 fingerprint for the remote host's CA certificate is F1:D2:22:95:7C:45:C9:69:CB:76:25:18:C7:33:30:43:7A:CA:98:B9 Do you want to trust this certificate? Are you sure? (yes|no) [no]: yes
-
On DD, Check the Hostnames for Host and CA certificate under the Subject column. It should be the same unlike below:
tassos1@jaxdd01# hostname The Hostname is: pqr.com assos1@jaxdd01# adminaccess certificate show Subject Type Application Valid From Valid Until Fingerprint ---------------------------------- ---- ----------- ------------------------ ------------------------ --------------------------------------- pqr.com host https Sun Dec 8 12:16:08 2013 Wed Nov 30 18:16:08 2044 2A:21:3E:1E:43:C9:77:F7:20:EF:E5:DF:D9:C9:9A:F8:4C:33:5E:0B pqr.ent.com ca trusted-ca Wed Feb 22 12:41:58 2012 Sat Feb 14 12:41:58 2043 AE:AF:8A:E9:0D:0C:F3:53:B5:A7:BF:D8:38:BC:2D:DA:CF:E5:E9:C8 ---------------------------------- ---- ----------- ------------------------ ------------------------ ---------------------------------------
If a mismatch is present, as in the above output or a certificate is expired, then regenerate the certificate on DD.
See KB Data Domain: Web UI Inaccessible Due to Expired https Certificate.# ddsh -a adminaccess certificate show Subject Type Application Valid From Valid Until Fingerprint -------------------------- ---- ----------- ------------------------ ------------------------ ------------------------------------------ pqr.com host https Sat Aug 8 06:39:31 2015 Wed Aug 1 10:39:31 2046 D5:26:79:20:3A:2F:73:41:7E:A8:5C:9B:69:54:11:8B:33:E9:BD:D9 pqr.com ca trusted-ca Sun Aug 9 11:39:31 2015 Wed Aug 1 10:39:31 2046 02:A0:F7:49:E1:16:BC:8E:FD:47:E4:24:C3:AE:45:7D:B1:8B:0C:3D -------------------------- ---- ----------- ------------------------ ------------------------ -----------------------------
-
On DDMC, verify that all valid hostnames are added as managed systems and under trust.
#adminaccess trust show #managed-system show
Compare the outputs of both the commands above and see if there is any mis-match.
Trust for invalid DD hostnames must be deleted from the DDMC.Run on DDMC
Remove DDR trust, run this CLI Command:#adminaccess trust del host <Data Domain Hostname> type mutual
Run on Data Domain
#adminaccess trust del host <DDMC hostname> type mutual
-
Now try to re-add the Data Domain to DDMC using CLI with force option
#managed-system add <DD Hostname> force
-
"Sync" command can be used anytime to sync managed systems on DDMC:
#managed-system sync #managed-system show