IDPA: RUCK error ACM is not able to establish an SSH connection
Summary: This KB covers how to resolve SSH connection issues detected by RUCK.
Symptoms
When running the Rapid Upgrade Checker (RUCK) tool, you get an error like the following:
ACM is not able to establish an SSH connection with server. Ensure that every component is reachable, powered-on and password is in sync.
You see one or more of the following check codes:
dd_sysadmin_test_connection
av_test_connection_admin
dpc_test_connection_admin
test_connection_dpa_app_server
test_connection_dpa_agent
test_connection_dpa_datastore
dps_test_connection_master
cdra_test_connection
vc_test_connection_root
vc_test_connection_administrator
Cause
There are several issues that can cause these errors.
- The VM could be powered off.
- One of the more services on the server may not be running.
- There may be a network connection issue.
- Passwords may not be in sync on the ACM.
Resolution
./goidpa system port-check
If it cannot connect to the system from ACM it checks the power state of the VM and prompt if you want to turn it on, if it is powered off.
If it can connect it will proved two tables as follows:
CONNECTION TABLE has the following:
- VM
- Port that was checked
- Status of port, Reachable/Not Reachable
PROCESS TABLE has the following:
- VM
- Port that process is running only
- Port Status, Listening/not Listening
CONNECTIONS TABLE
+---------+------+-------------+
| VM | Port | Port Status |
+---------+------+-------------+
| DPA-APP | 22 | Reachable |
| DPA-APP | 9002 | Reachable |
| DPA-APP | 3741 | Reachable |
+---------+------+-------------+
PROCESS TABLE
+---------+------+-------------+
| VM | Port | Port Status |
+---------+------+-------------+
| DPA-APP | 22 | Listening |
| DPA-APP | 9002 | Listening |
| DPA-APP | 3741 | Listening |
+---------+------+-------------+
The following instructions are provided to resolve any issues found on these tables:
- If all ports in the CONNECTION TABLE are 'Reachable' all port connections are good.
- If a port in the CONNECTION TABLE is 'BLOCKED', then check that port in the PROCESS TABLE to see if that port is 'Listening'.
- If a port is 'BLOCKED' but 'Listening' then this is a network connection issue.
- If a port is 'Not Listening' then there are services not running on the server.
For network connection issues test with the following command:
curl -kv <IP>:<port>
Where <IP> is the IP address of the system that is BLOCKED and <port> is the port number listed for the BLOCKED connection.
For services not running, check the service status for that port with the following commands:
For the Avamar, log in to the Avamar using SSH/PUTTY as admin and run the following commands:
dpnctl status
---emt status will show the status of the service for port 9443
--- su - to get to root user and then run the following to check service for port 5555
service postgresql status
--- For port 22, you will not be able to connect using SSH/PUTTY, but from the vCenter console as root user run:
service sshd status
For DPA, log in to the correct DPA server using SSH/PUTTY as root and run the following commands:
--- For port 22, you will not be able to connect using SSH/PUTTY, but from the vCenter console as root user run:
service sshd status
For the APP and DS server run the following:
dpa.sh svc status
--- This will show the DataStore or the Application service, this is port 9003 or 9002,
--- It will also show the Agent service, this is port 3741.
For the Collection (Agent) server run the following:
/opt/emc/dpa/etc/dpa status
--- This will show the agent services, this is port 3741
For Search, log in to the Search server using SSH/PUTTY as root and run the following commands:
--- For port 22, you will not be able to connect using SSH/PUTTY, but from the vCenter console as root user run:
service sshd status
---For port 443 check nginx status with the following command:
service nginx status
--- For port 8140 check the puppet status with the following command:
service puppet status
For Data Domain, open a Support Request with Dell support.
For DPC, log in to the DPC server using SSH/PUTTY as admin and su - to root and run the following commands:
--- For port 22, you will not be able to connect using SSH/PUTTY, but from the vCenter console as root user run:
service sshd status
--- For port 443, check the nginx serve with the following command:
service nginx status
--- For port 5671, check RabbitMQ service with the following command:
service rabbitmq-server status
For vCenter (VCSA), log in to the VCSA server using SSH/PUTTY as root and run the following commands:
--- For port 22, you will not be able to connect using SSH/PUTTY, but from the ESXi console as root user run:
service sshd status
--- For port 443, get into the shell prompt and run the following command:
service-control --status
Check if vsphere-ui is listed in the Running: or Stopped: apps.