IDPA System Manager root password goes Out of Sync frequently

1. Verify on DPC.

   ssh to DPC from ACM and by a separate putty session using admin and correct password
   Switch to root by below command:

   #su -

   If able to log in to DPC using admin/root successfully, then proceed below.

2. Check the logs on the ACM
   /usr/local/dataprotection/var/configmgr/server_data/logs/server.log:

   //Search for the below string to see when and how many times the DPC root password has gone out of sync:

   #grep -i "Setting dpcRootPasswordOutOfSync=true" server.log*
   server.log.18:2020-09-22 16:10:26,459 INFO [Thread-3200180]-dpcadapter.DPCUtil: Setting dpcRootPasswordOutOfSync=true
   server.log.7:2020-10-02 18:28:25,577 INFO [Thread-40894]-dpcadapter.DPCUtil: Setting dpcRootPasswordOutOfSync=true

   //Check for the Thread noted above to see following symptoms in logs:

   #grep -E  "Thread-3200180" server.log*

   • ACM Requests services status which comes fine:

     server.log.18:2020-09-22 14:09:55,332 INFO [Thread-3200180]-util.SSHUtil: Remote command using SSH execution status: Host : [10.199.193.14] User : [admin] Pass word : [**********] Command : [/usr/local/dpc/bin/dpc status] STATUS : [0]
     server.log.18:2020-09-22 14:09:55,332 INFO [Thread-3200180]-util.SSHUtil: STDOUT : [Version: 18.2.0-13
     server.log.18:2020-09-22 14:09:55,332 INFO [Thread-3200180]-util.SSHUtil: STDERR : []

   • Confirms that the password is synchronized:

     server.log.18:2020-09-22 14:09:55,340 INFO [Thread-3200180]-dao.DPCConfigDAOImpl: Successfully retrieved DPC configuration
     server.log.18:2020-09-22 14:09:55,340 INFO [Thread-3200180]-dpcadapter.DPCUtil: dpcRootPasswordOutOfSync=false
     server.log.18:2020-09-22 14:09:55,340 INFO [Thread-3200180]-dpcadapter.DPCUtil: dpcAdminPasswordOutOfSync=false

   • Fetches DPC Creds again and tries to SSH to DPC:

     server.log.18:2020-09-22 14:09:55,469 INFO [Thread-3200180]-util.SSHUtil: Creating session using SSH parameters: Host : [10.199.193.14] User : [admin] Pas sword : [**********]
     
     server.log.18:2020-09-22 14:09:55,469 INFO [Thread-3200180]-util.SSHUtil: Connecting to host [10.199.193.14] using provided credentials.
     server.log.18:2020-09-22 14:09:55,546 INFO [Thread-3200180]-util.SSHUtil: Connected to host [10.199.193.14] using provided credentials.
     server.log.18:2020-09-22 14:09:55,547 INFO [Thread-3200180]-util.SSHUtil: Opening a channel for communication.
     server.log.18:2020-09-22 14:09:55,547 INFO [Thread-3200180]-util.SSHUtil: Channel for communication opened successfully.
     server.log.18:2020-09-22 14:09:55,701 INFO [Thread-3200180]-util.SSHUtil: Expected Message:: ssword
     ....
     server.log.18:2020-09-22 14:10:25,732 ERROR [Thread-3200180]-util.SSHUtil: Error in command execution Expect operation fails (timeout: 30000 ms) for matcher: contains('ssword')

   • Channel remains open starting at 14:10:25

     2020-09-22 14:10:25,732 INFO [Thread-3200180]-util.SSHUtil: Waiting for channel close

   • Channel closed after 2 hours:

     server.log.18:2020-09-22 16:10:26,458 INFO [Thread-3200180]-util.SSHUtil: Disconnecting channel forcefully.
     server.log.18:2020-09-22 16:10:26,458 INFO [Thread-3200180]-util.SSHUtil: Channel disconnected successfully.
     server.log.18:2020-09-22 16:10:26,458 INFO [Thread-3200180]-util.SSHUtil: Processing the ssh command execution results standard output.
     server.log.18:2020-09-22 16:10:26,458 INFO [Thread-3200180]-util.SSHUtil: Processing the ssh command execution standard error.
     server.log.18:2020-09-22 16:10:26,459 INFO [Thread-3200180]-util.SSHUtil: Remote command using SSH execution status: Host : [10.199.193.14] User : [admin] Password : [**********] Command : [su -c 'date'] STATUS : [-1]

   • Marks it out of sync:

     server.log.18:2020-09-22 16:10:26,459 INFO [Thread-3200180]-dpcadapter.DPCUtil: DPC root password is out of sync, throwing exception
     server.log.18:2020-09-22 16:10:26,459 INFO [Thread-3200180]-dpcadapter.DPCUtil: Setting dpcRootPasswordOutOfSync=true

When DPC becomes slow to respond (maybe due to some jobs or activities), the command fired for checking the password sync does not get the prompt in expected time (which causes a command execution timeout) which results in failing scenarios.

Engineering has fixed this Issue in IDPA 2.7.0 code.