IDPA: ChangeAvamarPasswords failed to execute change-passwords command on Avamar Utility Node

• [FAILED] Backup Server - Failed to change password for Avamar OS user.
  [FAILED] Backup Server - Failed to change password for Avamar server user. 

• All other passwords update successfully
• ACM /usr/local/dataprotection/var/configmgr/server_data/logs/server.log file reports error:     

ERROR [Thread-256364]-avadapter.AvamarUtil: changeAvamarPasswords -->Failed to execute change-passwords command on avamar utility node

• MCUser
• viewuser
• root
• repluser

 

1. Log in to the ACM and verify components show green. If there is any component that is not showing green, that issue needs to be fixed first
2. Using an SSH client such as putty.exe, SSH to the Avamar Utility Node as user admin. The password is the original password prior to when the password change was invoked from the ACM
3. Issue command "unset TMOUT" to stop the admin SSH session from timing out
4. Switch user to user root - issue command:  "su -"
5. Issue command "unset TMOUT" to stop the root SSH session from timing out
6. duplicate the SSH session as user admin to the to the Avamar Utility Node and then switch user to user root, i.e. repeat steps 2,3,4 & 5. This is important,  as a precaution, to ensure you do not get locked out from the Avamar Utility Node after changing the OS root and admin user passwords.
7. From this second SSH session to the Avamar Utility Node, issue command: "change-passwords"  and follow the prompts.  
8. For the very first prompt, answer no:     

root@ave:~/#: change-passwords
[change-passwords version 1.32]
Identity added: /root/.ssh/rootid (/root/.ssh/rootid)
Identity added: /root/.ssh/rootid (/root/.ssh/rootid)
 
Do you wish to specify one or more additional SSH passphrase-less
    private keys that are authorized for root operations?
Answer n(o) here unless there are known inconsistencies in
    ~root/.ssh/authorized_keys files among the various nodes.
Note that the following key will be used automatically (i.e., there is
    no need to re-specify it here):
      /root/.ssh/rootid

y(es), n(o), h(elp), q(uit/exit): no

9. The remaining prompts asks to update the OS admin and root passwords. Once that concludes, it prompts to update the additional four internal Avamar user passwords for users, MCUser, root, repluser and viewuser. In addition, it asks to update the SSH keys for the OS root users. Be sure to select yes to update the SSH keys for the OS users root and admin. Below are the remaining updates:    

--------------------------------------------------------
The following is a test of OS root authorization with the currently
    loaded SSH key(s).

    If the authorization test fails, then you might be missing an
    appropriate private key, e.g., rootid or dpnid.
        -> In that event, re-run this program and, when prompted,
           specify as many SSH private key files as are necessary
           in order to complete root operations.

Starting root authorization test with 600 second timeout...
End of root authorization test.
--------------------------------------------------------

Change OS (login) passwords?
y(es), n(o), q(uit/exit): yes
change-passwords: INFO: Each OS password will be changed locally without further prompting as soon as you have (twice) entered a valid password.

--------------------------------------------------------
Change OS password for "admin"?
y(es), n(o), q(uit/exit): yes
Enter a new OS (login) password for user "admin".

(Entering an empty (blank) line twice quits/exits.)
>
Enter the same OS password again.

(Entering an empty (blank) line twice quits/exits.)
>
Changing password for admin.
change-passwords: INFO: the password for OS user admin has been updated on _this_ host.
change-passwords: INFO: the password will not be reverted if you later decline to update passwords/passphrases.
Accepted OS password for "admin".

--------------------------------------------------------
Change OS password for "root"?
y(es), n(o), q(uit/exit): yes
Enter a new OS (login) password for user "root".

(Entering an empty (blank) line twice quits/exits.)
>
Enter the same OS password again.

(Entering an empty (blank) line twice quits/exits.)
>
Changing password for root.
change-passwords: INFO: the password for OS user root has been updated on _this_ host.
change-passwords: INFO: the password will not be reverted if you later decline to update passwords/passphrases.
Accepted OS password for "root".

--------------------------------------------------------
Generate new SSH keys?
y(es), n(o), h(elp), q(uit/exit): yes

--------------------------------------------------------
Change Avamar Server passwords?
y(es), n(o), q(uit/exit): yes

--------------------------------------------------------
Please enter the CURRENT server password for "root"

(Entering an empty (blank) line twice quits/exits.)
>
Checking Avamar Server root password (1200 second timeout)...
Avamar Server current root password accepted.

--------------------------------------------------------
Change Avamar Server password for "MCUser"?
y(es), n(o), q(uit/exit): yes
Please enter a new Avamar Server password for user "MCUser".

(Entering an empty (blank) line twice quits/exits.)
>
Enter the same Avamar Server password again.

(Entering an empty (blank) line twice quits/exits.)
>
Accepted Avamar Server password for "MCUser".

--------------------------------------------------------
Change Avamar Server password for "root"?
y(es), n(o), q(uit/exit): yes
Please enter a new Avamar Server password for user "root".

(Entering an empty (blank) line twice quits/exits.)
>
Enter the same Avamar Server password again.

(Entering an empty (blank) line twice quits/exits.)
>
Accepted Avamar Server password for "root".

--------------------------------------------------------
Change Avamar Server password for "repluser"?
y(es), n(o), q(uit/exit): yes
Please enter a new Avamar Server password for user "repluser".

(Entering an empty (blank) line twice quits/exits.)
>
Enter the same Avamar Server password again.

(Entering an empty (blank) line twice quits/exits.)
>
Accepted Avamar Server password for "repluser".

--------------------------------------------------------
Change the viewuser password?
y(es), n(o), h(elp), q(uit/exit): yes
Checking Administrator Server status...
EUA2-Enter the NEW viewuser password.
Enter ? or help for help.

(Entering an empty (blank) line twice quits/exits.)
>
For verification, re-enter the NEW viewuser password.
Enter ? or help for help.

(Entering an empty (blank) line twice quits/exits.)
>

ERROR: your entries for "viewuser" did not match.
        Please try again.
Enter the NEW viewuser password.
Enter ? or help for help.

(Entering an empty (blank) line twice quits/exits.)
>
For verification, re-enter the NEW viewuser password.
Enter ? or help for help.

(Entering an empty (blank) line twice quits/exits.)
>

--------------------------------------------------------
Do you wish to proceed with your changes on the selected node?
        Answering y(es) will proceed to make changes.
        Answering n(o) or q(uit) will not proceed.

y(es), n(o), q(uit/exit): yes
Changing OS passwords...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done changing OS passwords...
Changing Avamar Server passwords...
Suspending maintenance cron jobs
Checking Administrator Server status...
Stopping Administrator Server...
Starting process of updating Administrator and Enterprise Manager configurations...
Running script to update Administrator and Enterprise Manager configurations on node 0.s...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating Administrator configuration on node 0.s...
Starting process of updating client configurations...
Running script to update client configuration on all+...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Updating client configuration on node 0.0...
Done updating client configuration on 0.0...
Starting process of updating mccli configuration files...
Running script to update mccli configuration files on node set "0.0"...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating mccli configuration files on node 0.0...
Checking Administrator Server status...
Starting Administrator Server...
Stopping dtlt (DT/LT) subsystem
Starting dtlt (DT/LT) subsystem
Resuming maintenance cron jobs
Starting process of changing SSH keys...
Running script to update SSH keys on node 0.s...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating SSH keys on node 0.s...
Starting process of updating viewuser password...
Checking Administrator Server status...
Stopping Administrator Server...
Running script to update mcdb viewuser password on node 0.0...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating mcdb viewuser password on node 0.0...
Checking Administrator Server status...
Starting Administrator Server...

--------------------------------------------------------
Done.
NOTES:
- If you had custom public keys present in the
      authorized_keys files of any Avamar OS users
      (admin, root) be aware that
      you may need to re-add your custom keys.
- If mccli (the Administrator command line interface)
      is used from any remote user accounts, then please update
      the password in each remote account's copy of the mccli
      preferences/configuration file, typically
      ~USER/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml.
- Please be sure to resume schedules via the
        Administrator GUI or via 'dpnctl start sched'.
root@idpa-avamar-lj:~/#:

10. As per the messages, it is important to confirm the the mccli preferences configuration file did get updated. This is best tested by issue a mccli command as user admin and as user root. Below is an example as user root. Both commands should work. If they do not, then update the ~USER/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml file:     
     
    root@idpa-avamar-lj:~/#: mccli user show
    0,23000,CLI command completed successfully.
    Name          Role                     Domain Authenticator
    ------------- ------------------------ ------ ---------------------------
    MCUser        Administrator            /      Axion Authentication System
    backuponly    Back up Only User        /      Axion Authentication System
    backuprestore Back up/Restore User     /      Axion Authentication System
    repluser      Replication User         /      Axion Authentication System
    restoreonly   Restore (Read) Only User /      Axion Authentication System
    root          Administrator            /      Axion Authentication System
    
    root@idpa-avamar-lj:~/#:
    
    
    	

11. Additionally, as per the message, it is also important to re-start the backup scheduler as this is disabled when the MCS service is stopped and restarted during the password change:      

11. Finally, verify that the OS admin and root user passwords updated successfully. To do this, open a new SSH session to the Avamar Backup server, log on as user admin with the updated common password. Once successfully logged on, switch user to user root using the "su -" and enter the same updated common password. Once both users have been verified,
12. Switch over to the ACM dashboard and refresh the screen. The following message should be seen. Click on the message and enter the same common password used for the admin user and save the password. Once successfully saved, a message "Backup Server, password updated successfully" should be seen:     

13. Once the backup server password has been updated successfully, the following second message is seen. Click on the message and enter the same common password used for the admin user and save the password. Once successfully saved, a message that the password was updated successfully is seen:

14. At this point, the ACM dasboard should show all components as  green. This concludes the workaround for the OS password for the DP4400 AVE issue