Integrated Data Protection Appliance - Avamar Password Out of Sync
Summary: Avamar password out of sync shown in IDPA dashboard UI
Symptoms
Here is a Password Out of Sync example:
Protection Software Proxy 'root' user out of sync. Please ensure 'root' password is same as 'admin' password
Cause
Possible causes:
- Password is changed or is reset directly at the Avamar server or proxy end instead of from the ACM dashboard
- Network latency from ACM to Avamar Server or AVProxy to query its status. (temporary issue)
Resolution
If updating the password in ACM UI is unable to resolve the password out of sync issues, following these steps:
Resolutions for Avamar server password out of sync:
Scenario #1:
Password changed from Avamar side instead of from ACM UI:
-
Avamar password out of sync (one or more of the following Avamar user passwords have changed)
- Avamar OS root user password
- Avamar OS admin user password
- Avamar Server root user password
- Avamar Server MCUser user password
- Avamar Server repluser user password
- Avamar PostgreSQL database viewuser user password
- Avamar vProxy OS root user password (Appliance Internal proxy VM)
- Avamar vProxy OS admin user password (Appliance Internal proxy VM)
If you change Avamar passwords from Avamar side instead of from ACM UI, IDPA requires that the Avamar OS admin and OS root accounts use the same password. The MCUser, repluser, viewuser, and Avamar server root user must share a same password which can be different than the OS admin and OS root passwords.
In the above password list, the passwords highlighted in the same color MUST share a same password, and also all passwords have to meet the IDPA global password policy shown as below:
- How to verify Avamar Server and Avamar Proxy machine OS root/admin passwords are the same?
su to root user with the same password to see if the password can log in both admin and root account.
- How to verify the Avamar Server root, MCUser and repluser passwords? (Avamar Server root is different from Avamar Server OS root. OS root is a Linux OS level login user, and Avamar Server root is an Avamar application level user)
# avmgr logn --id=<Avamar_Username> --ap=<password>
1 Request succeeded
7161 privilege level (enabled,create,read,backup,access,move,delete,maint,fullmanage,noticketrequired)
2 block type (directory)
- How to verify the Avamar Server mcdb viewuser password?
# psql -U viewuser -h <Avamar Server IP> -p 5555 mcdb -c "\d"
Password for user viewuser:
- Change Avamar Server-side passwords:
login as: admin
Password: xxxxx
admin@Avamar-svr:~/>: su -
Password: xxxxx
root@Avamar-svr:~/#:
root@Avamar-svr:~/#: change-passwords
[change-passwords version 2.1]
Identity added: /root/.ssh/rootid (/root/.ssh/rootid)
Identity added: /root/.ssh/rootid (/root/.ssh/rootid)
Identity added: /root/.ssh/rootid-save (/root/.ssh/rootid-save)
Do you wish to specify one or more additional SSH passphrase-less
private keys that are authorized for root operations?
Answer n(o) here unless there are known inconsistencies in
~root/.ssh/authorized_keys files among the various nodes.
Note that the following keys will be used automatically (i.e., there is
no need to re-specify them here):
/root/.ssh/rootid
/root/.ssh/rootid-save
y(es), n(o), h(elp), q(uit/exit): no
--------------------------------------------------------
The following is a test of OS root authorization with the currently
loaded SSH key(s).
If the authorization test fails, then you might be missing an
appropriate private key, e.g., rootid or dpnid.
-> In that event, re-run this program and, when prompted,
specify as many SSH private key files as are necessary
in order to complete root operations.
Starting root authorization test with 600 second timeout...
End of root authorization test.
--------------------------------------------------------
Change OS (login) passwords?
y(es), n(o), q(uit/exit): yes
change-passwords: INFO: Each OS password will be changed locally without further prompting as soon as you have (twice) entered a valid password.
--------------------------------------------------------
Change OS password for "admin"?
y(es), n(o), q(uit/exit): yes
Change password for user "admin".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same OS user password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
BAD PASSWORD: it is too simplistic/systematic
Backup lockbox file
Backup keystore files
Backup SSV files
Flush backup
Local backup dir: /usr/local/avamar/src/lockbox_backup/2023-06-26-22_00
Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
Updated with new value under name "admin".
Backup lockbox file
Backup keystore files
Backup SSV files
Flush backup
Local backup dir: /usr/local/avamar/src/lockbox_backup/2023-06-26-22_00
Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
change-passwords: INFO: The password for OS user admin has been updated on _this_ host.
change-passwords: INFO: The password will not be reverted if you later decline to update passwords/passphrases.
--------------------------------------------------------
Change OS password for "root"?
y(es), n(o), q(uit/exit): yes
Change password for user "root".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same OS user password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
BAD PASSWORD: it is too simplistic/systematic
change-passwords: INFO: The password for OS user root has been updated on _this_ host.
change-passwords: INFO: The password will not be reverted if you later decline to update passwords/passphrases.
--------------------------------------------------------
Generate new SSH keys?
y(es), n(o), h(elp), q(uit/exit): no
--------------------------------------------------------
Change Avamar Server passwords?
y(es), n(o), q(uit/exit): yes
--------------------------------------------------------
Please enter the CURRENT server password for "root"
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Checking Avamar Server root password (1200 second timeout)...
Avamar Server current root password accepted.
--------------------------------------------------------
Change Avamar Server password for "MCUser"?
y(es), n(o), q(uit/exit): yes
Please enter a new Avamar Server password for user "MCUser".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same Avamar Server password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Accepted Avamar Server password for "MCUser".
--------------------------------------------------------
Change Avamar Server password for "root"?
y(es), n(o), q(uit/exit): yes
Please enter a new Avamar Server password for user "root".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same Avamar Server password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Accepted Avamar Server password for "root".
--------------------------------------------------------
Change Avamar Server password for "repluser"?
y(es), n(o), q(uit/exit): yes
Please enter a new Avamar Server password for user "repluser".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same Avamar Server password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Accepted Avamar Server password for "repluser".
--------------------------------------------------------
Change the viewuser password?
y(es), n(o), h(elp), q(uit/exit): yes
Checking Administrator Server status...
Enter the NEW viewuser password.
Enter ? or help for help.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
For verification, re-enter the NEW viewuser password.
Enter ? or help for help.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
--------------------------------------------------------
Do you wish to proceed with your changes on the selected node?
Answering y(es) will proceed to make changes.
Answering n(o) or q(uit) will not proceed.
y(es), n(o), q(uit/exit): yes
Changing OS passwords...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done changing OS passwords...
Changing Avamar Server passwords...
Suspending maintenance cron jobs
Checking Administrator Server status...
Stopping Administrator Server...
Changing the passwords for the local Avamar Server...
The passwords for the local Avamar Server have been changed.
Starting process of updating Administrator and Enterprise Manager configurations...
Running script to update Administrator and Enterprise Manager configurations on node 0.s...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating Administrator configuration on node 0.s...
Starting process of updating client configurations...
Running script to update client configuration on all+...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Updating client configuration on node 0.0...
Done updating client configuration on 0.0...
Starting process of updating mccli configuration files...
Running script to update mccli configuration files on node set "0.0"...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating mccli configuration files on node 0.0...
Checking Administrator Server status...
Starting Administrator Server...
Resuming maintenance cron jobs
Starting process of updating viewuser password...
Checking Administrator Server status...
Stopping Administrator Server...
Running script to update mcdb viewuser password on node 0.0...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating mcdb viewuser password on node 0.0...
Checking Administrator Server status...
Starting Administrator Server...
Stopping EMT subsystem
Starting EMT subsystem
--------------------------------------------------------
Done.
NOTES:
- If mccli (the Administrator command line interface)
is used from any remote user accounts, then please update
the password in each remote account's copy of the mccli
preferences/configuration file, typically
~USER/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml.
- Please be sure to resume schedules via the
Administrator GUI or via 'dpnctl start sched'.
#: dpnctl start sched
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: Resuming backup scheduler...
dpnctl: INFO: Backup scheduler resumed.
dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.
- Change Avamar Proxy side passwords:
login as: admin
Password: xxxxx
su -
Password: xxxxx
# passwd admin
New password: xxxxx
BAD PASSWORD: it is too simplistic/systematic
BAD PASSWORD: is too simple
Retype new password: xxxxx
passwd: password updated successfully
# passwd root
New password: xxxxx
BAD PASSWORD: it is too simplistic/systematic
BAD PASSWORD: is too simple
Retype new password: xxxxx
passwd: password updated successfully
Scenario #2
Password out of sync error due to network latency when ACM trying to query its point-products:
-
This is a temporary issue and typically can be resolved if you refresh the ACM page after 1-2 minutes. This is a known issue, and the Dell engineering team is working on a fix in a future release.
Scenario #3
Even though password is synchronized and works on Avamar, ACM shows password out of sync for AV, due to SSH failure or test connection failure to AV. This may be due to ACM failing to log in to AV due to SSH issues like recent changes made on AV sshconfig, cipher negotiation, and so on.
- Run a test SSH connection from ACM to Avamar server. If it fails, log in to Avamar server and restart SSH service:
# service sshd restart
- If this does not help, gather the error message, troubleshooting steps you have performed and raise a ticket with Dell technical support for further assistance.
Scenario #4
Avamar MCUser or viewuser may show out of sync when ACM is unable to perform MCSDK call to Avamar to validate those user passwords. This can happen if the ACM MCSDK call fails to Avamar due to various reasons.
- IDPA: ACM Reports Avamar Server Passwords Out of Sync Following Upgrade to 2.7.1 Upgrade or Application of Avamar 19.4 MCS Hotfix 333618 (Log in as Dell Support registered user is required to view article)
- IDPA: ACM UI shows error "Backup Server viewuser user password is out of sync. Update the latest password" on Backup Server Dashboard
If the above provided scenarios and resolutions are unable to fix the issue, do the following:
- SSH Login to ACM as root, and stop and start ACM web application service:
# service dataprotection_webapp restart
# service dataprotection_webapp statu
- Refresh the ACM web page and login, it shows "Appliance Startup progress." It takes some time to resync up with all the appliance components, and once done it returns to the ACM dashboard. (This is not a process of restarting Appliance)
- If the issue still cannot be resolved, raise a support ticket with Dell Technologies.
Avamar password related KB references: