Avamar: Error "Exit Status 1" When Trying to Start Avamar Backup Scheduler

The Avamar Utility Node has been replaced. 

Restarting the Avamar backup scheduler results in an error:

admin@Lab-AVE:~/>: dpnctl start sched
dpnctl: INFO: Resuming backup scheduler...
dpnctl: ERROR: error return from "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mccli mcs resume-scheduler" - exit status 1
dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"] 

MCCLI commands fail with messages like these:

admin@Lab-AVE:~/>: mccli server show-services
1,22801,User login failure. 

admin@Lab-AVE:~/>: mccli server show-services
1,22801,User login failure.

The MCUser password in the mcclimcs.xml files differs from the password in the /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml file.

To find versions of mcclimcs.xml, run a find command as root:

root@Lab-AVE:~/#: find / -name "mcclimcs.xml" -print

Example outputs: 

Virtual Node:

/space/home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml 
/usr/local/avamar/lib/mcclimcs.xml
/root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

Physical Node:

/data01/home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml 
/usr/local/avamar/lib/mcclimcs.xml
/root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

1. Confirm that the encrypted MCUser password in mcclimcs.xml files does not match that of mcserver.xml:

root@Lab-AVE:~/#: grep MCUSERAP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="MCUSERAP" value="{AES}8gjyd5nnLnCa3e1cbNBAvg==" />

root@Lab-AVE:~/#: grep mcspasswd /space/home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml /usr/local/avamar/lib/mcclimcs.xml /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml |grep -v password
/space/home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml:       mcspasswd="{AES}abcdd5nnLnCa3e1cbNBAva=="
/usr/local/avamar/lib/mcclimcs.xml:       mcspasswd="{AES}abcdd5nnLnCa3e1cbNBAva=="
/root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml:       mcspasswd="{AES}abcdd5nnLnCa3e1cbNBAva==" 

2. Create a backup copy of the mcclimcs.xml file, existing in /usr/local/avamar/lib/ (the file is the same in all three locations, one backup copy is enough):

root@Lab-AVE:~/#: cp -p /usr/local/avamar/lib/mcclimcs.xml /usr/local/avamar/lib/orig_mcclimcs.xml_`date -I`

3. Verify that the backup copy exists:

root@Lab-AVE:~/#: ls -lah /usr/local/avamar/lib/ |grep mcclimcs.xml
-rw-r-----  1 root  admin  1.2K Nov  1 15:00 mcclimcs.xml
-rw-r-----  1 root  admin  1.2K Nov  1 15:00 orig_mcclimcs.xml_2025-03-26 

4. Use sed to edit each of the mcclimcs.xml files, replacing the incorrect encrypted password with the password copied from mcserver.xml.  

Example commands from the sample files above.

root@Lab-AVE:~/#: sed -i '/mcspasswd/s/"{AES}abcdd5nnLnCa3e1cbNBAva=="/"{AES}8gjyd5nnLnCa3e1cbNBAvg=="/g' /space/home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

root@Lab-AVE:~/#: sed -i '/mcspasswd/s/"{AES}abcdd5nnLnCa3e1cbNBAva=="/"{AES}8gjyd5nnLnCa3e1cbNBAvg=="/g' /usr/local/avamar/lib/mcclimcs.xml

root@Lab-AVE:~/#: sed -i '/mcspasswd/s/"{AES}abcdd5nnLnCa3e1cbNBAva=="/"{AES}8gjyd5nnLnCa3e1cbNBAvg=="/g' /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

5. Verify that the encrypted password matches in all files:

root@Lab-AVE:~/#: grep MCUSERAP /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
                <entry key="MCUSERAP" value="{AES}8gjyd5nnLnCa3e1cbNBAvg==" 

root@Lab-AVE:~/#: grep mcspasswd /space/home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml /usr/local/avamar/lib/mcclimcs.xml /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml |grep -v password
/space/home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml:       mcspasswd="{AES}8gjyd5nnLnCa3e1cbNBAvg=="
/usr/local/avamar/lib/mcclimcs.xml:       mcspasswd="{AES}8gjyd5nnLnCa3e1cbNBAvg=="
/root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml:       mcspasswd="{AES}8gjyd5nnLnCa3e1cbNBAvg==" 

6. The commands should now complete without issue:

admin@Lab-AVE:~/>: dpnctl status sched
dpnctl: INFO: Backup scheduler status: down. 

admin@Lab-AVE:~/>: dpnctl start sched
dpnctl: INFO: Resuming backup scheduler...
dpnctl: INFO: Backup scheduler resumed.
dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist. 

admin@Lab-AVE:~/>: dpnctl status sched
dpnctl: INFO: Backup scheduler status: up. 

admin@Lab-AVE:~/>: mccli server show-prop |egrep "Attrib|---|State|Sys|HFSAddr"
Attribute                                    Value
-------------------------------------------- ----------------------------
State                                        Full Access
System Name                                  LAB-AVE.company.com
System ID                                    123456789079@00:0C:29:2C:E8:F2
HFSAddr                                      Lab-AVE 

admin@Lab-AVE:~/>: mccli server show-services |egrep "Attrib|---|Hostname|Services"
---------------------------------- --------------------------
Hostname                           Lab-AVE.company.com
Web Services                       Running

If required, below is the list of the correct permissions and ownership for each of the mcclimcs.xml files:

ls -lah /space/home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml
-rw-r----- 1 admin admin 1.2K Mar 26 17:04 /space/home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

ls -lah /usr/local/avamar/lib/mcclimcs.xml
-rw-r----- 1 root admin 1.2K Mar 26 17:05 /usr/local/avamar/lib/mcclimcs.xml

ls -lah /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml
-rw-rw---- 1 root root 1.2K Mar 26 17:05 /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml