Avamar EMS and DTLT services cannot be restarted due to empty or malformed emserver.xml configuration file

This has been observed when EMS or MCS restores have been performed, a node was replaced, or a rollback performed.

After bringing down the Avamar services to perform maintenance, all the services could come back online apart from EMS and DTLT.

dpnctl status

Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: EMS status: down.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: dtlt status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

The DPN control log (/usr/local/avamar/var/log/dpnctl.log) shows the following sequence of events and reports an error:

less /usr/local/avamar/var/log/dpnctl.log

2017/11/02-12:52:56 dpnctl: INFO: - - - - - - - - - - - - - - - BEGIN
2017/11/02-12:52:56
2017/11/02-12:52:56 no element found at line 1, column 0, byte 0:
2017/11/02-12:52:56
2017/11/02-12:52:56 ^
2017/11/02-12:52:56  at /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi/XML/Parser.pm line 187
2017/11/02-12:52:56 dtltctl: INFO: Desktop/laptop services status: down.
2017/11/02-12:52:56 dpnctl: INFO: - - - - - - - - - - - - - - - END
2017/11/02-12:52:56 /bin/cat /tmp/dpnctl-dtlt-status-status-30477 2>&1
2017/11/02-12:52:56 [ "/bin/cat /tmp/dpnctl-dtlt-status-status-30477 2>&1" exit status = 0 ]
2017/11/02-12:52:56 dpnctl: INFO: "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/../lib/dpnutils/dtltctl status" exit status = 1
2017/11/02-12:52:56 rm -f /tmp/dpnctl-dtlt-status-status-30477 /tmp/dpnctl-dtlt-status-output-30477
2017/11/02-12:52:56 dpnctl: INFO: "rm -f /tmp/dpnctl-dtlt-status-status-30477 /tmp/dpnctl-dtlt-status-output-30477" - exit status 0
2017/11/02-12:52:56 dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
2017/11/02-12:52:56 rm -f /usr/local/avamar/var/dpnctl.lck
2017/11/02-12:52:56 ======= dpnctl 7.0.2-43 (1.139), running as admin, RETURNING at 2017-11-02 18:22:56 IST =======
2017/11/02-12:52:56 [user "admin"] program (pid 30477) exit status = 1 (error)
2017/11/02-12:52:56 ======= dpnctl 7.0.2-43 (1.139), running as admin, ENDING at 2017-11-02 18:22:56 IST =======
2017/11/02-12:52:56 [user "admin"] program (pid 30403) exit status = 1 (error)

• Attempting a manual flush of the EMS database fails with the error "no element found at line 1, column 0, byte 0:"

emserver.sh --flush

no element found at line 1, column 0, byte 0:

^
at /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi/XML/Parser.pm line 187

An attempt to start the EMS service with the "verbose" flag may reveal the cause.

emserver.sh --start --verbose

args = --start --verbose
---------------------------------
home_dir=/usr/local/avamar
db_dir=/usr/local/avamar/var/em/server_data/postgres/data
data_dir=/usr/local/avamar/var/em/server_data
postgres_dir=/usr/local/avamar/var/em/server_data/postgres
prefs_dir=/usr/local/avamar/var/em/server_data/prefs
mc_dir=/usr/local/avamar/var/mc
log_dir=/usr/local/avamar/var/em/server_log
lib_dir=/usr/local/avamar/lib
bin_dir=/usr/local/avamar/bin
---------------------------------

Processing /usr/local/avamar/lib/emserver.xml
Processing /usr/local/avamar/var/em/server_data/prefs/emserver.xml

no element found at line 1, column 0, byte 0:

^
at /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi/XML/Parser.pm line 187

The EMS service configuration file /usr/local/avamar/var/em/server_data/prefs/emserver.xml is empty. 

ls -thrl /usr/local/avamar/var/em/server_data/prefs/ | grep emserver

-rwxrwx--- 1 admin admin    0 Aug 10 16:05 emserver.xml

The EMS service configuration file /usr/local/avamar/var/em/server_data/prefs/emserver.xml is malformed.

admin@avamar:/usr/local/avamar/var/em/server_data/prefs> ls -l emserver.xml
-rw-r----- 1 admin admin 9497 Sep 27  2021 emserver.xml

admin@avamar:/usr/local/avamar/var/em/server_data/prefs> cat emserver.xml
admin@avamar:/usr/local/avamar/var/em/server_data/prefs>

admin@avamar:/usr/local/avamar/var/em/server_data/prefs> vi emserver.xml
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

 

Must find a valid copy of emserver.xml from an old EM_BACKUPS flush.

Use avtar to find EM_BACKUPS flushes from the past.
A count of flushes to list can be specified in case there are thousands.
Leave out the count flag to list all EM_BACKUPS.

avtar --backups --path=/EM_BACKUPS --count=500

Select a label number and restore to a temporary directory to inspect the old emserver.xml copy.

mkdir ~admin/em_redirected_restore
cd ~admin/em_redirected_restore
avtar -x --path=/EM_BACKUPS --labelnum=<selected_label_number> --target=.

Inspect the emserver.xml copy to see if it is valid.

cat var/em/server_data/prefs/emserver.xml

Optionally use xmllint to validate the xml file.

xmllint var/em/server_data/prefs/emserver.xml --noout

Once the emserver.xml file from an old flush has been validated, copy it to the proper location.

cp ~admin/em_redirected_restore/var/em/server_data/prefs/emserver.xml /usr/local/avamar/var/em/server_data/prefs/emserver.xml

Update permissions and ownership.

chown admin:admin /usr/local/avamar/var/em/server_data/prefs/emserver.xml
chmod 640 /usr/local/avamar/var/em/server_data/prefs/emserver.xml

Start EMT.

dpnctl start emt