Avamar: An MCS restore failure is caused by hidden files in the /usr/local/avamar/var/mc/server_data/postgres/data/ directory

A Management Console Server (MCS) restore is failing:

admin@avamar-hostname:/usr/local/avamar/var/>: dpnctl start mcs --force_mcs_restore

Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: Restoring MCS data...
dpnctl: ERROR: error return from "[ -r /etc/profile ] && . /etc/profile ; echo 'Y' | /usr/local/avamar/bin/mcserver.sh --restore --id='root' --hfsport='27000' --hfsaddr='11.12.13.14' --password='*************'" - exit status 2
dpnctl: ERROR: 2 errors seen in output of "[ -r /etc/profile ] && . /etc/profile ; echo 'Y' | /usr/local/avamar/bin/mcserver.sh --restore --id='root' --hfsport='27000' --hfsaddr='11.12.13.14' --password='*************'"
dpnctl: ERROR: MCS restore did not succeed, so not restarting MCS
dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

The /usr/local/avamar/var/log/dpnctl.log shows the following error:

2021/10/19-14:09:03 Missing /usr/local/avamar/var/mc/server_data/postgres/data/root.crtMissing /usr/local/avamar/var/mc/server_data/postgres/data/root.srlMissing /usr/local/avamar/var/mc/server_data/postgres/data/server.crtMissing /usr/local/avamar/var/mc/server_data/postgres/data/server.keyMissing /usr/local/avamar/var/mc/server_data/postgres/data/server.reqInitializing the database
2021/10/19-14:09:03 initdb: directory "/usr/local/avamar/var/mc/server_data/postgres/data" exists but is not empty
2021/10/19-14:09:03 It contains a dot-prefixed/invisible file, perhaps due to it being a mount point.
2021/10/19-14:09:03 Using a mount point directly as the data directory is not recommended.
2021/10/19-14:09:03 Create a subdirectory under the mount point.
2021/10/19-14:09:03 sh: -c: line 0: unexpected EOF while looking for matching `"'
2021/10/19-14:09:03 sh: -c: line 1: syntax error: unexpected end of file
2021/10/19-14:09:03 /bin/chmod: cannot access '/usr/local/avamar/var/mc/server_data/postgres/data/pg_hba.conf': No such file or directory
2021/10/19-14:09:03 Can't open /usr/local/avamar/var/mc/server_data/postgres/data/pg_hba.conf

This is caused by hidden files that were created by mistake in the postgres database configuration folder.

This can be confirmed by looking a few lines above the error in /usr/local/avamar/var/log/dpnctl.log:

2021/10/19-14:09:03 avtar Info <5314>: Command completed (1 warning, exit code 0: success)
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/adminlogpattern.xml
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/customerContactInfo.xml
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/ddr_rest_keystore
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/flush.timestamp
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/init.timestamp
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/mcs_data_dump.sql
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/datastore_v1_0/data.dat
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/datastore_v1_0/freespace.dat
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/datastore_v1_0/header.dat
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/datastore_v1_0/schema.dat
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/datastore_v1_0/
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/postgres/data/.pg_hba.conf.swo 
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/postgres/data/.pg_hba.conf.swp 
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/postgres/data/PG_VERSION
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/postgres/data/current_logfiles
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/postgres/data/pg_hba.conf
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/postgres/data/pg_ident.conf
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/postgres/data/postgresql.auto.conf
2021/10/19-14:09:03 /usr/local/avamar/var/mc/server_data/postgres/data/postgresql.conf

To resolve the issue:

1. Create the file /usr/local/avamar/var/avtar.cmd (or edit it if it already exists).

vi /usr/local/avamar/var/avtar.cmd

2. Add the following line to exclude all the hidden files from /usr/local/avamar/var/mc/server_data/postgres/data/

--exclude=var/mc/server_data/postgres/data/.*

3. Try another MCS restore:

dpnctl start mcs --force_mcs_restore