Avamar: Upgrade fails reporting MCS failed to update the new catalog

During an upgrade, the workflow fails with one of the following errors:

MCS failed to update the new catalog 

Updating Client Plug-in Catalog (nnn of nnn) Failed 

UpdateClientPluginCatalog|Updating Client Plug-in Catalog (nnn of nnn)|ERROR: UpdateClientPluginCatalog::do - operation failed

The workflow.log (/data01/avamar/repo/temp/AvamarUpgrade*/tmp/workflow.log) contains more information about the error:

tail -f workflow.log

...
2020-09-03 16:30:11 (+0530) 22794100 INFO: - - - - - - - END
2020-09-03 16:30:11 (+0530) 22794100 INFO: "rpm -qa --last | grep download-mcs-compat | cut --delimiter=' ' --fields=1 2>/dev/null", exit status=0 (success)
2020-09-03 16:30:11 (+0530) 22794100 INFO: download-mcs-compat-19.1.100-38-x86_64.rpm compare_rpms value: 0
2020-09-03 16:30:11 (+0530) 22794100 INFO: Rpm download-mcs-compat-19.1.100-38 is already installed on node 0.s
2020-09-03 16:30:11 (+0530) 22794100 INFO: post: Rpm download-mcs-compat-19.1.100-38 is already installed on node 0.s
2020-09-03 16:30:11 (+0530) 4107124 INFO: Processing rpm 1/1 succeeded
2020-09-03 16:30:11 (+0530) 4107124 INFO: post: Processing rpm 1/1 succeeded
2020-09-03 16:30:11 (+0530) 4107124 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=admin, ssh command is: (mcserver.sh --test --runasanyuser) >/usr/local/avamar/var/run_command-sysout.14142.4107124 2>&1
2020-09-03 16:30:15 (+0530) 4107124 INFO: - - - - - - - BEGIN(output) - - - - - - -
2020-09-03 16:30:15 (+0530) 4107124 INFO: INFO: Administrator Server status: up
2020-09-03 16:30:15 (+0530) 4107124 INFO: - - - - - - - END
2020-09-03 16:30:15 (+0530) 4107124 INFO: "mcserver.sh --test --runasanyuser", exit status=0 (success)
2020-09-03 16:30:15 (+0530) 4107124 INFO: Informing the MC to use the new catalog
2020-09-03 16:30:15 (+0530) 4107124 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=root, ssh command is: (mccli plugin update --force) >/usr/local/avamar/var/run_command-sysout.14142.4107124 2>&1
2020-09-03 16:30:18 (+0530) 4107124 INFO: - - - - - - - BEGIN(output) - - - - - - -
2020-09-03 16:30:18 (+0530) 4107124 INFO: 1,23996,CLI failed to connect to MCS.
2020-09-03 16:30:18 (+0530) 4107124 INFO: Attribute Value
2020-09-03 16:30:18 (+0530) 4107124 INFO: --------- ----------------------------------------------------
2020-09-03 16:30:18 (+0530) 4107124 INFO: option    AVAMAR001.corp.server.com: Name or service not known
2020-09-03 16:30:18 (+0530) 4107124 INFO:
2020-09-03 16:30:18 (+0530) 4107124 INFO: - - - - - - - END
2020-09-03 16:30:18 (+0530) 4107124 ERROR: "mccli plugin update --force", exit status=1 (error)
2020-09-03 16:30:18 (+0530) 4107124 INFO: post: "mccli plugin update --force", exit status=1 (error)
2020-09-03 16:30:18 (+0530) 4107124 ERROR: MCS failed to update the new catalog
2020-09-03 16:30:18 (+0530) 4107124 INFO: post: MCS failed to update the new catalog
2020-09-03 16:30:18 (+0530) 4107124 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=admin, ssh command is: (/usr/local/avamar/bin/avhostname -f) >/usr/local/avamar/var/run_command-sysout.14142.4107124 2>/usr/local/avamar/var/run_command-syserr.14142.4107124
2020-09-03 16:30:18 (+0530) 4107124 INFO: - - - - - - - BEGIN(output) - - - - - - -
2020-09-03 16:30:18 (+0530) 4107124 INFO: AVAMAR001.corp.company.com
2020-09-03 16:30:18 (+0530) 4107124 INFO: - - - - - - - END
2020-09-03 16:30:18 (+0530) 4107124 INFO: "/usr/local/avamar/bin/avhostname -f", exit status=0 (success)
2020-09-03 16:30:18 (+0530) 4107124 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=, ssh command is: (rpm -q dpnserver --last | head -1) >/usr/local/avamar/var/run_command-sysout.14142.4107124 2>&1
2020-09-03 16:30:18 (+0530) 4107124 INFO: Report home message: 2020-09-03 16:30:18 +0530|AVAMAR001.corp.server.com|19.1.0-38|AvamarUpgrade|19.1.0.38|/data01/avamar/repo/temp/AvamarUpgrade-19.1.0-38.avp_1599118779715/tmp|UpdateClientPluginCatalog|Updating Client Plug-in Catalog (172 of 226)|ERROR: UpdateClientPluginCatalog::do   - operation failed
2020-09-03 16:30:18 (+0530) 4107124 ERROR:  - operation failed
2020-09-03 16:30:19 (+0530) 4107124 INFO: post:  - operation failed
2020-09-03 16:30:19 (+0530) 4107124 INFO: ---------- END UpdateClientPluginCatalog::do ------------

Both the hostname and mccli commands report the correct value:

hostname -f 

AVAMAR001.corp.company.com

mccli server show-prop

0,23000,CLI command completed successfully.
Attribute                                    Value
-------------------------------------------- ----------------------------------
State                                        Full Access
Active sessions                              0
Total capacity                               3.9 TB
Capacity used                                101.2 GB
Server utilization                           2.6%
Bytes protected (client pre-comp size)       62.9 GB
Bytes protected quota (client pre-comp size) Not configured
License expiration                           Never
Time since Server initialization             365 days 02h:44m
Last checkpoint                              2020-09-03 13:36:24 IST
Last validated checkpoint                    2020-09-03 07:00:36 IST
System Name                                  servername
System ID                                    1567585332@00:00:00:00:00:00
HFSAddr                                      AVAMAR001.corp.company.com
HFSPort                                      27000
IP address                                   10.x.x.87
Number of nodes                              1
Nodes Online                                 1
Nodes Offline                                0
Nodes Read-only                              0
Nodes Timed-out                              0

• /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml
• /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

1. As root, check which file has the wrong mcsaddr value:

grep -i mcsaddr /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

<!-- mcsaddr    : network name or IP of Administrator Server node -->
mcsaddr="AVAMAR001.corp.company.com"

grep -i mcsaddr /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

<!-- mcsaddr    : network name or IP of Administrator Server node -->
mcsaddr="AVAMAR001.corp.server.com"

In this example, the incorrect version is /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml with a value of AVAMAR001.corp.server.com

2. Create a backup of the incorrect file:

cp -p /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml  /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml.`date +%y%m%d`

-- Or --

cp -p /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml  /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml.`date +%y%m%d`

3. Edit the incorrect file:

vi /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

-- Or --

vi /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

Update the incorrect mscaddr value.

4. Resume the workflow.

If issues remain, create a swarm with the Avamar SER team.