Integrated Data Protection Appliance and Avamar: All VM image level backup do not complete after vCenter root certificate change

# mccli server show-services | egrep "Name|----|vCenter connection"
Name                                  Status
------------------------------------- --------------------------------------
/vc01.example.local                   vCenter connection(s) down.

# mccli server start-service --service=/vc01.example.local
1,23996,CLI failed to connect to MCS.
Attribute Value
--------- -------------------------------------------------------------------------------------------------------------------------------------------------------------
option    Connection refused to host: vc01.example.local; nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)

1. Confirm the vCenter connection does not ignore certificate parameters. By default, is set to false:

# grep ignore_vc_cert /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
              <entry key="ignore_vc_cert" value="false" />

2. If step 1 shows that the ignore_vc_cert is false, it indicates the vCenter connection requires certificate. Run the following script to import new vCenter root certificate into MC RMI keystore.

#! /bin/bash

if [ $# != 1 ]
then
    echo "usage: $0 vCenter_fqdn"
    exit 1
fi

vc=$1

keystore=/usr/local/avamar/lib/rmi_ssl_keystore
alias=vCenter_$(echo $vc | sed 's/\./_/g')

rm -f /tmp/download.zip && rm -f -R /tmp/certs

wget --no-check-certificate --no-proxy https://$vc/certs/download.zip -P /tmp

if [[ ! -f /tmp/download.zip ]]; then exit; fi

unzip -o /tmp/download.zip -d /tmp

# Look for the latest vCenter root certificate
certfile=$(find /tmp/certs/lin/ ! -name *.r* -type f -printf '%T@ %p\n' | sort -r | head -1 | cut -d' ' -f2)
if [[ -z "$certfile" ]]; then
        echo "Error: no vc root certificate found!"
else
        if [[ ! -z $(keytool -list -keystore $keystore -storepass `ask_pass -r keystore_passphrase` | grep -i $alias) ]]; then
                keytool -delete -alias $alias -file $certfile -keystore $keystore  -storepass `ask_pass -r keystore_passphrase`
        fi
        keytool -import -alias $alias -trustcacerts -file $certfile -keystore $keystore  -storepass `ask_pass -r keystore_passphrase` -noprompt
        echo && echo "*** Please restart MCS service ***" && echo
fi

rm -f /tmp/download.zip && rm -f -R /tmp/certs

3. Run the script. Here is an example:

# sh import_vc_cert.sh vc01.example.local
--2023-04-04 15:29:15--  https://vc01.example.local/certs/download.zip
Resolving vc01.example.local (vc01.example.local)... 10.60.9.50
Connecting to vc01.example.local (vc01.example.local)|10.60.9.50|:443... connected.
WARNING: cannot verify vc01.example.local's certificate, issued by '/CN=vc01.example.local/DC=vSphere/DC=local/C=US/ST=California/O=vc01.example.local/OU=VMware Engineering':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 17593 (17K) [zip]
Saving to: '/tmp/download.zip'

100%[===================================================================================================================================================================================================>] 17,593      --.-K/s   in 0s

2023-04-04 15:29:15 (243 MB/s) - '/tmp/download.zip' saved [17593/17593]

Archive:  /tmp/download.zip
  inflating: /tmp/certs/lin/d1b17fac.r0
  inflating: /tmp/certs/mac/d1b17fac.r0
  inflating: /tmp/certs/win/d1b17fac.r0.crl
  inflating: /tmp/certs/lin/aebde1f5.r1
  inflating: /tmp/certs/mac/aebde1f5.r1
  inflating: /tmp/certs/win/aebde1f5.r1.crl
  inflating: /tmp/certs/lin/8f3736ab.r0
  inflating: /tmp/certs/mac/8f3736ab.r0
  inflating: /tmp/certs/win/8f3736ab.r0.crl
  inflating: /tmp/certs/lin/aebde1f5.0
  inflating: /tmp/certs/mac/aebde1f5.0
  inflating: /tmp/certs/win/aebde1f5.0.crt
  inflating: /tmp/certs/lin/d1b17fac.0
  inflating: /tmp/certs/mac/d1b17fac.0
  inflating: /tmp/certs/win/d1b17fac.0.crt
  inflating: /tmp/certs/lin/8f3736ab.0
  inflating: /tmp/certs/mac/8f3736ab.0
  inflating: /tmp/certs/win/8f3736ab.0.crt
Certificate was added to keystore

*** Please restart MCS service ***

4. Restart MCS service:

# dpnctl stop mcs && dpnctl start mcs && dpnctl start sched

5. Check and confirm vCenter connection is up, and VM backups are working:

# mccli server show-services | egrep "Name|----|vCenter connection"