Avamar: Plugin-based replication fails when the certificate and the key are mismatched

One of the following symptoms may be observed:

1. During the configuration of policy-based replication, "Verify Authentication" fails when adding another Avamar grid as a replication destination in the UI.

2. Policy-based replication fails to run. The log may show the following error message:

avagent Info <5008>: Logging to /usr/local/avamar/var/client/MOD-1476725811111-1008-Replicate-avagent.log
avagent Info <19803>: Ignoring the --service flag.
avagent Info <5702>: Command Line: /usr/local/avamar/bin/avagent.bin --gencerts="true" --mcsaddr="targetgrid.avamar.com" --mcsport="28001" --conntimeout="120" --logfile="/usr/local/avamar/var/client/MOD-1476725811111-1008-Replicate-avagent.log" --debug="false" --sysdir="/usr/local/avamar/etc/client" 
avagent Info <5703>: Parsed Flags: /usr/local/avamar/bin/avagent.bin --gencerts=true --mcsaddr=targetgrid.avamar.com --mcsport=28001 --conntimeout=120 --logfile=/usr/local/avamar/var/client/MOD-1476725811111-1008-Replicate-avagent.log --debug=false --sysdir=/usr/local/avamar/etc/client
[avagent]  ERROR: <0001> sslcontext::loadCert  certificate/key not found or invalid cert=/usr/local/avamar/etc/client/10.1.2.3/cert.pem key=/usr/local/avamar/etc/client/10.1.2.3/key.pem
avagent Error <5664>: SSL certificate/key not found or invalid.
avagent Info <0000>: Checking for stale certificate lock

In Avamar 7.3 or later, policy-based replication generates certificates for SSL connection between the source and target server. On the source utility node, it creates a folder for each replication target:

/usr/local/avamar/etc/client/<replication target>

For example: /usr/local/avamar/etc/client/10.10.10.2

1. Confirm that the issue is a mismatched certificate and key by performing the following steps:

a. Log in to the Avamar Utility Node as admin.

b. Elevate to root privilege:

su -

c. Identify the folder which stores the certificate and key pair using the following command:

ls -l /usr/local/avamar/etc/ | grep ^d

drwx------ 2 root  root  4096 Aug 22 13:28 10.10.10.2
drwx------ 2 root  root  4096 Aug 22 13:28 192.168.76.50
drwxr-xr-x 2 admin admin 4096 Apr 21 10:37 admin
drwxr-xr-x 2 admin admin 4096 Jun 25 09:09 akm 
drwxr-xr-x 2 admin admin 4096 Apr 21 10:37 avi
drwxr-xr-x 3 admin admin 4096 Jun 25 09:22 client
drwxr-xr-x 2 admin admin 4096 Apr 21 10:37 dtlt
drwxr-xr-x 5 root  root  4096 Jun 13 02:37 gsan-maint-scripts
drwxr-xr-x 2 admin admin 4096 Apr 21 10:37 mcs
drwxr-xr-x 2 admin admin 4096 Apr 21 10:37 server

In the output above, there should be a folder with the IP address of the replication target grid.

If there are multiple replication targets, there are multiple folders. As seen here, the replication partners are 10.10.1.2 and 192.168.76.50.

For this article, 192.168.76.50 is used as the example.

d. Verify that there are cert.pem and key.pem files:

ls -l /usr/local/avamar/etc/192.168.76.50/*.pem

Example output:

-rw-r--r-- 1 root root 2248 Jun 13 08:55 /usr/local/avamar/etc/192.168.76.50/cert.pem 
-rw-r--r-- 1 root root 1793 Jun 13 14:21 /usr/local/avamar/etc/192.168.76.50/chain.pem 
-rw-r--r-- 1 root root 2484 Jun 13 08:55 /usr/local/avamar/etc/192.168.76.50/key.pem

e. Check the certificates and keys:

openssl x509 -noout -modulus -in /usr/local/avamar/etc/192.168.76.50/cert.pem | openssl md5 

(stdin)= 2ba9eb24b22a0158ea4366e1971ad915

openssl rsa -noout -modulus -in /usr/local/avamar/etc/192.168.76.50/key.pem

(stdin)= b342ae5f5e044bbd8aff7d2dfb698bd3

f. If the two values above values DO NOT match, the issue is confirmed.

2. Once a mismatch is confirmed, perform the following steps to remove the old, and create the new certificate and key:

Remove the old certificate and key:

a. Log in to the Avamar Utility Node as admin.

b. Elevate to root privilege:

su -

c. Create a directory to backup the original certificate:

mkdir /tmp/certificatebackup

d. Back up the directory containing the certificate and key:

cp -p /usr/local/avamar/etc/192.168.76.50/* /tmp/certificatebackup/

e. Remove the old certificates:

rm -f /usr/local/avamar/etc/192.168.76.50/*

f. Verify that the certificates have been removed:

ls -l /usr/local/avamar/etc/192.168.76.50

total 0

Generate a new certificate and key:

MCS UI:

a. In the Avamar Administrator MCS UI, select the "Data Movement Policy" launcher.

b. Select the "Destinations" tab.

c. Highlight the required destination.

d. Right-click and select "Edit Destination"

e. Without making any changes, click "Verify Authentication".

If the verification succeeds, go to step 3. If the verification fails, contact Dell Technologies Avamar Support for assistance.

AUI:

a. In the AUI, select "System".

b. Within "System", select the "Replication Destination" launcher.

c. Select the radio button associated with the required destination.

d. Click "Edit".

e. Enter the credentials.

f. Click "Validate":

If the verification succeeds, go to step 3. If the verification fails, contact Dell Technologies Avamar Support for assistance.

3. Retry replication to verify that the issue has been resolved. 

If the issue persists, create a swarm with the Avamar SCR team for assistance.