Cloud Disaster Recovery - How to update the Cloud DR Add-On when Avamar has a Signed Certificate added
Summary: To show the steps to enable SSL validation between the Avamar and Cloud Disaster Recovery Add-ON (CDRA) when Avamar has added a Signed Certificate Authority.
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Instructions
If the following Knowledge Base article has been applied to the Avamar and the CDRA has already been deployed in the same premise environment, the communication between both instances stop.
KB:000158014 - Avamar 19.2: How to install CA Signed certificate using AUI
Impact: Cloud DR cannot receive virtual machine backups from Avamar. A backup completes on a Data Domain Cloud Disaster Recovery enabled policy, which starts reading the virtual machine backup copy from the Data Domain to be uploaded to the cloud.
Example of the error seen in the CDRA logs when CA is applied at the Avamar level:
Steps to Add the Certificate from the Avamar to the CDRA:
1. Connect to CDRA using SSH, and run the following openssl command to retrieve the Avamar certificate. Ensure you replace the Avamar IP address with your own. The command saves each certificate in the Avamar chain into its own file labeled certificate1.pem, certificate2.pem, certificate3.pem and so forth. The certificate-labeled number 1 will be the server certificate, and each certificate after that is any intermediate + root certificate authority:
2. Review the generated certificate files: home/cdr/
[cdr@cdra ~]$ ls -ltr
total 52
-rw-rw-r--. 1 cdr cdr 2300 Oct 7 12:07 certificate3.pem
-rw-rw-r--. 1 cdr cdr 2300 Oct 7 12:07 certificate2.pem
-rw-rw-r--. 1 cdr cdr 2300 Oct 7 12:07 certificate1.pem
drwxr-xr-x. 13 cdr cdr 32768 Oct 7 12:07 cdra
3. When working with keystores, you must import the root certificate authority to complete SSL validation at connection time. In this instance, certificate3.pem is the root certificate authority, since it is the highest numbered certificate.
4.Insert the PEM file to the CDRA and import the certificate authority cert with the following command:
Note: Validate below keytool path is present before running -> ls -l /usr/java/
sudo /usr/java/jre1.8.0_202-amd64/bin/keytool -import -trustcacerts -keystore /usr/java/jre1.8.0_202-amd64/lib/security/cacerts -storepass <storepass> -alias <alias> -import -file <file.pem>
Note: Default password for cacerts keystore is 'changeit'
Alias can be whatever name, in example used avamarcert-alias
[cdr@cdra ~]$ sudo /usr/java/jre1.8.0_202-amd64/bin/keytool -import -trustcacerts -keystore /usr/java/jre1.8.0_202-amd64/lib/security/cacerts -storepass changeit -alias avamarcert-alias -import -file certificate3.pem
Owner: O=getaCert - www.getacert.com, L=Seattle, ST=Washington, C=US
Issuer: O=getaCert - www.getacert.com, L=Seattle, ST=Washington, C=US
Serial number: b8d1b40f2af4ad49
Valid from: Sun Nov 06 11:46:48 EST 2016 until: Fri Jan 15 11:46:48 EST 2038
Certificate fingerprints:
MD5: CE:8E:85:A9:D0:22:2C:25:0B:0E:09:D0:36:AE:51:B9
SHA1: 16:F7:42:75:33:8B:12:25:F0:04:F3:CB:3D:18:07:04:2B:FC:4D:A5
SHA256: BA:A5:72:72:61:7A:34:67:45:13:EE:14:E4:10:6D:14:72:70:DC:98:4F:90:B4:A6:C6:2E:2A:0C:25:DB:D3:5D
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Trust this certificate? [no]: yes
Certificate was added to keystore
5.Validate the PEM is imported successfully with the following command:
/usr/java/jre1.8.0_202-amd64/bin/keytool -list -v -keystore /usr/java/jre1.8.0_202-amd64/lib/security/cacerts -storepass <storepass> -alias <alias>
[cdr@cdra ~]$ /usr/java/jre1.8.0_202-amd64/bin/keytool -list -v -keystore /usr/java/jre1.8.0_202-amd64/lib/security/cacerts -storepass changeit -alias avamarcert-alias
Alias name: avamarcert-alias
Creation date: Oct 7, 2021
Entry type: trustedCertEntry
Owner: O=getaCert - www.getacert.com, L=Seattle, ST=Washington, C=US
Issuer: O=getaCert - www.getacert.com, L=Seattle, ST=Washington, C=US
Serial number: b8d1b40f2af4ad49
Valid from: Sun Nov 06 11:46:48 EST 2016 until: Fri Jan 15 11:46:48 EST 2038
Certificate fingerprints:
MD5: CE:8E:85:A9:D0:22:2C:25:0B:0E:09:D0:36:AE:51:B9
SHA1: 16:F7:42:75:33:8B:12:25:F0:04:F3:CB:3D:18:07:04:2B:FC:4D:A5
SHA256: BA:A5:72:72:61:7A:34:67:45:13:EE:14:E4:10:6D:14:72:70:DC:98:4F:90:B4:A6:C6:2E:2A:0C:25:DB:D3:5D
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
6.Open CDRA swagger/authorize (https://CDRA-IP/api-doc) and enable the following tweak under: System V2 -> PUT /v2/system/tweaks
{
"key": "CDRA_TO_AVAMAR_VALIDATE_PEERS",
"value": "true"
}
7. Login over ssh to CDRA to restart CDRA service:
sudo service cdra restart
Then validate in CDRA user interface -> Configuration -> Local Backup and CDRS user interface -> System -> Health -> on-premise lists Avamar green again - takes a few minutes.
8. Allow a backup or upload cycle to complete to validate uploads on the CDRS Protection page.
NOTE: The above steps can only be applied to versions 19.7 and lower. The CDRA Operating system changes in versions 19.8 and higher:
The keytool is located in /usr/lib64/jvm/ and not /usr/java/. Steps 4 and 5 are replaced with the following:
4. Insert the PEM file to the CDRA and import the cacerts.
KB:000158014 - Avamar 19.2: How to install CA Signed certificate using AUI
Impact: Cloud DR cannot receive virtual machine backups from Avamar. A backup completes on a Data Domain Cloud Disaster Recovery enabled policy, which starts reading the virtual machine backup copy from the Data Domain to be uploaded to the cloud.
Example of the error seen in the CDRA logs when CA is applied at the Avamar level:
ERROR [2021-09-29 18:07:57,605] [https-jsse-nio-443-exec-4] [AvamarSoapClientAPI:connect:99]: RemoteException caught in AvamarSoapClientAPI due to invalid host or port number Connection has been shutdown: javax.net.ssl.SSLHandshakeExcept
ion: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Could not build a validated path.
ERROR [2021-09-29 18:07:57,606] [https-jsse-nio-443-exec-4] [BackupServersManager:validateAvamarForUpdate:331]: Host unreachable.
com.emc.cloud_dr.cdr.commons.common_models.exceptions.NoConnectivityException: No response due to invalid host name: avamar.com, or port number 9443, org.apache.axis2.AxisFault: Connection has been shutdown: javax.net.ssl.SSLHand
shakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Could not build a validated path.
Steps to Add the Certificate from the Avamar to the CDRA:
1. Connect to CDRA using SSH, and run the following openssl command to retrieve the Avamar certificate. Ensure you replace the Avamar IP address with your own. The command saves each certificate in the Avamar chain into its own file labeled certificate1.pem, certificate2.pem, certificate3.pem and so forth. The certificate-labeled number 1 will be the server certificate, and each certificate after that is any intermediate + root certificate authority:
openssl s_client -connect <avamar_ip_address>:9443 -showcerts </dev/null 2>/dev/null | awk '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/' | awk 'BEGIN { cert=0; pem=""; } /-----END CERTIFICATE-----/ {cert++; pem = pem $0 "\n"; print pem > "certificate" cert ".pem";pem=""; next;} { pem = pem $0 "\n"; }'
2. Review the generated certificate files: home/cdr/
[cdr@cdra ~]$ ls -ltr
total 52
-rw-rw-r--. 1 cdr cdr 2300 Oct 7 12:07 certificate3.pem
-rw-rw-r--. 1 cdr cdr 2300 Oct 7 12:07 certificate2.pem
-rw-rw-r--. 1 cdr cdr 2300 Oct 7 12:07 certificate1.pem
drwxr-xr-x. 13 cdr cdr 32768 Oct 7 12:07 cdra
3. When working with keystores, you must import the root certificate authority to complete SSL validation at connection time. In this instance, certificate3.pem is the root certificate authority, since it is the highest numbered certificate.
4.Insert the PEM file to the CDRA and import the certificate authority cert with the following command:
Note: Validate below keytool path is present before running -> ls -l /usr/java/
sudo /usr/java/jre1.8.0_202-amd64/bin/keytool -import -trustcacerts -keystore /usr/java/jre1.8.0_202-amd64/lib/security/cacerts -storepass <storepass> -alias <alias> -import -file <file.pem>
Note: Default password for cacerts keystore is 'changeit'
Alias can be whatever name, in example used avamarcert-alias
[cdr@cdra ~]$ sudo /usr/java/jre1.8.0_202-amd64/bin/keytool -import -trustcacerts -keystore /usr/java/jre1.8.0_202-amd64/lib/security/cacerts -storepass changeit -alias avamarcert-alias -import -file certificate3.pem
Owner: O=getaCert - www.getacert.com, L=Seattle, ST=Washington, C=US
Issuer: O=getaCert - www.getacert.com, L=Seattle, ST=Washington, C=US
Serial number: b8d1b40f2af4ad49
Valid from: Sun Nov 06 11:46:48 EST 2016 until: Fri Jan 15 11:46:48 EST 2038
Certificate fingerprints:
MD5: CE:8E:85:A9:D0:22:2C:25:0B:0E:09:D0:36:AE:51:B9
SHA1: 16:F7:42:75:33:8B:12:25:F0:04:F3:CB:3D:18:07:04:2B:FC:4D:A5
SHA256: BA:A5:72:72:61:7A:34:67:45:13:EE:14:E4:10:6D:14:72:70:DC:98:4F:90:B4:A6:C6:2E:2A:0C:25:DB:D3:5D
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Trust this certificate? [no]: yes
Certificate was added to keystore
5.Validate the PEM is imported successfully with the following command:
/usr/java/jre1.8.0_202-amd64/bin/keytool -list -v -keystore /usr/java/jre1.8.0_202-amd64/lib/security/cacerts -storepass <storepass> -alias <alias>
[cdr@cdra ~]$ /usr/java/jre1.8.0_202-amd64/bin/keytool -list -v -keystore /usr/java/jre1.8.0_202-amd64/lib/security/cacerts -storepass changeit -alias avamarcert-alias
Alias name: avamarcert-alias
Creation date: Oct 7, 2021
Entry type: trustedCertEntry
Owner: O=getaCert - www.getacert.com, L=Seattle, ST=Washington, C=US
Issuer: O=getaCert - www.getacert.com, L=Seattle, ST=Washington, C=US
Serial number: b8d1b40f2af4ad49
Valid from: Sun Nov 06 11:46:48 EST 2016 until: Fri Jan 15 11:46:48 EST 2038
Certificate fingerprints:
MD5: CE:8E:85:A9:D0:22:2C:25:0B:0E:09:D0:36:AE:51:B9
SHA1: 16:F7:42:75:33:8B:12:25:F0:04:F3:CB:3D:18:07:04:2B:FC:4D:A5
SHA256: BA:A5:72:72:61:7A:34:67:45:13:EE:14:E4:10:6D:14:72:70:DC:98:4F:90:B4:A6:C6:2E:2A:0C:25:DB:D3:5D
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
6.Open CDRA swagger/authorize (https://CDRA-IP/api-doc) and enable the following tweak under: System V2 -> PUT /v2/system/tweaks
{
"key": "CDRA_TO_AVAMAR_VALIDATE_PEERS",
"value": "true"
}
7. Login over ssh to CDRA to restart CDRA service:
sudo service cdra restart
Then validate in CDRA user interface -> Configuration -> Local Backup and CDRS user interface -> System -> Health -> on-premise lists Avamar green again - takes a few minutes.
8. Allow a backup or upload cycle to complete to validate uploads on the CDRS Protection page.
NOTE: The above steps can only be applied to versions 19.7 and lower. The CDRA Operating system changes in versions 19.8 and higher:
The keytool is located in /usr/lib64/jvm/ and not /usr/java/. Steps 4 and 5 are replaced with the following:
4. Insert the PEM file to the CDRA and import the cacerts.
sudo /usr/lib64/jvm/jre-1.8.0-openjdk/bin/keytool -import -trustcacerts -keystore /usr/lib64/jvm/jre-1.8.0-openjdk/lib/security/cacerts -storepass <storepass> -alias <alias> -import -file <file.pem> i.e. sudo /usr/lib64/jvm/jre-1.8.0-openjdk/bin/keytool -import -trustcacerts -keystore /usr/lib64/jvm/jre-1.8.0-openjdk/lib/security/cacerts -storepass changeit -alias *avamarcert-alias* -import -file Avamar.pem
5. Validate the PEM is imported successfully.
/usr/lib64/jvm/jre-1.8.0-openjdk/bin/keytool -list -v -keystore /usr/lib64/jvm/jre-1.8.0-openjdk/lib/security/cacerts -storepass <storepass> -alias <alias> i.e. /usr/lib64/jvm/jre-1.8.0-openjdk/bin/keytool -list -v -keystore /usr/lib64/jvm/jre-1.8.0-openjdk/lib/security/cacerts -storepass *changeit *-alias *avamarcert-alias*
All other steps remain the same.
Affected Products
Cloud Disaster RecoveryProducts
Avamar, Integrated Data Protection Appliance Family, Integrated Data Protection Appliance SoftwareArticle Properties
Article Number: 000192334
Article Type: How To
Last Modified: 29 May 2023
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.