Avamar: Information About Goav dd check-ssl Feature

Download and Install Goav tool

See Dell article 000192151 Avamar: Goav tool to download and install the Avamar Goav tool.

Once Goav is placed on Avamar, go to the directory and make the tool executable.

chmod a+x goav

Command

Use the Data Domain check-ssl feature by running the following command:

./goav dd check-ssl

Check the help screen for usage:

./goav dd check-ssl -h
Sign new DD imported-host ddboost certificate, and import Avamar root ca as imported-ca ddboost

Usage:
  goav dd check-ssl [flags]

Flags:
      --fix    Push MCS to automatically sign new DD imported-host ddboost certificate, and import Avamar root ca as imported-ca ddboost
  -h, --help   help for check-ssl

Global Flags:
  -d, --debug      debug output
  -f, --force      Ignore Configuration
  -n, --noheader   Do no display header

Use Case

When Data Domain is integrated with Avamar and session security is enabled, there may be a certificate issue between them.

Use this tool to diagnose potential certificate issues between Avamar and Data Domain.

Examples

Run passive checks which guarantee no changes are made to the Avamar or Data Domain.

./goav dd check-ssl
===========================================================
GoAv    :        1.61
Avamar  :        19.4
Date    :        12 Jul 2023 02:04 JST
===========================================================
COMMAND :  ./goav dd check-ssl
NOTE: This is not an official tool
===========================================================
Session Security Enabled                                              PASSED
DDR Security Feature Manual Flag                                      PASSED
DDR Host Cert Auto Refresh Flag                                       PASSED
GSAN Cert Chain Expiration                                            PASSED
GSAN Server Cert Expiration                                           PASSED
Get Attached Data Domains                                             PASSED
Check DDR Key exists                                                  PASSED
Test Port 22: ddsm-1111-els.example.dell.com.tester                   PASSED
Test ddr_key ssh auth: ddsm-1111-els.example.dell.com.tester          PASSED
DD SCP enabled: ddsm-1111-els.example.dell.com.tester                 PASSED
DD NFS enabled: ddsm-1111-els.example.dell.com.tester                 PASSED
DD system passphrase is set: ddsm-1111-els.example.dell.com.tester    PASSED
DD imported-host ddboost: ddsm-1111-els.example.dell.com.tester       PASSED
DD host issuer is attached: ddsm-1111-els.example.dell.com.tester     PASSED
Av chain.pem imported to DD: ddsm-1111-els.example.dell.com.tester    PASSED
Backup Scheduler Status                                               FAILED
Backup Scheduler Status is Down, start scheduler [yes/no]? n

If a check fails, there is an error message associated with it.

./goav dd check-ssl
===========================================================
GoAv    :        1.61
Avamar  :        19.4
Date    :        12 Jul 2023 02:09 JST
===========================================================
COMMAND :  ./goav dd check-ssl
NOTE: This is not an official tool
===========================================================
Session Security Enabled                                              PASSED
DDR Security Feature Manual Flag                                      FAILED
Error: ddr_security_feature_manual flag in mcserver.xml is true, should be false

Use the fix flag to allow automatic remediation of issues encountered.

./goav dd check-ssl --fix
===========================================================
GoAv    :        1.61
Avamar  :        19.4
Date    :        12 Jul 2023 04:11 JST
===========================================================
COMMAND :  ./goav dd check-ssl --fix
NOTE: This is not an official tool
===========================================================
This feature may need to restart MCS/DDboost. Ok [yes/no]? y
Session Security Enabled                                              PASSED
DDR Security Feature Manual Flag                                      PASSED
DDR Host Cert Auto Refresh Flag                                       PASSED
GSAN Cert Chain Expiration                                            PASSED
GSAN Server Cert Expiration                                           PASSED
Get Attached Data Domains                                             PASSED
Check DDR Key exists                                                  PASSED
Test Port 22: ddsm-1111-els.example.dell.com.tester                   PASSED
Test ddr_key ssh auth: ddsm-1111-els.example.dell.com.tester          PASSED
DD SCP enabled: ddsm-1111-els.example.dell.com.tester                 PASSED
DD NFS enabled: ddsm-1111-els.example.dell.com.tester                 FAILED
enabling nfs...                                                       FIXED
DD system passphrase is set: ddsm-1111-els.example.dell.com.tester    PASSED
DD imported-host ddboost: ddsm-1111-els.example.dell.com.tester       PASSED
DD host issuer is attached: ddsm-1111-els.example.dell.com.tester     PASSED
Av chain.pem imported to DD: ddsm-1111-els.example.dell.com.tester    PASSED
Backup Scheduler Status                                               FAILED
Backup Scheduler Status is Down, start scheduler [yes/no]? n

Explanation of Checks

Session Security

• When Session Security is enabled, certificates are exchanged between Avamar and Data Domain.
• When Session Security is disabled, certificates are not exchanged between Avamar, and Data Domain so there is no reason to run this tool.
• When the fix flag is used, if session security is disabled, it is not automatically enabled.

DDR Security Feature Manual and Host Cert Auto Refresh Flag

• These flags should be false in Avamar's mcserver.xml in most cases.
• Setting the manual flag to false allows MCS to sign the Data Domain certificate signing request and generate a signed host certificate for the Data Domain.
• If the manual flag is true, MCS does not attempt to generate a signed host certificate for the Data Domain.
• Setting the host certificate auto refresh flag to false is the usual setting as the signed host certificate is regenerated whenever it is missing.
• The tool ensures that both flags are set to the default value of false.
• When the fix flag is used, these flags are automatically changed to false if they are set to true.

GSAN Certificate Chain and Server Certificate Expiration

• GSAN runs on port 29000 which hosts a secure TCP connection with a keypair and certificate chain.
• The tool checks that these certificates are not expired.
• When the fix flag is used, if the GSAN certificate chain is expired, it is regenerated using mcrootca and enable_secure_config.sh.
• If only the GSAN server certificate is expired, it can be regenerated using "enable_secure_config.sh" without any downtime. The fix flag does this automatically.

Get Attached Data Domains

• Retrieve added Data Domains from ddrmaint persistent store.
• The following checks are done for each attached Data Domain.

Check DDR Key Exists

• Ensure that the DDR private key used for password-less authentication from Avamar to Data Domain exists and is readable.
• When the fix flag is used, if the key does not exist, it is regenerated automatically using mcddrsetup_sshkey and the new key imported to each attached Data Domain.

Test Port 22

• Ensure that port 22 is open between Avamar and Data Domain, and the Data Domain is listening.
• If the port is closed and the fix flag is used, no changes are made.

Test DDR Key SSH Authentication

• Attempt to connect to attached Data Domain using the existing DDR private key, instead of a password.
• When the fix flag is used, if this connection fails, the tool automatically imports the existing DDR private key to the Data Domain.

Data Domain SCP Enabled

• Ensure that the Secure Copy Protocol (SCP) is enabled on Data Domain.
• SCP is used to transfer files such as the certificates between Avamar and Data Domain, if it is disabled Avamar is unable to send the certificates to the Data Domain.
• If SCP is disabled and the fix flag is used, the tool automatically enables SCP on Data Domain.

Data Domain NFS Enabled

• Ensure that version 3 or 4 of NFS is enabled on Data Domain.
• When the fix flag is used, if NFS is disabled the tool automatically enabled NFS which defaults to enabling NFS version 3.
• If NFS is disabled, Avamar and Data Domain cannot properly communicate.

Data Domain System Passphrase

• Ensure that the Data Domain system passphrase is set.
• The Data Domain system passphrase must be set before the system can support data encryption, request digital certificates and to protect against data shredding.
• When the fix flag is used, if the passphrase is not set, no changes are made and the user must go set the Data Domain system passphrase at a convenient time preferably using the Data Domain web interface at Administration -> Access -> Administrator Access.

Data Domain Imported Host DDBoost

• Ensure that the signed host certificate (imported-host ddboost) on Data Domain exists and is not expired.
• This certificate is signed by MCS (GSAN certificate chain).
• When the fix flag is used, if it is missing or expired, then the tool attempts to regenerate it with the following steps.
  • Delete the existing imported host ddboost certificate.
  • Load the Avamar GSAN certificate chain.
  • Delete the existing imported ca ddboost and login-auth certificate (Avamar GSAN certificate chain) on Data Domain.
  • Restart DDBoost.
  • Restart MCS.
  • Perform a Data Domain sync by sending a "mccli dd edit" command, this is what pushes MCS to generate and import the new certificates to Data Domain.

Data Domain Host Issuer is Attached

• Check that the Data Domain signed host certificate (imported-host ddboost) has a paired Avamar GSAN certificate chain (imported ca ddboost) that it can export successfully.
• When the fix flag is used, if this fails to validate then the tool attempts to regenerate it with the following steps.
  • Delete the existing imported host ddboost certificate.
  • Load the Avamar GSAN certificate chain.
  • Delete the existing imported ca ddboost and login-auth certificate (Avamar GSAN certificate chain) on Data Domain.
  • Restart DDBoost.
  • Restart MCS.
  • Perform a Data Domain sync by sending a "mccli dd edit" command, this is what pushes MCS to generate and import the new certificates to Data Domain.
• The significance of this check is that the Data Domain may be used by several Avamar servers each with their own GSAN certificate chain. The Data Domain signed host certificate is only signed by one Avamar and must be able to find the certificate that signed it.

Avamar Chain Imported to Data Domain

• Check that the Avamar GSAN certificate chain is imported to Data Domain.
• Perform a comparison of the SHA1 fingerprint of the chain present on the Avamar server and one or more imported ca ddboost certificates present on the Data Domain.
• This check is done because the Avamar chain may not be imported, or an old Avamar chain may exist in its place from the same server maybe in the instance of regenerating certificates on Avamar.
• When several Avamar servers use the same Data Domain, each Avamar GSAN certificate chain must be present on the Data Domain as an imported ca ddboost and login-auth. Only one of these is the issuer to the Data Domain signed host certificate.
• When the fix flag is used, if this check fails then the tool attempts to regenerate it with the following steps.
  • Delete the existing imported host ddboost certificate.
  • Load the Avamar GSAN certificate chain.
  • Delete the existing imported ca ddboost and login-auth certificate (Avamar GSAN certificate chain) on Data Domain.
  • Restart DDBoost.
  • Restart MCS.
  • Perform a Data Domain sync by sending a "mccli dd edit" command, this is what pushes MCS to generate and import the new certificates to Data Domain.

Backup Scheduler Status

• This is a helper check to ensure that the backup scheduler is in a known state to the user.
• This is the only check that prompts to start the service when the fix flag is used or not.

See this video:

(You can also watch this same video on YouTube.)