Avamar: Lockbox and Keystore restore

MCS fails to start with this message:

 MCS Preference has been decrypted failed: 

When running the mcrootca all command (possibly using the Session Security Configuration workflow), this failure message appears:

INFO: INFO: Executing mcrootca...
INFO: INFO: Initializing, may take a few moments...
INFO: INFO: Generating, saving and verifying MC EC root key and certificate...
INFO: ERROR: Failed to generate or save or verify MC EC root key and certificate.
INFO: ERROR: Error message: null
INFO: INFO: Generating, saving and verifying MC RSA root key and certificate...
INFO: ERROR: Failed to generate or save or verify MC RSA root key and certificate.
INFO: ERROR: Error message: null
INFO: INFO: mcrootca exited with return value = 1

The /usr/local/avamar/var/mc/server_log/mcserver.log.0 file shows exceptions or errors:

Listing the keystore, either avamar_keystore or rmi_ssl_keystore, the following errors appear:

root@hostname:/#: keytool -list -v -keystore /usr/local/avamar/lib/rmi_ssl_keystore
Enter keystore password:
keytool error: java.io.IOException: Could not decrypt data.
java.io.IOException: Could not decrypt data.
        at com.rsa.cryptoj.o.lu.a(Unknown Source)
        at com.rsa.cryptoj.o.lu.a(Unknown Source)
        at com.rsa.cryptoj.o.lu.a(Unknown Source)
        at com.rsa.cryptoj.o.lu.a(Unknown Source)
        at com.rsa.cryptoj.o.lu.a(Unknown Source)
        at com.rsa.cryptoj.o.lu.a(Unknown Source)
        at com.rsa.cryptoj.o.lu.a(Unknown Source)
        at com.rsa.cryptoj.o.lu.engineLoad(Unknown Source)
        at java.security.KeyStore.load(Unknown Source)
        at sun.security.tools.keytool.Main.doCommands(Unknown Source)
        at sun.security.tools.keytool.Main.run(Unknown Source)
        at sun.security.tools.keytool.Main.main(Unknown Source)
Caused by: java.security.UnrecoverableKeyException: Invalid password.
        ... 12 more

Note: If the keystore password is unknown for the above command, run the following:

/usr/local/avamar/bin/avlockbox.sh -r keystore_passphrase

A similar output is seen:

admin@hostname:~/>: avtar --backups --path=/MC_BACKUPS --count=10
avtar Info <5551>: Command Line: /usr/local/avamar/bin/avtar.bin --flagfile=/usr/local/avamar/etc/usersettings.cfg --server=<*hostname*> --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --id=root --password=**************** --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc --backups --account=/MC_BACKUPS --count=10
avtar Info <7977>: Starting at 2021-01-13 11:29:04 EST [avtar May 20 2020 08:10:00 19.3.100-149 Linux-x86_64]
avtar Info <6555>: Initializing connection
2021/01/13-16:29:05.00542 [avtar]  ERROR: <0001> sslcontext::loadCert  certificate/key not found or invalid cert=/usr/local/avamar/etc/cert.pem key=/usr/local/avamar/etc/key.pem
avtar Error <5664>: SSL certificate/key not found or invalid.
avtar FATAL <8606>: GComMgr::init() Unable to initialize socket library.
avtar FATAL <8604>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
avtar FATAL <5308>: Failed to initiate session with server
avtar Info <6149>: Error summary: 5 errors: 8606, 5308, 8604, 1, 5664
avtar Info <5314>: Command failed (5 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))

1. Download the lockbox_restore.pl script from the Central Dell site. Transfer the file to the Avamar server using this article: How to upload or download files from an Avamar server.
2. Upload the lockbox_restore.pl to the Single-Node or the Avamar Utility Node in the /home/admin directory.
3. Log in to Avamar as admin and switch to root.
4. [Optional] Run:

ls -al /usr/local/avamar/var/mc/server_data/lockbox_backup/

Output:

root@example:/usr/local/avamar/var/mc/server_data/lockbox_backup/#: ls -l
total 0
drwxrwxr-x 2 admin admin 281 Dec 16 16:17 2020-12-16-16_17
root@example:/usr/local/avamar/var/mc/server_data/lockbox_backup/#:

5. Change the directory to /home/admin.
6. Set permission to run the script:

chmod 755 lockbox_restore.pl

7. Run the script (for Avamar 19.10, go to the end of this Resolution section):

perl lockbox_restore.pl 

8. If the following message appears:

11:24:54 lockbox_restore.pl Version 1.01-181972 on AVE or Physical box
Selected Restored subdir=/usr/local/avamar/var/mc/server_data/lockbox_backup/2020-12-16-16_17 .. starting restore.

WARNING: Selected 'Restore lockbox dir' appears to be 28.0 days old.
Enter `yes`<enter> to proceed, `q` to quit : 

9. Type in yes<enter> to run the lockbox restore. 

10. Start the MCS process on Avamar with the dpnctl start mcs command.
11. Then create a lockbox backup by running: avlockbox.sh -b 
12. [Optional] Then run a ls -al and there are two lockbox backups:

root@example:/usr/local/avamar/var/mc/server_data/lockbox_backup/#: ls -l
total 0
drwxrwxr-x 2 admin admin 281 Dec 16 16:17 2020-12-16-16_17
drwxrwxr-x 2 admin admin 217 Jan 13 11:41 2021-01-13-11_41
root@example:/usr/local/avamar/var/mc/server_data/lockbox_backup/#:

The primary use case for this article is for Avamar Cyber Recovery. It focuses on scenarios where an air-gapped (vault) Avamar must restore data from a DD checkpoint backup.

Here are the steps involved:

1. Checkpoint Restoration:

   • The air-gapped Avamar restores a checkpoint from a DD checkpoint backup.
   • The Global Storage Area Network (GSAN) is rolled back to this checkpoint.

2. MCS Restores and Start:

   • Data is successfully restored, but the lockboxes and keystores are not.
   • For the initial restore, the Avamar vault likely does not have a current lockbox or keystore.
   • Attempting to start the Management Console Server (MCS) under this condition fails.

3. Fixing Lockboxes and Keystores:

   • To address this issue, run the avlockbox.pl script after restoring MCS data.
   • This script ensures that the lockboxes and keystores are placed in their correct locations.
   • Once this step is completed, the MCS can start successfully.

4. Root-to-Root Replication Scenario:

   • A similar use case arises during root-to-root replication MCS restore on the target system.
   • In this case, the required lockboxes and keystores may not match the current state of the system.
   • To resolve this, use the lockbox_restore.pl script.

5. Subsequent Restores:

   • After running lockbox_restore.pl once, subsequent restores will work without requiring it again.

6. Certificates and Passwords:

   • Keep in mind that if certificates or passwords have changed since the last run, use lockbox_restore.pl again.

Remember to adapt these steps to your specific Avamar environment. Properly managing lockboxes and keystores is crucial for successful Cyber Recovery operations.