Dell PowerProtect Cyber Recovery 19.15 Product Guide

Performing manual steps for Avamar recovery

After initiating an Avamar recovery in the Cyber Recovery UI, perform the following steps on the Avamar server host in the Cyber Recovery vault.

Prerequisites

You have successfully created a Recovery Sandbox in Cyber Recovery.
You have downloaded an executable copy of the lockbox_restore.pl script to the /home/admin/ directory on the Avamar server in the Cyber Recovery vault by typing the following command:
```
curl -O ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/lockbox_restore.pl
```
NOTE:If the link in the command does not work, see Knowledge Base Article 181972 for up-to-date information. Access to this document depends on your login credentials. If you do not have access to the document, contact your Dell Technologies representative.
The admin user must own the script.

You have the required credentials:

Table 1. Required credentials for an Avamar recoveryThe two-column table lists the required component in the first column and the corresponding description in the second column.
Component	Description
Application	Login credentials for PuTTY and Avamar
Avamar	The admin and root user accounts, which might be stored on a specific system or in a document
DD Boost	Avamar DD Boost user id and password on the Cyber Recovery vault
Cyber Recovery username	`cradmin`

About this task

Use a PuTTY or SSH session that is connected to the Avamar server in the Cyber Recovery vault to perform the following procedure.

Steps

Use PuTTY to log in to the Avamar server as the admin user.
Stop the Avamar services by typing the following command:
```
dpnctl stop all
```
Answer Yes to the query about shutting down the local instance of EM Tomcat.
Confirm that the services are stopped properly by typing the following command:
```
dpnctl status
```
Switch to the Avamar root user by typing the following command:
```
su -
```
Verify that the IP address of the vault DD system resolves to the production DD name by typing the following command:
CAUTION:This step is critical for performing the recovery. This step ensures that the Avamar server can connect to the vault DD system and perceives it as the production DD system in the vault.
```
cat /etc/hosts
```
The following example shows sample ouput:
```
# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
  ::1 localhost.localdomain localhost
  #(ave-03 is the production hostname 
  # but the IP specified must point to the vault IP.) 
192.168.2.83 ave-03.vcorp.local ave-03 
192.168.2.106 ddve-prod-05.vcorp.local ddve-prod-05 ddve-cr-06.vcorp.local ddve-cr-06 
```
In the preceding example, ddve-prod-05 is the name of the production DD system and 192.168.2.106 is the IP address of the vault DD system (also known as ddve-cr-06). Both the DD FQDNs and short names are assigned to the 192.168.2.106 IP address.
NOTE:The following FQDN names are used in examples throughout the rest of this document:
- <Production_DD-FQDN>: ddve-prod-05.vcorp.local
- <Production_Stager_Avamar-FQDN>: ave-03.vcorp.local
Modify the Avamar /etc/hosts file on the Cyber Recovery vault for both Avamar and Data Domain, as needed.

As the root user, run a checkpoint restore operation using the hfsctime noted during the recovery sandbox process and using the following syntax:

cprestore --hfsctime=<hfsctime> --ddr-server=<Production_DD-FQDN> --ddr-user=<ddboost user name>

For example:

# cprestore --hfsctime=1560177494 --ddr-server=ddve-05.vcorp.local --ddr-user=ddboost

When prompted, enter the DD Boost user password.
A list of available checkpoints that can be used to restore is displayed. The checkpoint at the bottom of the list is the most recent checkpoint.
Enter the name of the checkpoint that you want to restore, for example cp.20211216090102, ensuring that the name is exact. Press Enter and when prompted, type yes to confirm your entry.

NOTE:This step might take some time as it is copying data from the vault DD system to the staging Avamar server to perform the recovery steps. Press Enter every few minutes in the PuTTY window to avoid timing out.
When the checkpoint restore operation completes, enter the DD Boost user password when you are prompted for a password.
Switch to the Avamar admin user by typing the following command:
```
su - admin
```
Start the Avamar rollback procedure using the same checkpoint name as the checkpoint name that you selected for the checkpoint restore operation by typing the following command:
```
rollback.dpn --cptag=cp.20211216090102 --noddrollback --nogetserverlogs 2>&1 | tee -a rollback.out
```
This step can take some time. When it is completed, it displays the output of the status.dpn command.
As the admin user, list the hostname of the Avamar server by typing the following command:
```
hostname -f
```
Begin the rollback of the MCS services by typing by typing the following command:
```
mcserver.sh --restore --norestart --v 2>&1 |tee -a mcs_restore.out
```
When prompted, enter Y to proceed with the restore, enter the <Production_Stager_Avamar-FQDN> obtained from the hostname -f command; press Enter for port 27000.
Switch to the Avamar root user by typing the following command:
```
su -
```

Run the lockbox_restore.pl script by typing the following command and then answering yes to all the prompts:

/home/admin/lockbox_restore.pl

NOTE:If an error is displayed for the lockbox, type yes to proceed and then provide the correct operating system password for the admin user.

The following example shows sample ouput:

Sample run updating “admin”: (Note – this run entered a BAD password the first time. Second time was successful):

Your keystore contains 4 entries
Keystore certs: [mcectls, Nov 10, 2021] [mcrsatls, Nov 10, 2021] [mcecroot, Nov 10, 2021] [mcrsaroot, Nov 10, 2021]
DEBUG: Checking lockbox 'admin' key...
Sorry, try again.
Sorry, try again.
sudo: 3 incorrect password attempts
ERROR: 'sudo -A' failed. Error=256
        This indicates a problem with the 'admin' password stored in the lockbox.
        This will cause downstream problems with MCS startup.

Lockbox verification FAILED for admin. Proceed ?
  Enter `yes`<enter> to proceed, `q` to quit :yes
[LOCKBOX] Enter New lockbox entry for 'admin':*********
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
>>Updated with new value under name "admin".
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup

Sorry, try again.
Sorry, try again.
sudo: 3 incorrect password attempts
ERROR: 'sudo -A' failed. Error=256
        This indicates a problem with the 'admin' password stored in the lockbox.
        This will cause downstream problems with MCS startup.

Lockbox verification FAILED for admin. Proceed ?
  Enter `yes`<enter> to proceed, `q` to quit :yes
[LOCKBOX] Enter New lockbox entry for 'admin':**************
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
>>Updated with new value under name "admin".
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup

DEBUG: Avagent   version:     19.4.100-116
DEBUG: Avagent   OS version:  SLES-64

Switch to the Avamar admin user by typing the following command:
```
su - admin
```

Start the MCS by tby typing the following command:

mcserver.sh --start --v 2>&1 |tee -a mcs_start.out

Verify the services are up and running by typing the following command:
```
dpnctl status
```
Start any subsystems that are stopped by typing the following command:
```
dpnctl start <subsystem>
```
NOTE:Leave the scheduler and maintenance processes as down.
Do the following;
1. Ensure that emt is started by typing the following command:
```
dpnctl start emt
```
2. Ensure that ddrmaint-service is started by typing the following command:
```
dpnctl start ddrmaint-service
```
Switch to the Avamar root user by typing the following command:
```
su -
```

Add the SSH key for the Data Domain FQDN, using the following syntax:

cat ~admin/.ssh/ddr_key.pub | ssh <ddboost_user>@<Production_DD-FQDN>adminaccess add ssh-key

For example:

# cat ~admin/.ssh/ddr_key.pub | ssh ddboost@ddve-05.vcorp.local adminaccess add ssh-key

When prompted, enter the password for the ddboost username.

As the root user, regenerate the certificates by typing the following command:
```
enable_secure_config.sh –-certs
```

Verify the security settings by typing the following command.

enable_secure_config.sh –-showconfig

The following example shows sample ouput:

# enable_secure_config.sh –-showconfig


Current Session Security Settings
----------------------------------
"encrypt_server_authenticate"="true" 
"secure_agent_feature_on" ="true" "session_ticket_feature_on"="true"
"secure_agents_mode"="secure_only" 
"secure_st_mode" ="secure_only"
"secure_dd_feature_on" ="true" 
"verifypeer" ="yes"

NOTE: If the value for the first two options is false, type enable_secure_config.sh --enable-secure-all and then type enable_secure_config.sh --showconfig to check the security settings again.

Switch to the Avamar admin user by typing the following command:
```
su - admin
```
Restart the MCS by typing the following command:
```
mcserver.sh --restart --v  2>&1 |tee -a mcs_start.out
```
Enter Y to proceed.
Edit the DD properties by typing the following command:
```
mccli dd edit --name=<Production_DD-FQDN>
```
Confirm the DD properties by typing the following command:
```
mccli dd show-prop --name=<Production_DD-FQDN>
```
This step takes several minutes as it edits the DD name in the MCS. When the step is completed, the DD <Production_DD-FQDN> is displayed in several lines.
Switch to the Avamar root user by typing the following command:
```
su -
```
Revoke the token access using the following syntax:
```
ssh cradmin@<Production_DD-FQDN> "ddboost user revoke token-access <ddboost username>"
```
For example:
```
# ssh sysadmin@ddve-prod-05.vcorp.local"ddboost user revoke token-accessddboostuser"
```
Enter the password for the sysadmin.
NOTE: This command can use the sysadmin or cradmin user to revoke the token access. The command output displays the following message:
```
Revoked token access for user <ddboost username>
```
As the root user, stop the Avamar Agent service by typing the following command:
```
/etc/init.d/avagent stop
```
Delete the Avamar Client ID (cid.bin) by typing the following two commands:
```
cd /usr/local/avamar/var/client

rm -f cid.bin
```
Switch to the Avamar admin user by typing the following command:
```
su - admin
```

Edit the client properties by typing the following two commands: :

hostname -f

mccli client edit --domain=/MC_SYSTEM --name=<Production_Stager_Avamar-FQDN> --activated=false

Switch to the Avamar root user by typing the following command:
```
su -
```
Start the Avamar Agent service by typing the following command:
```
/etc/init.d/avagent start
```
Switch to the Avamar admin user by typing the following command:
```
 su - admin
```

To take a checkpoint and validate it, type the following five commands:

dpnctl start ddrmaint-service

dpnctl stop maint

mcserver.sh --flush

avmaint checkpoint --ava <Wait a few minutes while the checkpoint is being created.>

cplist --lscp <A new checkpoint is displayed based on the current date.>

To view a status, type the following two commands:

avmaint hfscheck --ava --full 

-watch -d -n5 'avmaint hfscheckstatus'

Restart the maintenance service by typing the following command:
```
dpnctl maint start
```
Log in to the Avamar UI using the MCUser on the Avamar host server (https://<avamar-host>/aui ). From the left navigation pane, go to Administration > System and then select Data Domain on the right pane.
1. Verify that the DD system is displayed in the main window.
2. Verify that the data represented on the DD properties matches the data of the Avamar DD system. The icons that precede the entry must be green or at least amber.
3. From the Avamar navigation menu options, verify that all the policies, clients, and other configuration items match those items of the production system.
Return to PuTTY to ensure that the hfscheck procedure is completed and the status is complete. Press Ctrl-c to exit PuTTY.
See Avamar's standard operating procedures to reactivate clients in the Cyber Recovery vault and perform the required application recoveries.

Next steps

Delete the recovery sandbox. See Cleaning up after an Avamar recovery.

Welcome

Welcome to Dell