Dell PowerProtect Cyber Recovery 19.14 Product Guide

Performing manual steps for Avamar recovery

After initiating an Avamar recovery in the Cyber Recovery UI, perform the following steps on the Avamar server host in the Cyber Recovery vault.

Prerequisites

You have successfully created a Recovery Sandbox in Cyber Recovery.
You have downloaded an executable copy of the lockbox_restore.pl script to the /home/admin/ directory on the Avamar server in the Cyber Recovery vault by using the following command:
```
# curl -O ftp://avamar_ftp:anonymous@ftp.avamar.com/software/scripts/lockbox_restore.pl
```
NOTE:If the link in the command does not work, see Knowledge Base Article 181972 for up-to-date information. Access to this document depends on your login credentials. If you do not have access to the document, contact your Dell Technologies representative.
The admin user must own the script.

You have the required credentials:


Component	Description
Application	Login credentials for PuTTY and Avamar
Avamar	The admin and root user accounts, which might be stored on a specific system or in a document
DD Boost	Avamar DD Boost user id and password on the Cyber Recovery vault
Cyber Recovery username	`cradmin`

About this task

Use a PuTTY or SSH session that is connected to the Avamar server in the Cyber Recovery vault to perform the following procedure.

Steps

Use PuTTY to log in to the Avamar server as the admin user.
Stop the Avamar services:
```
# dpnctl stop all
```
Answer Yes to the query about shutting down the local instance of EM Tomcat.
Confirm that the services are stopped properly:
```
# dpnctl status
```
Switch to the Avamar root user:
```
# su -
```
Verify that the IP address of the vault DD system resolves to the production DD name. This step ensures that the Avamar server can connect to the vault DD system and perceives it as the production DD system in the vault. Modify the Avamar /etc/hosts file on the Cyber Recovery vault for both Avamar and Data Domain, as needed.
CAUTION:This step is critical for performing the recovery.

In the following example, ddve-prod-05 is the name of the production DD system and 192.168.2.106 is the IP address of the vault DD system (also known as ddve-cr-06). Both the DD FQDNs and short names are assigned to the 192.168.2.106 IP address.
```
# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
  ::1 localhost.localdomain localhost
  #(ave-03 is the production hostname 
  # but the IP specified must point to the vault IP.) 
192.168.2.83 ave-03.vcorp.local ave-03 
192.168.2.106 ddve-prod-05.vcorp.local ddve-prod-05 ddve-cr-06.vcorp.local ddve-cr-06 
```
NOTE:The following FQDN names are used in examples throughout the rest of this document:
- <Production_DD-FQDN>: ddve-prod-05.vcorp.local
- <Production_Stager_Avamar-FQDN>: ave-03.vcorp.local
As the root user, run a checkpoint restore operation using the hfsctime noted during the recovery sandbox process and using the following syntax:
cprestore --hfsctime=<hfsctime> --ddr-server=<Production_DD-FQDN> --ddr-user=<ddboost user name>
For example:
```
# cprestore --hfsctime=1560177494 --ddr-server=ddve-05.vcorp.local --ddr-user=ddboost
```
When prompted, enter the DD Boost user password.
A list of available checkpoints that can be used to restore is displayed. The checkpoint at the bottom of the list is the most recent checkpoint.
Enter the name of the checkpoint that you want to restore, for example cp.20211216090102, ensuring that the name is exact. Press Enter and when prompted, type yes to confirm your entry.

NOTE:This step might take some time as it is copying data from the vault DD system to the staging Avamar server to perform the recovery steps. Press Enter every few minutes in the PuTTY window to avoid timing out.
When the checkpoint restore operation completes, enter the DD Boost user password when you are prompted for a password.
Switch to the Avamar admin user:
```
# su - admin 
		  
```
Start the Avamar rollback procedure using the same checkpoint name as the checkpoint name that you selected for the checkpoint restore operation:
```
# rollback.dpn --cptag=cp.20211216090102   --noddrollback --nogetserverlogs 2>&1 | tee -a rollback.out
```
This step can take some time. When it is completed, it displays the output of the status.dpn command.
As the admin user, list the hostname of the Avamar server:
```
# hostname -f
```
Begin the rollback of the MCS services:
```
# mcserver.sh --restore --norestart --v 2>&1 |tee -a mcs_restore.out
```
When prompted, enter Y to proceed with the restore, enter the <Production_Stager_Avamar-FQDN> obtained from the hostname -f command; press Enter for port 27000.
Switch to the Avamar root user:
```
# su -
```

Run the lockbox_restore.pl script and answer yes to all the prompts:

# /home/admin/lockbox_restore.pl

NOTE:If an error is displayed for the lockbox, type yes to proceed and then provide the correct operating system password for the admin user.

For example:

Sample run updating “admin”: (Note – this run entered a BAD password the first time. Second time was successful):

Your keystore contains 4 entries
Keystore certs: [mcectls, Nov 10, 2021] [mcrsatls, Nov 10, 2021] [mcecroot, Nov 10, 2021] [mcrsaroot, Nov 10, 2021]
DEBUG: Checking lockbox 'admin' key...
Sorry, try again.
Sorry, try again.
sudo: 3 incorrect password attempts
ERROR: 'sudo -A' failed. Error=256
        This indicates a problem with the 'admin' password stored in the lockbox.
        This will cause downstream problems with MCS startup.

Lockbox verification FAILED for admin. Proceed ?
  Enter `yes`<enter> to proceed, `q` to quit :yes
[LOCKBOX] Enter New lockbox entry for 'admin':*********
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
>>Updated with new value under name "admin".
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup

Sorry, try again.
Sorry, try again.
sudo: 3 incorrect password attempts
ERROR: 'sudo -A' failed. Error=256
        This indicates a problem with the 'admin' password stored in the lockbox.
        This will cause downstream problems with MCS startup.

Lockbox verification FAILED for admin. Proceed ?
  Enter `yes`<enter> to proceed, `q` to quit :yes
[LOCKBOX] Enter New lockbox entry for 'admin':**************
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
>>Updated with new value under name "admin".
>>Backup lockbox file
>>Backup keystore files
>>Backup SSV files
>>Flush backup
>>Local backup dir: /usr/local/avamar/src/lockbox_backup/2022-05-24-20_33
>>Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup

DEBUG: Avagent   version:     19.4.100-116
DEBUG: Avagent   OS version:  SLES-64

Switch to the Avamar admin user:
```
# su - admin
```

Start the MCS:

# mcserver.sh --start --v 2>&1 |tee -a mcs_start.out

Verify the services are up and running:
```
# dpnctl status
```
Start any subsystems that are stopped:
```
# dpnctl start <subsystem>
```
Ensure that the emt and ddrmaint-service are started:
```
# dpnctl start emt
# dpnctl start ddrmaint-service
```
NOTE:Leave the scheduler and maintenance processes as down.
Switch to the Avamar root user:
```
# su -
```

Add the SSH key for the Data Domain FQDN, using the following syntax:

# cat ~admin/.ssh/ddr_key.pub | ssh <ddboost_user>@<Production_DD-FQDN>adminaccess add ssh-key

For example:

# cat ~admin/.ssh/ddr_key.pub | ssh ddboost@ddve-05.vcorp.local adminaccess add ssh-key

When prompted, enter the password for the ddboost username.

As the root user, regenerate the certificates:
```
# enable_secure_config.sh –-certs
```

Verify the security settings:

# enable_secure_config.sh –-showconfig


Current Session Security Settings
----------------------------------
"encrypt_server_authenticate"="true" 
"secure_agent_feature_on" ="true" "session_ticket_feature_on"="true"
"secure_agents_mode"="secure_only" 
"secure_st_mode" ="secure_only"
"secure_dd_feature_on" ="true" 
"verifypeer" ="yes"

NOTE: If the value for the first two options is false, type enable_secure_config.sh --enable-secure-all and then type enable_secure_config.sh --showconfig to check the security settings again.

Switch to the Avamar admin user:
```
# su - admin
```

Restart the MCS:

# mcserver.sh --restart --v  2>&1 |tee -a mcs_start.out

Enter Y to proceed.

Edit the DD properties:

# mccli dd edit --name=<Production_DD-FQDN>

Confirm the DD properties:
```
# mccli dd show-prop --name=<Production_DD-FQDN>
```
This step takes several minutes as it edits the DD name in the MCS. When the step is completed, the DD <Production_DD-FQDN> is displayed in several lines.
Switch to the Avamar root user:
```
# su -
```
Revoke the token access using the following syntax:
```
# ssh cradmin@<Production_DD-FQDN> "ddboost user revoke token-access <ddboost username>"
```
For example:
```
# ssh sysadmin@ddve-prod-05.vcorp.local"ddboost user revoke token-accessddboostuser"
```
Enter the password for the sysadmin.
NOTE: This command can use the sysadmin or cradmin user to revoke the token access. The command output displays the following message:
```
Revoked token access for user <ddboost username>
```
As the root user, stop the Avamar Agent service:
```
# /etc/init.d/avagent stop
```

Delete the Avamar Client ID (cid.bin):

# cd /usr/local/avamar/var/client 
 # rm -f cid.bin

Switch to the Avamar admin user:
```
# su - admin
```

Edit the client properties:

# hostname -f
# mccli client edit --domain=/MC_SYSTEM --name=<Production_Stager_Avamar-FQDN> --activated=false

Switch to the Avamar root user:
```
# su -
```
Start the Avamar Agent service:
```
# /etc/init.d/avagent start
```
Switch to the Avamar admin user:
```
# su - admin
```

To take a checkpoint and validate it, type the following commands:

# dpnctl start ddrmaint-service
# dpnctl stop maint
# mcserver.sh --flush
# avmaint checkpoint --ava <Wait a few minutes while the checkpoint is being created.>
# cplist --lscp <A new checkpoint is displayed based on the current date.>

Type the following commands to view a status:

# avmaint hfscheck --ava --full 
# -watch -d -n5 'avmaint hfscheckstatus'

Restart the maintenance service:
```
# - dpnctl maint start
```
Log in to the Avamar UI using the MCUser on the Avamar host server (https://<avamar-host>/aui ). From the left navigation pane, go to Administration > System and then select Data Domain on the right pane.
1. Verify that the DD system is displayed in the main window.
2. Verify that the data represented on the DD properties matches the data of the Avamar DD system. The icons that precede the entry must be green or at least amber.
3. From the Avamar navigation menu options, verify that all the policies, clients, and other configuration items match those items of the production system.
Return to PuTTY to ensure that the hfscheck procedure is completed and the status is complete. Press Ctrl-c to exit PuTTY.
See Avamar's standard operating procedures to reactivate clients in the Cyber Recovery vault and perform the required application recoveries.

Next steps

Delete the recovery sandbox. See Cleaning up after an Avamar recovery.

Welcome

Welcome to Dell