Avamar: How to Safely Remove a Data Domain System from an Avamar grid.

This article provides the steps to remove a Data Domain system from an Avamar grid.

avmaint config --ava |grep "immutable\|governancemode"

immutablebackups="false"
periodimmutablebackups="false"
governancemode="false"

• All the backups stored on the Data Domain by this Avamar should be deleted prior to the removal process. 
• After all the backups are deleted, it is recommended to run a full maintenance cycle (manual or scheduled).
  • A full maintenance cycle includes garbage collection (GC), checkpoint (CP), checkpoint validation (hfscheck), checkpoint, and the removal of old checkpoints (RMCP).

Ensure all backup data on Data Domain, and metadata on Avamar are deleted before proceeding with the final removal of the Data Domain system. 

This article aims to address two scenarios: 

Scenario 1: Unclean removal: The Data Domain system is no longer active or is not reachable from Avamar.

• Data Domain system has already been turned off.
• Data Domain was disconnected from the network before being removed from Avamar. 
• The file system on Data Domain was destroyed or reinitialized before removing the Data Domain from the Avamar (in other words, the Avamar mtree on the Data Domain no longer exists)  

Scenario 2: Clean removal: The Data Domain system is active and is reachable from the Avamar. All the backup data corresponding to the metadata stored on the Avamar are still accessible.

The easiest way to test this is to run the " ddrmaint cplist" command from the Avamar Utility Node. If this returns, Avamar can connect to the Data Domain, read the Avamar mtree (storage unit) on the Data Domain and list the checkpoints. 

Example:

ddrmaint cplist

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<checkpointlist
  cmd="cplist"
  lsu="avamar-1643691625"
  count="3">
  <ddrservers count="1">
    <ddrserver
      name="dd01.company.com"
      index="1"
      ddrcreatetime="1685672902"/>
  </ddrservers>
  <checkpoint
    tag="cp.20240724220026"
    mtime="1721858446"
    isvalid="true"
    validcheck="true"/>
  <checkpoint
    tag="cp.20240801021435"
    mtime="1722478550"
    isvalid="true"
    validcheck="false"/>
  <checkpoint
    tag="cp.20240801045429"
    mtime="1722488144"
    isvalid="true"
    validcheck="false"/>
</checkpointlist>

Scenario 1 and Scenario 2 Common Steps:

These common steps should be followed for both a clean and unclean removal: 

1. Log in to the Avamar Utility Node as admin.

2. Locate the Data Domain hostname and index for the Data Domain that is being removed.
(The DDR index is a numerical value that represents the number of Data Domain systems attached to Avamar.)

The index can be found by running the following command as admin on the Avamar Utility Node:

ddrmaint read-ddr-info

Sample output:

<ddrconfig client-map-default="true" cloud_enabled="false" cloud_unit_name="" dd-cert-chain="" ddos-version="7.13.0.20-1082704" ddrcreatetime="1685672902" ddrid="A42AF86128712C2361D8E072BA5E20E745C5F8A5" gsan-backup-target-default="true" hostname="dd01.company.com" index="1" instant-access-limit="32" ipv4-hostname="dd01.company.com" ipv6-hostname="" max-streams="50" max-streams-for-cp-backup="2" modelno="DD VE" mtree_name="/data/col1/avamar-1234567890" password="Onlh2XP9xEpvI2exqGQwqA==" policy_id="%2Fdata%2Fcol1%2Favamar-1643691625%3AECS_Unit1" serialno="xxxxxxxxxxxxxxx" token="AQAjwjOLe0cAcXbsTvn4AYhPc274/LNOErIAjGbfELzhoQ==" username="ddboost">
...
<ddrconfig client-map-default="false" cloud_enabled="false" cloud_unit_name="" dd-cert-chain="" ddos-version="6.2.1.80-692201" ddrcreatetime="1722482083" ddrid="3EBBA48A04A3DE3D2CAC662CF32E5DDBD0E68C58" gsan-backup-target-default="false" hostname="dd02.company.com" index="2" instant-access-limit="32" ipv4-hostname="dd02.company.com" ipv6-hostname="" max-streams="50" max-streams-for-cp-backup="0" modelno="DD VE Version 4.0" mtree_name="/data/col1/avamar-1234567890" password="Si+8QUMErXcQuq99uQ+IpQ==" policy_id="" serialno="xxxxxxxxxxxxxxx" token="AQA51kB/zWJVUc2Hso5NqQVla+XC71ucYATwtbP4Rq1f8A==" username="ddboost">

In this sample output, there are two Data Domains attached.

• Data-Domain-hostname: dd01.company.com
• ddr-index-value: 1.

3. Check to see which datasets, if any, that have the Data Domain selected:

The easiest way to do this is by running commands from the CLI to determine which datasets must be edited. The UI of choice (AUI or MCS) can then be used to update the datasets.

For an Avamar grid with only one Data domain attached, use the following command:

psql -p 5555 mcdb -c "select dataset_name,plugin_name,domain from v_ds_commands where command_name='ddr-index' and value NOT IN ('0') order by dataset_name asc;"

Example:

psql -p 5555 mcdb -c "select dataset_name,plugin_name,domain from v_ds_commands where command_name='ddr-index' and value NOT IN ('0') order by dataset_name asc;"

  dataset_name   |       plugin_name       | domain
-----------------+-------------------------+--------
 Default Dataset | Linux Fast Incrementals | /
 Default Dataset | VMware vCloud vApp      | /
 Default Dataset | Windows File System     | /
 dd01_dst        | Windows File System     | /clients
(4 rows)

-- Or --

For an Avamar grid with multiple Data Domain systems attached, run the following command:

psql -p 5555 mcdb -c "select dataset_name,plugin_name,domain from v_ds_commands where command_name='ddr-index' and value='<ddr-index-value>' order by dataset_name asc;"

Where <ddr-index-value> is that index noted in in step 2 above (in our sample output, this was 1)

Example:

psql -p 5555 mcdb -c "select dataset_name,plugin_name,domain from v_ds_commands where command_name='ddr-index' and value='1' order by dataset_name asc;"

  dataset_name   |       plugin_name       | domain
-----------------+-------------------------+--------
 Default Dataset | Linux Fast Incrementals | /
 Default Dataset | VMware vCloud vApp      | /
 Default Dataset | Windows File System     | /
 dd01_dst        | Windows File System     | /clients
(4 rows)

If there are no datasets reported, go to step 6, otherwise continue from step 4.

4. Log in to the Avamar UI (AUI or MCS) and remove the Data Domain from each dataset listed:

Avamar Administrator - AUI:

a. From the Administration heading, select Setting.

b. Click Dataset.

c. For each dataset: 

i. Select the required dataset and click edit.

ii. Select each of the plugins listed in turn from the output in step 3.

In the sample output for the "Default Dataset" this would be "Linux Fast Incrementals," "VMware vCloud vApp," and "Windows File System."

iii. Either clear "Store backup on Data Domain system" or, where there are multiple Data Domains attached, replace the Data Domain name if applicable.

iv. Click submit.

v. Repeat for all listed plugins

-- Or --

Avamar Administrator - MCS:

a. Go to the Tools Menu.

b. Select Manage Datasets.

c. For each dataset: 

i. Select the required dataset and click edit.

ii. Go to the options tab.

iii. Select each plug-in listed in step 3’s output—such as "Linux Fast Incrementals," "VMware vCloud vApp," and "Windows File System" from the Default Dataset.

iv. Clear the "Store backups on Data Domain" option or, where there are multiple Data Domain systems attached, replace the Data Domain name if applicable.

v. Click OK.

vi. Repeat for each required dataset

5. When complete, rerun the same command that was run in earlier in step 2b. There should be no datasets listed:

Examples:

psql -p 5555 mcdb -c "select dataset_name,plugin_name,domain from v_ds_commands where command_name='ddr-index' and value='1' order by dataset_name asc;"

-- Or --

psql -p 5555 mcdb -c "select dataset_name,plugin_name,domain from v_ds_commands where command_name='ddr-index' and value='<ddr-index-value>' order by dataset_name asc;"

The expected output is the same from either command:

 dataset_name | plugin_name | domain
--------------+-------------+--------
(0 rows)

6. If multiple Data Domains are attached and Avamar is a replication source, ensure the DD being removed is not set as the default replication storage system:

Log in to the Avamar UI (AUI or MCS) 

Avamar Administrator - AUI:

a. From the Administration heading, select System.

b. Click Data Domain.

c. Select the Data Domain to be removed. 

d. If the "Default Replication Storage System" value is "yes," go to step e, otherwise go to step 7.

e. Click cancel.

f. Of the remaining Data Domains (not being removed), select the alternate that becomes the default replication storage system.

g. Click edit.

h. Enter and reenter the ddboost account password.

i. Select "Use system as default replication storage"

j. Select validate, next, next, finish. This may take some time to update.

-- Or --

Avamar Administrator - MCS:

a. Select Server from the main dashboard.

b. Select the Server Management tab.

c. If only the "Bytes Protected Summary" window is seen, click the divider and expand the window until the Servers (Avamar and Data Domain) are seen:

d. Highlight the Data Domain to be removed. 

e. If, within the "Node Information" window, the "Default Replication Storage System" is Yes," go to step f, otherwise go to step 7.

f. Of the remaining Data Domains (not being removed), select the alternate that becomes the default replication storage system.

g. From the Actions Menu, select "Edit Data Domain system"

h. Select the "Use system as default replication storage" checkbox and click OK. 

i. Click Yes to confirm that changing the default replication storage mapping. This may take some time to update.

• For an unclean removal (Scenario 1), go to step 8
• For a clean removal (Scenario 2), go to step 11

Scenario 1: Steps specific to an Unclean Removal

• The Data Domain was turned off before being removed from Avamar. 
• The Data Domain was disconnected from the network before being removed from Avamar. 
• The file system on the Data Domain was reinitialized before being removed from Avamar (in other words, the Avamar mtree no longer exists on the Data Domain system).
• The file system on the Data Domain is destroyed or corrupted

8. Run the following command:  

mccli dd delete --name=<Data-Domain-hostname> --force

Where <Data-Domain-hostname> is the name obtained from step 2 (dd01.company.com in the sample output)

Example:

mccli dd delete --name=dd01.company.com --force

0,30936,Deleted Data Domain system.
Attribute                 Value 
------------------------- -------------------------------------------------------------------
ipv6Hostname
ipv4Hostname              dd01.company.com

0,30936,Deleted Data Domain system.
Attribute                 Value
------------------------- -------------------------------------------------------------------
replication-storage-check Data Domain system is the only default replication storage system.
ipv6Hostname
force-delete              Force delete option was in use for the Data Domain system deletion.
ipv4Hostname              dd01.company.com

If any other errors are encountered, stop and create a service request. This article, along with any errors should be in included when creating the Service Request.  

9. Delete any references to backups stored on the Data Domain:

Any references to backups stored on the inaccessible Data Domain must be deleted. Although the Data Domain and backup data are inaccessible, the metadata is still present on Avamar.

The script remove_ddr_backups can be used to detect and display backups referencing Data Domain and is located in the /usr/local/avamar/bin directory on the Avamar Utility Node.

a. Run the script in nondestructive mode:

remove_ddr_backups <ddr-index-value>

Where <ddr-index-value> is the index of the Data Domain noted in step 2 (In the sample output, this was 1)

Example:

remove_ddr_backups 1

Running in non-destructive mode. No data will be destroyed.
To run in destructive mode add the command line parameter '--delete-backups'.
Total Data Domain backups 5422.
Total Avamar backups 10296.
Total backups 15718.
Matched 2232 out of 5422 total Data Domain backups with ddrindex 1.
Total Data Domain backups to be deleted 8014.
Backup counts for each ddr index before any backup deletions:
    ddrindex 001: 2232
    ddrindex 002: 3190
remove_ddr_backups completed successfully in non-destructive mode.

As the output states, the above command runs in "nondestructive" mode and does not delete any backups. 

This output shows that there are 2232 backups stored on ddrindex 1.

b. Remove the backups referenced on the inaccessible Data Domain by adding the "--delete-backups" flag:

remove_ddr_backups <ddr-index-value> --delete-backups

Where <ddr-index-value> is the index of the Data Domain noted in step 2 (In our sample output, this was 1)

Example:

remove_ddr_backups 1 --delete-backups

remove_ddr_backups will irrevocably destroy backup data on server 'dd01.company.com'.
Type 'yes' to permanently remove backup data on this server: 

c. Verify again that this is the correct Data Domain as this process is not reversible.

d. Type yes

Expected output:

1  Request succeeded

0  ERROR!  Exit code 116: Cannot delete client's last backup remove_ddr_backups: 
Error: remove_ddr_backups: 
Error: Failed to execute 'avmgr delb  --path=/MC_RETIRED/test_client_A2014.01.23.16.18.41 --date=0x1CEF631A5EDEA52', result code '29696', error string 'Bad file descriptor'
Error: Failed to execute 'avmgr delb  --path=/clients/test_client_B --date=0x1CEF631A5EDEB922', result code '29696', error string 'Bad file descriptor'

See the Additional Information section for instructions on how to delete the locked backups manually.

After removing any locked backups, repeat steps 9b-9d to remove the Data Domain.

10. Continue from Step 14.

Scenario 2: Steps specific to a Clean Removal

11. Any backups stored on the Data Domain to be removed must be deleted:

The script remove_ddr_backups can be used to detect and display backups referencing Data Domain and is located in the /usr/local/avamar/bin directory on the Avamar Utility Node.

a. Run the script in nondestructive mode:

remove_ddr_backups <ddr-index-value>

Where <ddr-index-value> is the index of the Data Domain noted in step 2 (In our sample output, this was 1)

Example:

remove_ddr_backups 1

Running in non-destructive mode. No data will be destroyed.
To run in destructive mode add the command line parameter '--delete-backups'.
Total Data Domain backups 5422.
Total Avamar backups 10296.
Total backups 15718.
Matched 2232 out of 5422 total Data Domain backups with ddrindex 1.
Total Data Domain backups to be deleted 8014.
Backup counts for each ddr index before any backup deletions:
    ddrindex 001: 2232
    ddrindex 002: 3190
remove_ddr_backups completed successfully in non-destructive mode.

As the output states, the above command runs in "nondestructive" mode and does not delete any backups. 

This output shows that there are 2232 backups stored on ddrindex 1.

b. Remove the backups referenced on the inaccessible Data Domain by adding the "--delete-backups" flag:

remove_ddr_backups <ddr-index-value> --delete-backups

Where <ddr-index-value> is the index of the Data Domain noted in step 2 (In our sample output, this was 1)

Example:

remove_ddr_backups 1 --delete-backups

remove_ddr_backups will irrevocably destroy backup data on server 'dd01.company.com'.
Type 'yes' to permanently remove backup data on this server: 

c. Verify again that this is the correct Data Domain as this process is not reversible.

d. Type yes

Expected output:

1  Request succeeded

0  ERROR!  Exit code 116: Cannot delete client's last backup remove_ddr_backups: 
Error: remove_ddr_backups: 
Error: Failed to execute 'avmgr delb  --path=/MC_RETIRED/test_client_A2014.01.23.16.18.41 --date=0x1CEF631A5EDEA52', result code '29696', error string 'Bad file descriptor'
Error: Failed to execute 'avmgr delb  --path=/clients/test_client_B --date=0x1CEF631A5EDEB922', result code '29696', error string 'Bad file descriptor'

See the Additional Information section for instructions on how to delete the locked backups manually.

After removing the locked backups, repeat steps 11b-11d to remove the Data Domain.

12. Once the client backups are removed, it is necessary to remove all checkpoints which were associated to any Data Domain backups from the Avamar grid:

a. Check when the last checkpoint completed: 

status.dpn |grep "Last checkpoint"

b. Perform an MCS flush:

mcserver.sh --flush

Example:

mcserver.sh --flush

=== BEGIN === check.mcs (preflush)
check.mcs                        passed
=== PASS === check.mcs PASSED OVERALL (preflush)
Flushing Administrator Server...
Administrator Server flushed.

c. Take a new checkpoint:

avmaint checkpoint --ava --wait

The prompt does not return until the checkpoint has been completed.

d. Check for the last checkpoint:

status.dpn |grep "Last checkpoint"

Verify that this differs from the output from 12a.

e. Validate the checkpoint: 

avmaint hfscheck --full --keep --ava

f. Monitor hfscheck to completion.

g. Once hfscheck has been completed successfully, take another checkpoint:

avmaint checkpoint --ava --wait

Again, the prompt does not return until the checkpoint has been completed.

h. Remove any old checkpoints:

avmaint rmcp --full --ava

13. Remove the Data Domain from Avamar using the CLI or UI:

CLI

mccli dd delete --name=<Data-Domain-hostname>

Where <Data-Domain-hostname> is the name obtained from step 2 (dd01.company.com in our sample output)

Example:

mccli dd delete --name=dd01.company.com

0,30936,Deleted Data Domain system.
Attribute                 Value
------------------------- -------------------------------------------------------------------
ipv6Hostname
ipv4Hostname              dd01.company.com

The command may take some time to return to a prompt. Let the command complete without interruption.

-- Or --

Avamar Administrator - AUI:

a. From the Administration heading, select System.

b. Click Data Domain.

c. Select the Data Domain to be removed. 

d. Click Delete.

e. Review the confirmation to ensure that it has the name of the Data Domain to be removed.

f. Click Yes to confirm the deletion of the Data Domain. This may take some time to update.

g. The Data Domain should no longer be present.

-- Or --

Avamar Administrator - MCS:

a. Select Server from the main dashboard.

b. Select the Server Management tab.

c. If only the "Bytes Protected Summary" window is seen, click the divider and expand the window until the Servers (Avamar and Data Domain) are seen:

d. Select the Data Domain to be removed. 

e. From the Actions Menu, select "Delete Data Domain system"

f. Review the confirmation to ensure that it has the name of the Data Domain to be removed.

g. Click Yes to confirm the deletion of the Data Domain. This may take some time to update.

h. The Data Domain should no longer be present.

14. Run a manual maintenance cycle (even if this was performed in Step 12):

a. Check when the last checkpoint completed: 

status.dpn |grep "Last checkpoint"

b. Perform an MCS flush:

mcserver.sh --flush

Example:

mcserver.sh --flush

=== BEGIN === check.mcs (preflush)
check.mcs                        passed
=== PASS === check.mcs PASSED OVERALL (preflush)
Flushing Administrator Server...
Administrator Server flushed.

c. Take a new checkpoint:

avmaint checkpoint --ava --wait

The prompt does not return until the checkpoint has been completed.

d. Check for the last checkpoint:

status.dpn |grep "Last checkpoint"

Verify that this differs from the output from 14a.

e. Validate the checkpoint: 

avmaint hfscheck --full --keep --ava

f. Monitor hfscheck to completion.

g. Once hfscheck has been completed successfully, take another checkpoint:

avmaint checkpoint --ava --wait

Again, the prompt does not return until the checkpoint has been completed.

h. Remove any old checkpoints:

avmaint rmcp --full --ava

15. Return the grid to a production state using Avamar: How to Set the Avamar Server into a Known Controlled State as a guide.

How to manually delete locked backups:

Manual backup deletion of locked backups can be completed using the UI (MCS or AUI) or CLI.

1. Take the output from the remove_ddr_backups and note all the paths whose backups could not be removed. 

0  ERROR!  Exit code 116: Cannot delete client's last backup remove_ddr_backups: 
Error: remove_ddr_backups: 
Error: Failed to execute 'avmgr delb  --path=/MC_RETIRED/test_client_A2014.01.23.16.18.41 --date=0x1CEF631A5EDEA52', result code '29696', error string 'Bad file descriptor'
Error: Failed to execute 'avmgr delb  --path=/clients/test_client_B --date=0x1CEF631A5EDEB922', result code '29696', error string 'Bad file descriptor'

In this sample output, these are /MC_RETIRED/test_client_A2014.01.23.16.18.41 and /clients/test_client_B. 

2. Remove the locked backups for each client using the preferred method:

CLI:

a. Retrieve the backup information using getb command: 

avmgr getb --format=xml --incpartials --path=<full-path-to-client> |grep -i <date>

Where <full-path-to-client> is the client name from our sample output, and <date> is the hexadecimal value in the date field.

Example and sample output:

avmgr getb --format=xml --incpartials --path=/MC_RETIRED/test_client_A2014.01.23.16.18.41 | grep -i 0x1CEF631A5EDEA52

<backuplistrec flags="20185089" labelnum="165" label="Production 2300-Virtual Servers - Production-138" created="1386739633" roothash="d244f1dd698949e71022b9c975e311913236dc87" totalbytes="25585092608.00" ispresentbytes="0.00" pidnum="3016" percentnew="0" expires="1389157200" created_prectime="0x1cef631a5edea52" partial="0" retentiontype="daily,weekly" backuptype="Full" ddrindex="1" locked="1"/>

b. Delete the backup using the delb command: 

avmgr delb --path=<full-path-to-client> --date=<date> --risklosingallbackups

Example and sample output: 

avmgr delb  --path=/MC_RETIRED/test_client_backup_A2014.01.23.16.18.41 --date=0x1CEF631A5EDEA52 --risklosingallbackups

1  Request succeeded

c. Rerun the getb command from to verify that the backup has been deleted:

avmgr getb --format=xml --incpartials --path=<full-path-to-client> |grep -i <date>

Example: 

avmgr getb --format=xml --incpartials --path=/MC_RETIRED/test_client_A2014.01.23.16.18.41 | grep -i 0x1CEF631A5EDEA52

d. Repeat for each of the clients listed in the "remove_ddr_backups" output.

From the Avamar Administrator - AUI:

a. Select the Administration heading, select Asset Management.

b. Navigate the tree and locate the first client in the list. 

c. Click "View More" to see the backups associated with the client.

d. Select the backup to delete. 

e. Click Delete.

g. Confirm when prompted.

h. Repeat for each of the clients listed in the "remove_ddr_backups" output.

From the Avamar Administrator - MCS:

a. Select Backup and Restore.

b. Navigate the tree and locate the first client in the list. 

c. Ensure that "manage" is selected, and select everything under "By Retention":

d. Click Retrieve.

e. Once the backup has been returned, right-click the backup.

f. Select "Delete Backup"

g. Confirm when prompted.

h. Repeat for each of the clients listed in your "remove_ddr_backups" output.