Avamar: How to update the backup metadata reference a backup was manually recalled from the Data Domain Cloud Tier

The correct recall procedure for a backup from the Data Domain (DD) Cloud Tier to the Active Tier uses the Avamar "avtier" command, as documented in Avamar: How to recall backups that were migrated to Data Domain Cloud Tier.

This article addresses scenarios where the correct procedure was not used, and the recall was manually performed on the DD without changes to the Avamar.

After the manual DD recall, the backup is on the DD active tier, but the Avamar metadata for the backup shows it on the cloud tier.

For example:

avmgr getb --path=/clients/ClientName --format=xml --incpartials | grep -i 1D359BBB62CE6BA

backuplistrec flags="24117249" labelnum="592" label="***_Exchange_Full-1510201386783#0" created="1510274087" roothash="fff989cfe0fe0654abc5453466fcbe7b12879207" 
totalbytes="3537383718912.00" ispresentbytes="0.00" pidnum="3018" percentnew="0" expires="0" created_prectime="0x1d359bbb62ce6ba" partial="0" retentiontype=
"daily,weekly,monthly" backuptype="Full" ddrindex="1" locked="0" direct_restore="1" tier="2" appconsistent="not_available" sealstate="COMPLETE"/>

(Output wrapped for readability)

From this output, tier="2" indicates that Avamar still recognizes the backup as stored on Cloud Tier.

Attempts to restore this backup from Avamar causes new (unnecessary) data movement on the DD between the active and cloud tiers when the recall is invoked.

If a backup recalled from the Data Domain Cloud Tier to the Active Tier was not performed by Avamar, the metadata for that backup on Avamar is out-of-sync with the location of the data on Data Domain.

There are two workarounds to fix this:

Method 1: Synchronize the backup metadata between GSAN and Data Domain by running Garbage Collection (GC) with "checkalltierduringgc" enabled:

1. Stop the maintenance scheduler: 

dpnctl stop maint 

2. Enable the changealltiering parameter:

avmaint --ava config checkalltierduringgc=true

3. Start garbage collection:

 avmaint --ava garbagecollect 

4. Monitor the grid until the GC has completed using one of the commands below:

avmaint gcstatus -- or -- status.dpn

5. Once GC completes, disable the changealltiering parameter:

avmaint --ava config checkalltierduringgc=false

6. Restart the maintenance scheduler:

dpnctl start maint

Method 2: Manually update backup metadata using "avmgr chgt":

1. Create an Avamar checkpoint in case any issues occur.

2. Query the status of the backup to be recalled from the Data Domain Cloud Tier by using the command below. Record the parameter "created_prectime" value.

avmgr getb --path=/clients/ClientName --format=xml --incpartials |grep 'labelnum="<labelnum>"' 

For example:  

avmgr getb --path=/clients/ClientName --format=xml --incpartials |grep 'labelnum="592"'

backuplistrec flags="24117249" labelnum="592" label="***_Exchange_Full-1510201386783#0" created="1510274087" roothash="fff989cfe0fe0654abc5453466fcbe7b12879207" totalbytes="3537383718912.00" ispresentbytes="0.00" pidnum="3018" percentnew="0" expires="0" created_prectime="0x1d359bbb62ce6ba" partial="0" retentiontype="daily,weekly,monthly" backuptype="Full" ddrindex="1" locked="0" direct_restore="1" tier="3" ...

3. Using the "created_prectime" from above, run the following command:  

avmgr chgt --path=/clients/clientName  --date="<created_prectime>" --tiering=0 --ava

For example: 

avmgr chgt --path=/clients/clientName  --date="0x1d359bbb62ce6ba" --tiering=0 --ava 

The "--tiering=0" changes the backup metadata location from the Cloud Tier to the Active Tier.

4. Rerun the command from step 2 to verify that the backup now reports the correct tier:

avmgr getb --path=/clients/ClientName --format=xml --incpartials |grep 'labelnum="<labelnum>"' 

For example:  

avmgr getb --path=/clients/ClientName --format=xml --incpartials |grep 'labelnum="592"'

avmgr getb --path=/clients/ClientName --format=xml --incpartials  labelnum=592
<backuplistrec flags="24117249" labelnum="592" label="***_Exchange_Full-1510201386783#0" created="1510274087" roothash="fff989cfe0fe0654abc5453466fcbe7b12879207" totalbytes="3537383718912.00" ispresentbytes="0.00" pidnum="3018" percentnew="0" expires="0" created_prectime="0x1d359bbb62ce6ba" partial="0" retentiontype="daily,weekly,monthly"   backuptype="Full" ddrindex="1" locked="0" direct_restore="1" tier="0" ...