Avamar: Data Domain Integration: Hfscheck Errors Related to Cloud Tier File Migration Feature

Avamar 7.4 and later provides "Managed LTR on Cloud Tier"

Backups that existed before this conversion did not create the XML file (ddr_files.xml) with a future date and the V2 file (v2_ddr_files.xml) with the current date given the system was a single node entity at the time.
 
If only a few backups are impacted, this can be done manually. Otherwise, recovery musts be performed with scripts.

Cloud Tier File Migration feature is only supported from Avamar version 7.4.x.

When Cloud Tier is enabled on Avamar version 7.2.x and 7.3.x, Avamar hfscheck fails with MSG_ERR_DDR_ERROR. hfscheck fails because the XML files present in the backups no longer exist on the Data Domain Active Tier (as they were moved to the Cloud Tier without Avamar's knowledge).

The scenario occurs when the Cloud Tiering feature is enabled or configured at the Avamar MTree on the Data Domain directly (and NOT setup in Avamar)

The following output (located on the Data Domain) correctly shows the two .xml files:

-rw-rw-rw-  1 ddboost users 123456784580 Feb  1 23:14 container.1.cdsf 
-rw-rw-rw-  1 ddboost users        12768 Feb  1 23:14 container.1.cdsf.trace 
-rw-rw-rw-  1 ddboost users          419 Feb  1 23:14 v2_ddr_files.xml 
-rw-rw-rw-  1 ddboost users          419 Jan  1  2037 ddr_files.xml 

• The v2_ddr_files.xml is created when enabling Cloud Tiering on the Data Domain
  • Contains the current retention settings for the Backup ID (BID)
• The ddr_files.xml
  • The file has a future date of January 1, 2037, so it is never sent to the Cloud Unit.

If the Cloud Tiering feature is configured directly on the Data Domain without Avamar being aware of it, no v2_ddr_files.xml is created on the Data Domain MTree.

This can cause the files to be moved to the Cloud Tier when the Data Domain runs its data-movement process.

Therefore the Avamar integrity check (hfscheck) fails as it cannot file the corresponding backups on the Avamar MTree.

1. A Data Domain is integrated to an Avamar (Single node Avamar server with no active or cloud tier on the Data Domain)
2. All backups on the Data Domain created using Avamar have the current expected ddr_files.xml file. 
3. The Cloud Tier is enabled on the Data Domain
4. A 'data movement' policy is created on the Data Domain to migrate the backups to the Cloud Tier
   • Avamar is not aware of the backdoor migration (like a Data Domain Archiver or Extended Retention system)
5. The ddr_files.xml files are migrated to the Cloud Tier as part of the 'data-movement' 

The command "data-movement policy show" can be run on the Data Domain shell to check the current 'data-movement' policies:

data-movement policy show

Mtree                          Target(Tier/Unit Name)   Policy          Value
----------------------------   ----------------------   -------------   -------
/data/col1/avamar-14146xxxxx   Cloud/vxx_virtustream    age-threshold   30 days
/data/col1/cloudtest           Cloud/vxx_virtustream    age-threshold   14 days
----------------------------   ----------------------   -------------   -------

Mtree                          Target(Tier/Unit Name)   Policy        Value
----------------------------   ----------------------   -----------   -------
/data/col1/avamar-148545xxxx   Cloud/exx1               app-managed   enabled
----------------------------   ----------------------   -----------   -------
code or text goes here 

The command "data-movement schedule show" shows the schedule:

data-movement schedule show 

Data-movement is scheduled to run on day(s) "mon" at "17:00" hrs.

The command "data-movement throttle show" shows the throttle:

data-movement throttle show 

Data-movement throttle is set to 100 percent.

...
<ddrconfig client-map-default="true" cloud_enabled="true" cloud_unit_name="cloud...
...mtree_name="/data/col1/avamar-1xxxxxxxx"
...
<snip>