Avamar: Garbage Collection fails after a recall failure caused by a DDFS Panic

Summary: Avamar Garbage collection may fail with MSG_ERR_DDR_ERROR after a Cloud recall failure caused due to DDFS panic.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Avamar Garbage collection (GC) fails with MSG_ERR_DDR_ERROR.

The following errors are seen in the Data Domain (DD) maintenance logs (ddrmaint.log):

grep -i "Failed to read" /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log 
Aug 20 08:27:52  ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/f0d09b747587b71c34f90b73a84b3a7866cb289c/1D3F60FB8F8404C/ddr_files.xml'
Aug 20 08:27:52  ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/f7f529c2372cdfb63eb9ca80ca70bbfc111c6680/1D3F608866DFF24/ddr_files.xml'
Aug 20 08:27:53  ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/b2d365bfe514e3dd8fd039cdeed24d5ce81b74ba/1D3F60961EEEDB0/ddr_files.xml'
Aug 20 08:27:53  ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/db5d9e3227847903d6a59ff0f395f0766c266bf5/1D3F6065268E920/ddr_files.xml'
Aug 20 08:27:55  ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/c97f81c65d7aa401076b3731e7d5b67a530e3da8/1D3F60F7EA6EB0A/ddr_files.xml'
Aug 20 08:27:56  ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2/ddr_files.xml'
  
grep -i "err.*maintain_cloud_copy: Cloud copy" /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log 
Aug 20 14:00:11  ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/f0d09b747587b71c34f90b73a84b3a7866cb289c/1D3F60FB8F8404C' is corrupt
Aug 20 14:00:12  ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/f7f529c2372cdfb63eb9ca80ca70bbfc111c6680/1D3F608866DFF24' is corrupt
Aug 20 14:00:12  ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/b2d365bfe514e3dd8fd039cdeed24d5ce81b74ba/1D3F60961EEEDB0' is corrupt
Aug 20 14:00:12  ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/db5d9e3227847903d6a59ff0f395f0766c266bf5/1D3F6065268E920' is corrupt
Aug 20 14:00:14  ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/c97f81c65d7aa401076b3731e7d5b67a530e3da8/1D3F60F7EA6EB0A' is corrupt
Aug 20 14:00:15  ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2' is corrupt
Note: root privilege is required to review the ddrmaint.log.
 
 

The logs also contain information about a recall failure:

grep -i "recall-backup.*" ddrmaint.log*
 

Example and sample output:

grep -i "recall-backup.*9f30e9cd4833708ec979338a1d7ea8050d1eaf30" ddrmaint.log* 
ddrmaint.log.3:Aug 13 02:25:53  ddrmaint.bin[28806]: Info: recall-backup cmdline: '/usr/local/avamar/bin/ddrmaint.bin --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc/client --vardir=/usr/local/avamar/var/client recall-backup --flagfile=/usr/local/avamar/etc/client/usersettings.cfg --server=avamartest --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --id=root --password=**************** --dpnid=1522437690 --account=/clients/testabc --client=9f30e9cd4833708ec979338a1d7ea8050d1eaf30 --backups=1D3F6089358D0E2:1 --logfile=/usr/local/avamar/var/client/MOD-1533734850483-1051-Tiering_tmpdir/MOD-1533734850483-1051-Tiering-#154-ddrmaint.log'
ddrmaint.log.3:Aug 13 02:27:45  ddrmaint.bin[28806]: Error: recall-backup::recall_backup - failed to clone cur/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2 to cloud-copy/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2, error code: 5040
ddrmaint.log.3:Aug 13 02:27:45  ddrmaint.bin[28806]: Error: recall-backup::recall_backup - failed to clone backup from cur/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2 to cloud-copy/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2

Cause

Avamar garbage collection may fail with MSG_ERR_DDR_ERROR after a Cloud recall failure caused due to DDFS panic.
  • If the Data Domain file system panicked while files were being recalled from the cloud tier, the ongoing cloud recall process leaves some partially recalled backup files in the "/data01/col1/avamar-xxxxxx/cloud-copy/<CID>/<BID>" directory.
Avamar Garbage collection (GC) then fails with MSG_ERR_DDR_ERROR. Garbage collection fails to read ddr_files.xml for backups in the cloud-copy directory on Data Domain.
  • It is possible that the ddr.xml file is present in the cloud-copy directory under the <BID> directory however when the Garbage collection reads the ddr.xml file, it finds that all required backup files are not present which may cause the GC to fail with this error.
  • This maybe because the ddr.xml is missing under directory "/data01/col1/avamar-xxxxxx/cloud-copy/<CID>/<BID>" or any of the backup files are missing.

Resolution

Create a Service Request, referencing this article.

Affected Products

Avamar

Products

Avamar, Avamar Server, Avamar Virtual Edition, Avamar with CloudBoost
Article Properties
Article Number: 000033327
Article Type: Solution
Last Modified: 23 Sep 2025
Version:  6
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.