Avamar: Garbage Collection MSG_ERR_DDR_ERROR after client retired on Avamar source IB/RLCE

This behavior is seen in the following configuration and scenario:

• Avamar source with immutable backups enabled (IB)
• Avamar target with immutable backups enabled (IB)
• Data Domain source with Retention-Lock Compliance Mode enabled (RLCE)
• Data Domain target with Retention-Lock Compliance Mode enabled (RLCE)

Replication policy configuration does not include the /MC_RETIRED domain.

admin@av-target:~/usr/local/avamar/var/ddrmaintlogs$ view ddrmaint.log
...
2024-09-29T06:14:35.625021-07:00 av-target ddrmaint.bin[4202]: Error: gc-finish::process_all_backups: Failed to delete empty client dir 'cur/51e83a682d09d64199cf00469262db1b07f5ef04' on ddr, client '/MC_DELETED/prod-fs-20240921_190514', rename to 'cur/DELETED/1727740800/51e83a682d09d64199cf00469262db1b07f5ef04_E0F1E861F424A74A6A31B8B4963A4CB036ACC77D', ddr result code: 5034, desc: the user has insufficient privilege
2024-09-29T06:14:35.625126-07:00 av-target ddrmaint.bin[4202]: Warning: gc-finish::process_all_cids - Failed to process backups in client dir '51e83a682d09d64199cf00469262db1b07f5ef04' for client '/MC_DELETED/prod-fs-20240921_190514'
admin@av-target:~/usr/local/avamar/var/ddrmaintlogs$

In the Avamar source server, there is a client with the following information:

CID: 51e83a682d09d64199cf00469262db1b07f5ef04
Client hostname: /clients/Windows/prod-fs

On September 21, this client was retired. The client was moved to /MC_RETIRED/prod-fs2024.09.21.01.23.45.

In the same day, this client was re-added creating a new client ID (CID) and using the same client hostname:

CID: 4140cb8f428aa413154139b186fdcaafa19eb04e
Client hostname: /clients/Windows/prod-fs

The replication policy does not contain /MC_RETIRED, and replication occurred.

On Avamar target (av-target) the /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log shows ddrmaint move-backup command:

admin@av-target:~/usr/local/avamar/var/ddrmaintlogs$ view ddrmaint.log
....
2024-09-21T12:05:22.483376-07:00 av-target ddrmaint.bin[14757]: Info: move-backup cmdline: '/usr/local/avamar/bin/ddrmaint.bin --flagfile=/usr/local/avamar/etc/usersettings.cfg --server=av-target --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --id=root --password=**************** --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc move-backup --source-client=51e83a682d09d64199cf00469262db1b07f5ef04 --target-client=4140cb8f428aa413154139b186fdcaafa19eb04e --backups=0x1db0aed1ab804ea:2 --dpnid=1234567890'
2024-09-21T12:05:23.604561-07:00 av-target ddrmaint.bin[14757]: Info: RL: As the DD is in Compliance Enabled mode move-backup::movebackups will be copy-backup with no retention on ddr index(2) from cur/51e83a682d09d64199cf00469262db1b07f5ef04/1DB0AED1AB804EA to cur/4140cb8f428aa413154139b186fdcaafa19eb04e/1DB0AED1AB804EA
2024-09-21T12:05:24.610402-07:00 av-target ddrmaint.bin[14757]: Info: move-backup Successfully copied (for move) client backup cur/51e83a682d09d64199cf00469262db1b07f5ef04/1DB0AED1AB804EA:2 to cur/4140cb8f428aa413154139b186fdcaafa19eb04e/1DB0AED1AB804EA:2.
...
admin@av-target:~/usr/local/avamar/var/ddrmaintlogs$

After 6 days (September 27) until September 29, there are messages that the backups of old CID no longer exist on GSAN:

admin@av-target:~/usr/local/avamar/var/ddrmaintlogs$ view ddrmaint.log
...
2024-09-27T06:21:50.895312-07:00 av-target ddrmaint.bin[8923]: Info: gc-finish::process_all_backups: Backup '1DB0A201463FE64' not on gsan, deleted on ddr, client dir '51e83a682d09d64199cf00469262db1b07f5ef04', client '/MC_DELETED/prod-fs-20240921_190514'
2024-09-27T06:21:50.910186-07:00 av-target ddrmaint.bin[8923]: Info: gc-finish::process_all_backups: Backup '1DB0A22594F1476' not on gsan, deleted on ddr, client dir '51e83a682d09d64199cf00469262db1b07f5ef04', client '/MC_DELETED/prod-fs-20240921_190514'
2024-09-28T06:19:59.618801-07:00 av-target ddrmaint.bin[27334]: Info: gc-finish::process_all_backups: Backup '1DB0AE8A240B1AA' not on gsan, deleted on ddr, client dir '51e83a682d09d64199cf00469262db1b07f5ef04', client '/MC_DELETED/prod-fs-20240921_190514'
2024-09-28T06:19:59.631409-07:00 av-target ddrmaint.bin[27334]: Info: gc-finish::process_all_backups: Backup '1DB0AED1AB804EA' not on gsan, deleted on ddr, client dir '51e83a682d09d64199cf00469262db1b07f5ef04', client '/MC_DELETED/prod-fs-20240921_190514'
...
admin@av-target:~/usr/local/avamar/var/ddrmaintlogs$

After 8 days, the error message "Failed to delete empty client dir" appears:

The Garbage Collection (GC) failure issue arose when clientA_with_cid-1 was retired and re-added as clientA_with_cid-2. During replication, the replicator created a new path for clientA_with_cid-2 at the target and copied old backups from clientA_with_cid-1 to clientA_with_cid-2. The GC process attempted to delete locked files, causing errors since the retention lock prevents deletion until the retention period expires.

The core problem occurred when the Garbage Collection (GC) process at the target failed to identify these orphan backups under the clientA_with_cid-1 directory path. The GC process tried to delete locked files, causing errors because retention lock prevents deletion until the retention period expires. This redundancy and inability to remove the orphan copy caused the GC failure.

Apply the following hotfix:

• 19.10.0.166: 338891
• 19.12.0.186: 338894