PPDM: Microsoft App Agent Backup Fails for Large Database When Using Data Domain Retention Lock

All other databases complete successfully, but one large database fails after 18 hours. In SQL, we see that the backup goes to 100% completion, but then fails at the end when closing a file.

2024-01-18T05:58:32.904Z [TRACE_ID:b0158f733e817c46JOB_ID:8ec0693fe01c6f68]154540 ddbmsqlsv PowerProtect Data Manager info BACKUP database [MTV] TO virtual_device='EMC#8e445e6a-3c22-4ff8-89ec-57965c19b486' WITH name=N'DDBMMSSQL', description=N'SACSQLIDS100A-FULL:MTV', mediadescription=N'5edf1429-59c3-409b-bdb7-9e3634a4d466', maxtransfersize = 4194304, EXPIREDATE=N'2024-02-16T21:55:21' . 20
24-01-19T01:58:17.450Z 38008 ddbmsqlsv PowerProtect Data Manager info Processed 52593328 pages for database 'MTV', file 'MTVPRD' on file 1.
2024-01-19T01:58:20.184Z [TRACE_ID:b0158f733e817c46JOB_ID:8ec0693fe01c6f68]107497 ddbmsqlsv PowerProtect Data Manager info Closing a file failed [5034] ([12752] [8516] Thu Jan 18 17:58:20 2024 ddp_close_file() failed, Err: 5034-async writeback or commit failure ).
2024-01-19T01:58:20.184Z [TRACE_ID:b0158f733e817c46JOB_ID:8ec0693fe01c6f68]38008 ddbmsqlsv PowerProtect Data Manager critical XBSA Internal system error.
2024-01-19T01:58:20.230Z [TRACE_ID:b0158f733e817c46JOB_ID:8ec0693fe01c6f68]172257 ddbmsqlsv PowerProtect Data Manager critical Backup of MTV failed.
2024-01-19T01:58:20.230Z [TRACE_ID:b0158f733e817c46]0 ddbmsqlsv PowerProtect Data Manager info Close agentbus write connection.
2024-01-19T01:58:20.246Z [TRACE_ID:b0158f733e817c46]0 ddbmsqlsv PowerProtect Data Manager info Successfully closed the agentbus write connection. 2024-01-19T01:58:20.246Z [TRACE_ID:b0158f733e817c46]0 ddbmsqlsv PowerProtect Data Manager info Failed databases ================
2024-01-19T01:58:20.246Z [TRACE_ID:b0158f733e817c46]0 ddbmsqlsv PowerProtect Data Manager info MTV
2024-01-19T01:58:20.246Z [TRACE_ID:b0158f733e817c46]0 ddbmsqlsv PowerProtect Data Manager info
2024-01-19T01:58:20.246Z [TRACE_ID:b0158f733e817c46]43709 ddbmsqlsv PowerProtect Data Manager notice Stop time: Thu Jan 18 17:58:20 2024,

This is due to Data Domain Retention Lock delay setting being less than the time that it takes to complete the backup.

The retention lock delay defines the time period after which a file was modified before the retention lock time period is applied. The minimum value that this can be set to is 5 minutes and the default value is 120 minutes.

This causes the backup to run to completion, but then fails to write the meta data at the end if the backup of the database took longer than the retention lock delay setting. For example, the default retention lock delay is 12 hours, but the backup took 18 hours. As a result, it failed to write the meta data at the end and that explains why only large databases are affected.

From the Data Domain UI, check under Data Management > MTree > DD Retention Lock.

If enabled, verify what the retention lock delay is set to, and increase it to be longer than the longest database backup.

Or run the below command to check the current retention lock delay setting.

mtree retention-lock status mtree