Data Domain: DDFS PANIC: dmbt_update_entry_nolock

The affected Data Domain (DD) reports a panic with the following message for systems with Automatic Retention Lock (ARL) enabled mtrees used for replication:

PANIC: ddr/dm/dmbt_iface.c: dmbt_update_entry_nolock

• PowerProtect Data Domains with Automatic Retention Lock (ARL) Enabled.

• Crash of the Data Domain file system
• System instability 
• Logs indicating the above panic message immediately before a system crash

This issue arises when a mtree replication snapshot occurs between the DDBOOST open and close calls.

• The nfsproc3_ddcp_open_file_3_svc() function identifies a Retention lock (RL) file being modified within the ARL cool-off-period (COP) and resets the tag value to 0 (DM_TAG_DEFAULT).
• The subsequent nfsproc3_ddcp_close_file_3_svc() function normally resets the tag to a valid value.
• However, if a file is created with an extended attribute and is modified within the ARL COP, and a snapshot occurs between the DDBOOST open and close calls, this sequence can trigger the PANIC.

This occurs on 7.7.x and 7.10.x, and the trigger is when retention lock (usually ARL) is enabled on the mtrees.

To resolve this issue, do the following: 

1. Break any replication context of the original source and destination mtree.
   1. replication break destination
2. Fastcopy the problematic mtree on the source DD. This creates a new, clean copy of the mtree without the problematic context.
   1. filesys fastcopy [retention-lock [new-lock-duration]] source src destination dest
3. Do not enable Automatic Retention Lock (ARL) on the new "fastcopied" mtree. This prevents the issue from recurring.
4. Redirect backups to the newly fastcopied mtree on Source the DD.
   Update any backup jobs to target the new mtree.  The steps needed here are system configuration specific.  Make the changes as needed for the backup application or protocol used for backup.  Modify the fastcopy mtree to a DDBoost storage unit if needed.
5. Create a new replication context between the newly fastcopied mtree and the destination DD.
   1. replication add source source destination destination [low-bw-optim {enabled | disabled}] [encryption {enabled [authentication-mode {one-way | two-way | anonymous}] | disabled}] [propagate-retention-lock {enabled | disabled}] [ipversion {ipv4 | ipv6}] [max-repl-streams n] [destination-tenant-unit tenant-unit]
6. Ensure that the new replication setup is fully synchronized.
   1. replication status [destination | tenant-unit tenant-unit | all] [detailed]
      Show the current status of replication.
7. Keep this mtree ARL disabled from preventing the PANIC to recur. 
   1. Leave the original mtrees on both the source and destination systems for the time being.
   2. Once the new replication is synchronized and the retention period for the old mtree files has expired, delete the files in the old mtrees.
   3. Then delete the old mtree on both the source and destination.

• It is crucial to monitor the system for any further anomalies after performing the above steps.
• Ensure that all backup policies and replication contexts are correctly updated to reflect the changes.