Data Domain：无法删除被复制上下文软锁定的快照

Summary: 找到几年前创建的快照，即使过期，快照仍会保留。无法删除，因为它是软锁定的

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

- 几年前创建的快照，尽管已标记为已过期，但仍保留：

ddboost@dd02# snapshot list mtree /data/col1/avamar-1493130520
Snapshot Information for MTree: /data/col1/avamar-1493130520
----------------------------------------------
Name                Pre-Comp (GiB)   Create Date         Retain Until        Status
-----------------   --------------   -----------------   -----------------   -------
cp.20250309131143         213612.6   Mar  9 2025 07:13   Mar 10 2025 07:39   expired
cp.20250309133359         214727.3   Mar  9 2025 07:34   Mar 10 2025 07:41   expired
cp.20250310131141         207612.0   Mar 10 2025 07:13
cp.20250310133957         208726.9   Mar 10 2025 07:40
cp.20230919134341         403018.8   Sep 19 2023 07:44   Sep 20 2023 07:34   expired 
-----------------   --------------   -----------------   -----------------   -------

- 快照在日志文件中显示为软锁定 ddfs.info
注意：捕获该文件的最简单方法是使用 scp 将其从 Data Domain （DD） 复制到另一台 Linux 计算机。确保手头有登录凭据才能使用以下语法：

admin@anylinuxmachine:~/>: scp dd_username@dd_hostname:/ddr/var/log/debug/ddfs.info .

 

admin@anylinuxmachine:~/>: grep "cp.20230919134341" ddfs.info
03/11 05:54:20.569 (tid 0x7fb71d0e6f10): rmsnapshot: mid=1583172823 s_name=cp.20230919134341.
03/11 05:54:20.569 (tid 0x7fb71d0e6f10): _dm_rmsnapshot: snapshot 'cp.20230919134341' is softlocked.

- Data Domain 警报显示一个与一个复制上下文不同步相关的警报：

ddboost@dd02# alerts show current
Id      Post Time                Severity Class       Object             Message															
------- ------------------------ -------- ----------- ------------------ ----------------------------------------------------------------
p0-924  Tue Nov 28 09:35:07 2023 CRITICAL Network     Interface Index=11 EVT-NETM-00001: Network interface connectivity is down on ethMc.
m0-1971 Mon Mar 10 06:56:36 2025 WARNING  Replication context=1          EVT-REPL-00006: Sync-as-of time is more than 13109 hours ago.
------- ------------------------ -------- ----------- ------------------ ----------------------------------------------------------------
There are 2 active alerts.

- 列出 Data Domain 复制源中的复制上下文可以获取有关该上下文 #1 的详细信息。
- 复制上下文 #1 的源和目标都引用包含快照软锁定的 mtree。

avamar@dd02# replication show config
CTX Source                                                 	   Destination                                               	 Connection                        
																															 Host and Port                     
--- ---------------------------------------------------------- ------------------------------------------------------------- ------------------------------------- 
1   mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault.yourdomain.com.br/data/col1/avamar-1493130520 ddvault.yourdomain.com.br   (default) 
2   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01     mtree://ddvault02.vault/data/col1/SysDR_ppmsv01               ddvault02.vault         	 (default) 
3   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01     mtree://ddvault02.vault/data/col1/SysDR_ppmsv01repl           ddvault02.vault         	 (default) 
4   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv02     mtree://dd03.yourdomain.com.br/data/col1/SysDR-R_ppmsv02      dd03.yourdomain.com.br      (default) 
5   mtree://dd02.yourdomain.com.br/data/col1/MtreeVault        mtree://ddvault02.vault/data/col1/Mtreevaultdest          	 ddvault02.vault         	 (default) 
6   mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault02.vault/data/col1/avamar-1493130520dest   	 ddvault02.vault         	 (default) 
7   mtree://dd02.yourdomain.com.br/data/col1/CritMtreeVault    mtree://ddvault02.vault/data/col1/CritMtreeVaultDest      	 ddvault02.vault         	 (default) 
8   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01     mtree://ddvault02.vault/data/col1/SysDR_ppmsv01dest       	 ddvault02.vault         	 (default) 
--- ---------------------------------------------------------- --------------------------------------------------------- 	 ------------------------------------- 
DD System default Max-repl-streams per context: 32

 

提醒：在故障处理期间，名称为 ddvault.yourdomain.com.br 的已诊断目标主机不再存在。它已关闭并删除。

Cause

在源 Data Domain 上引用快照所在的 mtree 的旧的且不再使用的复制上下文持有该软锁。
- Data Domain 已从环境中移除/删除，但引用已删除 Data Domain 的复制上下文未删除，因此被留下。 

Resolution

- 我们需要从复制源 Data Domain 中断（这是删除该复制上下文的过程）。
- 登录到 Data Domain 并列出复制配置，以查看并确认要中断/删除的上下文（在此示例中为上下文 #1）：

ddboost@dd02# replication show config
CTX Source                                                 	   Destination                                               	 Connection                       
																															 Host and Port                    
--- ---------------------------------------------------------- ------------------------------------------------------------- -------------------------------------
1   mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault.yourdomain.com.br/data/col1/avamar-1493130520 ddvault.yourdomain.com.br   (default)
2   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01     mtree://ddvault02.vault/data/col1/SysDR_ppmsv01           	 ddvault02.vault         	 (default)
3   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01     mtree://ddvault02.vault/data/col1/SysDR_ppmsv01repl       	 ddvault02.vault         	 (default)
4   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv02     mtree://dd03.yourdomain.com.br/data/col1/SysDR-R_ppmsv02      dd03.yourdomain.com.br      (default)
5   mtree://dd02.yourdomain.com.br/data/col1/MtreeVault        mtree://ddvault02.vault/data/col1/Mtreevaultdest          	 ddvault02.vault         	 (default)
6   mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault02.vault/data/col1/avamar-1493130520dest   	 ddvault02.vault         	 (default)
7   mtree://dd02.yourdomain.com.br/data/col1/CritMtreeVault    mtree://ddvault02.vault/data/col1/CritMtreeVaultDest      	 ddvault02.vault         	 (default)
8   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01     mtree://ddvault02.vault/data/col1/SysDR_ppmsv01dest       	 ddvault02.vault         	 (default)
--- ---------------------------------------------------------- ------------------------------------------------------------- -------------------------------------
DD System default Max-repl-streams per context: 32

- 发出命令以中断该复制上下文：

ddboost@dd02# replication break rctx://1
The 'replication break' command irrevocably turns off logical
replication from this mtree.  To reconfigure the mtree
for replication, the destination mtree must not exist,
or, alternatively, 'replication resync' must be used.
        Are you sure? (yes|no) [no]: yes
ok, proceeding.
ddboost@dd02#
提醒：中断执行期间显示的消息以独占方式应用于要删除的复制上下文。它不会影响其余的复制上下文。


- 再次列出复制配置，以确保复制上下文 #1 消失：

ddboost@dd02# replication show config
CTX Source                                                 	   Destination                                             	Connection                    
																													    Host and Port                 
--- ---------------------------------------------------------- -------------------------------------------------------- ----------------------------------
2   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01     mtree://ddvault02.vault/data/col1/SysDR_ppmsv01         	ddvault02.vault      	 (default)
3   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01     mtree://ddvault02.vault/data/col1/SysDR_ppmsv01repl     	ddvault02.vault      	 (default)
4   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv02     mtree://dd03.yourdomain.com.br/data/col1/SysDR-R_ppmsv02 dd03.yourdomain.com.br   (default)
5   mtree://dd02.yourdomain.com.br/data/col1/MtreeVault        mtree://ddvault02.vault/data/col1/Mtreevaultdest        	ddvault02.vault      	 (default)
6   mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault02.vault/data/col1/avamar-1493130520dest 	ddvault02.vault      	 (default)
7   mtree://dd02.yourdomain.com.br/data/col1/CritMtreeVault    mtree://ddvault02.vault/data/col1/CritMtreeVaultDest    	ddvault02.vault      	 (default)
8   mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01     mtree://ddvault02.vault/data/col1/SysDR_ppmsv01dest     	ddvault02.vault      	 (default)
--- ---------------------------------------------------------- -------------------------------------------------------- ----------------------------------
DD System default Max-repl-streams per context: 32

- 删除复制上下文 #1 后，验证旧快照会自动标记为未过期：

ddboost@dd02# snapshot list mtree /data/col1/avamar-1493130520
Snapshot Information for MTree: /data/col1/avamar-1493130520
----------------------------------------------
Name              Pre-Comp (GiB) Create Date        Retain Until      Status
----------------- -------------- -----------------  ----------------- -------
cp.20250317130903       214805.7 Mar 17 2025 07:10  Mar 18 2025 07:32 expired
cp.20250317132705       215924.2 Mar 17 2025 07:28  Mar 18 2025 07:33 expired
cp.20250318131055       215418.1 Mar 18 2025 07:12
cp.20250318133213       216537.6 Mar 18 2025 07:33
cp.20230919134341       403018.8 Sep 19 2023 07:44  Apr  8 2025 13:25			
----------------- -------------- -----------------  ----------------- -------

- 再次将该快照标记为已过期：

ddboost@dd02# snapshot expire cp.20230919134341 mtree /data/col1/avamar-1493130520
Snapshot "cp.20230919134341" for mtree "/data/col1/avamar-1493130520" will be retained until Mar 18 2025 13:30.

ddboost@dd02# snapshot list mtree /data/col1/avamar-1493130520
Snapshot Information for MTree: /data/col1/avamar-1493130520
----------------------------------------------
Name              Pre-Comp (GiB) Create Date       Retain Until      Status
----------------- -------------- ----------------- ----------------- -------
cp.20250318131055       215418.1 Mar 18 2025 07:12
cp.20250318133213       216537.6 Mar 18 2025 07:33
cp.20230919134341       403018.8 Sep 19 2023 07:44 Mar 18 2025 13:30 expired 
----------------- -------------- ----------------- ----------------- -------

- 在 ddfs.info 的更新副本中查找该快照，捕获删除软锁的那一刻：

admin@anylinuxmachine:~/>: grep cp.20230919134341 ddfs.info
03/18 13:25:16.663 (tid 0x7fb7969b2b40): updatesnapshot: mid=1583172823 m_name=avamar-1493130520 sid=35512 name=cp.20230919134341 retention is changed from 1741692419 to 1744140316 err :0
03/18 13:25:16.666 (tid 0x7fb7969b2b40): mrepl_snap_deref deref name=cp.20230919134341 (1583172823:35512) 1 -> 0
03/18 13:25:16.669 (tid 0x7fb7969b2b40): repl ctx 1: mrepl_protect_new_base deref'ing base cp.20230919134341 Not Referenced, USER
03/18 13:25:18.680 (tid 0x7fb7969b2b40): repl ctx: mrepl_snap_ref_cleanup Unsoftlocking snapshot name=cp.20230919134341 (1583172823:35512) 
03/18 13:30:41.710 (tid 0x7fb7969b2b40): updatesnapshot: mid=1583172823 m_name=avamar-1493130520 sid=35512 name=cp.20230919134341 retention is changed from 1744140316 to 1742326241 err :0
03/18 13:30:41.710 (tid 0x7fb7969b2b40): FM fms_snapshot_update:1933 - Updated "cp.20230919134341", mtree "/data/col1/avamar-1493130520" 1742326241
03/18 13:32:50.061 (tid 0x7fb79ba174e0): rmsnapshot: mid=1583172823 s_name=cp.20230919134341.
03/18 13:32:50.076 (tid 0x7fb79ba174e0): rmsnapshot: done, mid=1583172823 m_name=avamar-1493130520 sid=35512 s_name=cp.20230919134341 retention=1742326241 ch=<dm_mtree:40000010000,1,0,400,6,0,e98eb815,440357a6,18045d05,d3463134,cadda83,9dfa4504:> cpg_id=1 cp_uuid=cf984be60de2e768:887b264e6a0bd7d6.
admin@anylinuxmachine:~/>:

- 启动新的文件系统清理

ddboost@dd02# fi clean start
Cleaning started.  Use 'filesys clean watch' to monitor progress.

ddboost@dd02# fi clean status
Cleaning started at 2025/03/18 13:32:49: phase 1 of 6 (pre-merge)
  0.0% complete, 25616 GiB free; time: phase  0:00:14, total  0:00:14

- 验证旧快照现在是否已消失，即使清理仍在进行中。
- 问题已解决：

ddboost@dd02# snapshot list mtree /data/col1/avamar-1493130520
Snapshot Information for MTree: /data/col1/avamar-1493130520
----------------------------------------------
Name              Pre-Comp (GiB) Create Date       Retain Until      Status
----------------- -------------- ----------------- ----------------- ------
cp.20250318131055       215418.1 Mar 18 2025 07:12
cp.20250318133213       216537.6 Mar 18 2025 07:33
----------------- -------------- ----------------- ----------------- ------

Affected Products

Data Domain Boost – File System
Article Properties
Article Number: 000296945
Article Type: Solution
Last Modified: 24 Mar 2025
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.