Data Domain: No se puede eliminar una instantánea bloqueada temporalmente por contexto de replicación
Summary: Instantánea encontrada, creada hace años, y que aún se conserva aunque esté vencida. No se puede eliminar porque está bloqueado temporalmente
Symptoms
- La instantánea creada hace años, aún conservada aunque ya marcada como caducada:
ddboost@dd02# snapshot list mtree /data/col1/avamar-1493130520 Snapshot Information for MTree: /data/col1/avamar-1493130520 ---------------------------------------------- Name Pre-Comp (GiB) Create Date Retain Until Status ----------------- -------------- ----------------- ----------------- ------- cp.20250309131143 213612.6 Mar 9 2025 07:13 Mar 10 2025 07:39 expired cp.20250309133359 214727.3 Mar 9 2025 07:34 Mar 10 2025 07:41 expired cp.20250310131141 207612.0 Mar 10 2025 07:13 cp.20250310133957 208726.9 Mar 10 2025 07:40 cp.20230919134341 403018.8 Sep 19 2023 07:44 Sep 20 2023 07:34 expired ----------------- -------------- ----------------- ----------------- -------
- La instantánea se muestra como parcialmente bloqueada en el archivo de registro ddfs.info
NOTA: La manera más sencilla de capturar ese archivo sería usar scp para copiarlo desde Data Domain (DD) a otra máquina Linux. Asegúrese de tener las credenciales de inicio de sesión a mano para utilizar esta sintaxis:
admin@anylinuxmachine:~/>: scp dd_username@dd_hostname:/ddr/var/log/debug/ddfs.info .
admin@anylinuxmachine:~/>: grep "cp.20230919134341" ddfs.info
03/11 05:54:20.569 (tid 0x7fb71d0e6f10): rmsnapshot: mid=1583172823 s_name=cp.20230919134341.
03/11 05:54:20.569 (tid 0x7fb71d0e6f10): _dm_rmsnapshot: snapshot 'cp.20230919134341' is softlocked.
- La alerta de Data Domain mostraba una alerta relacionada con un contexto de replicación desincronizado:
ddboost@dd02# alerts show current Id Post Time Severity Class Object Message ------- ------------------------ -------- ----------- ------------------ ---------------------------------------------------------------- p0-924 Tue Nov 28 09:35:07 2023 CRITICAL Network Interface Index=11 EVT-NETM-00001: Network interface connectivity is down on ethMc. m0-1971 Mon Mar 10 06:56:36 2025 WARNING Replication context=1 EVT-REPL-00006: Sync-as-of time is more than 13109 hours ago. ------- ------------------------ -------- ----------- ------------------ ---------------------------------------------------------------- There are 2 active alerts.
- Es posible enumerar los contextos de replicación en el origen de replicación de Data Domain para obtener los detalles de ese contexto #1.
- El contexto de replicación #1 hace referencia tanto al origen como al destino que hacen referencia al mtree donde se encuentra el bloqueo temporal de la instantánea.
avamar@dd02# replication show config CTX Source Destination Connection Host and Port --- ---------------------------------------------------------- ------------------------------------------------------------- ------------------------------------- 1 mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault.yourdomain.com.br/data/col1/avamar-1493130520 ddvault.yourdomain.com.br (default) 2 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01 mtree://ddvault02.vault/data/col1/SysDR_ppmsv01 ddvault02.vault (default) 3 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01 mtree://ddvault02.vault/data/col1/SysDR_ppmsv01repl ddvault02.vault (default) 4 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv02 mtree://dd03.yourdomain.com.br/data/col1/SysDR-R_ppmsv02 dd03.yourdomain.com.br (default) 5 mtree://dd02.yourdomain.com.br/data/col1/MtreeVault mtree://ddvault02.vault/data/col1/Mtreevaultdest ddvault02.vault (default) 6 mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault02.vault/data/col1/avamar-1493130520dest ddvault02.vault (default) 7 mtree://dd02.yourdomain.com.br/data/col1/CritMtreeVault mtree://ddvault02.vault/data/col1/CritMtreeVaultDest ddvault02.vault (default) 8 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01 mtree://ddvault02.vault/data/col1/SysDR_ppmsv01dest ddvault02.vault (default) --- ---------------------------------------------------------- --------------------------------------------------------- ------------------------------------- DD System default Max-repl-streams per context: 32
Cause
- Se quitó/eliminó un Data Domain del entorno, pero un contexto de replicación que hace referencia a ese Data Domain eliminado no se eliminó, por lo que se dejó atrás.
Resolution
- Tendremos que interrumpir (es el proceso para eliminar ese contexto de replicación), del Data Domain de origen de replicación.
- Inicie sesión en Data Domain y enumere la configuración de replicación para revisar y confirme el contexto que se interrumpirá/eliminará (en este ejemplo, contexto #1):
ddboost@dd02# replication show config CTX Source Destination Connection Host and Port --- ---------------------------------------------------------- ------------------------------------------------------------- ------------------------------------- 1 mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault.yourdomain.com.br/data/col1/avamar-1493130520 ddvault.yourdomain.com.br (default) 2 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01 mtree://ddvault02.vault/data/col1/SysDR_ppmsv01 ddvault02.vault (default) 3 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01 mtree://ddvault02.vault/data/col1/SysDR_ppmsv01repl ddvault02.vault (default) 4 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv02 mtree://dd03.yourdomain.com.br/data/col1/SysDR-R_ppmsv02 dd03.yourdomain.com.br (default) 5 mtree://dd02.yourdomain.com.br/data/col1/MtreeVault mtree://ddvault02.vault/data/col1/Mtreevaultdest ddvault02.vault (default) 6 mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault02.vault/data/col1/avamar-1493130520dest ddvault02.vault (default) 7 mtree://dd02.yourdomain.com.br/data/col1/CritMtreeVault mtree://ddvault02.vault/data/col1/CritMtreeVaultDest ddvault02.vault (default) 8 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01 mtree://ddvault02.vault/data/col1/SysDR_ppmsv01dest ddvault02.vault (default) --- ---------------------------------------------------------- ------------------------------------------------------------- ------------------------------------- DD System default Max-repl-streams per context: 32
- Ejecute el comando para interrumpir ese contexto de replicación:
ddboost@dd02# replication break rctx://1
The 'replication break' command irrevocably turns off logical
replication from this mtree. To reconfigure the mtree
for replication, the destination mtree must not exist,
or, alternatively, 'replication resync' must be used.
Are you sure? (yes|no) [no]: yes
ok, proceeding.
ddboost@dd02#
- Vuelva a enumerar la configuración de replicación para asegurarse de que el contexto de replicación #1 haya desaparecido:
ddboost@dd02# replication show config CTX Source Destination Connection Host and Port --- ---------------------------------------------------------- -------------------------------------------------------- ---------------------------------- 2 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01 mtree://ddvault02.vault/data/col1/SysDR_ppmsv01 ddvault02.vault (default) 3 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01 mtree://ddvault02.vault/data/col1/SysDR_ppmsv01repl ddvault02.vault (default) 4 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv02 mtree://dd03.yourdomain.com.br/data/col1/SysDR-R_ppmsv02 dd03.yourdomain.com.br (default) 5 mtree://dd02.yourdomain.com.br/data/col1/MtreeVault mtree://ddvault02.vault/data/col1/Mtreevaultdest ddvault02.vault (default) 6 mtree://dd02.yourdomain.com.br/data/col1/avamar-1493130520 mtree://ddvault02.vault/data/col1/avamar-1493130520dest ddvault02.vault (default) 7 mtree://dd02.yourdomain.com.br/data/col1/CritMtreeVault mtree://ddvault02.vault/data/col1/CritMtreeVaultDest ddvault02.vault (default) 8 mtree://dd02.yourdomain.com.br/data/col1/SysDR_ppmsv01 mtree://ddvault02.vault/data/col1/SysDR_ppmsv01dest ddvault02.vault (default) --- ---------------------------------------------------------- -------------------------------------------------------- ---------------------------------- DD System default Max-repl-streams per context: 32
- Verifique una vez que se elimine el contexto de replicación #1, esa instantánea antigua se marque automáticamente como no vencida:
ddboost@dd02# snapshot list mtree /data/col1/avamar-1493130520 Snapshot Information for MTree: /data/col1/avamar-1493130520 ---------------------------------------------- Name Pre-Comp (GiB) Create Date Retain Until Status ----------------- -------------- ----------------- ----------------- ------- cp.20250317130903 214805.7 Mar 17 2025 07:10 Mar 18 2025 07:32 expired cp.20250317132705 215924.2 Mar 17 2025 07:28 Mar 18 2025 07:33 expired cp.20250318131055 215418.1 Mar 18 2025 07:12 cp.20250318133213 216537.6 Mar 18 2025 07:33 cp.20230919134341 403018.8 Sep 19 2023 07:44 Apr 8 2025 13:25 ----------------- -------------- ----------------- ----------------- -------
- Vuelva a marcar esa instantánea como vencida:
ddboost@dd02# snapshot expire cp.20230919134341 mtree /data/col1/avamar-1493130520
Snapshot "cp.20230919134341" for mtree "/data/col1/avamar-1493130520" will be retained until Mar 18 2025 13:30.
ddboost@dd02# snapshot list mtree /data/col1/avamar-1493130520
Snapshot Information for MTree: /data/col1/avamar-1493130520
----------------------------------------------
Name Pre-Comp (GiB) Create Date Retain Until Status
----------------- -------------- ----------------- ----------------- -------
cp.20250318131055 215418.1 Mar 18 2025 07:12
cp.20250318133213 216537.6 Mar 18 2025 07:33
cp.20230919134341 403018.8 Sep 19 2023 07:44 Mar 18 2025 13:30 expired
----------------- -------------- ----------------- ----------------- -------
- Se captura la búsqueda de esa instantánea en una copia actualizada de ddfs.info, en el momento en que se elimina el bloqueo suave:
admin@anylinuxmachine:~/>: grep cp.20230919134341 ddfs.info
03/18 13:25:16.663 (tid 0x7fb7969b2b40): updatesnapshot: mid=1583172823 m_name=avamar-1493130520 sid=35512 name=cp.20230919134341 retention is changed from 1741692419 to 1744140316 err :0
03/18 13:25:16.666 (tid 0x7fb7969b2b40): mrepl_snap_deref deref name=cp.20230919134341 (1583172823:35512) 1 -> 0
03/18 13:25:16.669 (tid 0x7fb7969b2b40): repl ctx 1: mrepl_protect_new_base deref'ing base cp.20230919134341 Not Referenced, USER
03/18 13:25:18.680 (tid 0x7fb7969b2b40): repl ctx: mrepl_snap_ref_cleanup Unsoftlocking snapshot name=cp.20230919134341 (1583172823:35512)
03/18 13:30:41.710 (tid 0x7fb7969b2b40): updatesnapshot: mid=1583172823 m_name=avamar-1493130520 sid=35512 name=cp.20230919134341 retention is changed from 1744140316 to 1742326241 err :0
03/18 13:30:41.710 (tid 0x7fb7969b2b40): FM fms_snapshot_update:1933 - Updated "cp.20230919134341", mtree "/data/col1/avamar-1493130520" 1742326241
03/18 13:32:50.061 (tid 0x7fb79ba174e0): rmsnapshot: mid=1583172823 s_name=cp.20230919134341.
03/18 13:32:50.076 (tid 0x7fb79ba174e0): rmsnapshot: done, mid=1583172823 m_name=avamar-1493130520 sid=35512 s_name=cp.20230919134341 retention=1742326241 ch=<dm_mtree:40000010000,1,0,400,6,0,e98eb815,440357a6,18045d05,d3463134,cadda83,9dfa4504:> cpg_id=1 cp_uuid=cf984be60de2e768:887b264e6a0bd7d6.
admin@anylinuxmachine:~/>:
- Iniciar una nueva limpieza del sistema de archivos
ddboost@dd02# fi clean start Cleaning started. Use 'filesys clean watch' to monitor progress. ddboost@dd02# fi clean status Cleaning started at 2025/03/18 13:32:49: phase 1 of 6 (pre-merge) 0.0% complete, 25616 GiB free; time: phase 0:00:14, total 0:00:14
- Verifique que la instantánea antigua ya no esté, aunque la limpieza aún esté en curso.
- El problema está resuelto:
ddboost@dd02# snapshot list mtree /data/col1/avamar-1493130520 Snapshot Information for MTree: /data/col1/avamar-1493130520 ---------------------------------------------- Name Pre-Comp (GiB) Create Date Retain Until Status ----------------- -------------- ----------------- ----------------- ------ cp.20250318131055 215418.1 Mar 18 2025 07:12 cp.20250318133213 216537.6 Mar 18 2025 07:33 ----------------- -------------- ----------------- ----------------- ------