VNX: Cleaning up replication when Replication Root Checkpoints are corrupted/inactive (User Correctable)

Summary: Cleaning up replication when Replication Root Checkpoints are corrupted/inactive (User Correctable)

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Root checkpoints corrupted (common reason is backend luns has uncorrectables due to disk failures)
  
[nasadmin@CS0 ~]$ server_mount ALL | grep unmount
root_rep_ckpt_28_242474_1 on /root_rep_ckpt_28_242474_1 ckpt,perm,ro,<unmounted>
root_rep_ckpt_28_242474_2 on /root_rep_ckpt_28_242474_2 ckpt,perm,ro,<unmounted>
root_rep_ckpt_27_242517_1 on /root_rep_ckpt_27_242517_1 ckpt,perm,ro,<unmounted>
root_rep_ckpt_27_242517_2 on /root_rep_ckpt_27_242517_2 ckpt,perm,ro,<unmounted>

In this above output, only root checkpoints can be seen and this indicates that, main file system is good. This happens usually if savvol is built on different Storage Pool which had corruption and the associated file system is safe. 
[nasadmin@CS0 ~]$ nas_replicate -l
Name                      Type       Local Mover               Interconnect         Celerra      Status
rep_fs1                filesystem server_2                  -->Replication       Remote_CS  Critical 8865448248:The replication session encountered an error that halted progress.
rep_fs2                filesystem server_2                  -->Replication       Remote_CS  OK
rep_fs3                filesystem server_2                  -->Replication       Remote_CS  Critical 8865448248:The replication session encountered an error that halted progress.
Above output will reflect the error as evident from the 2 replications critical errors (which matches the 2 set of root checkpoint unmount status seen in server_mount output)

Cause

If the data mover panics, most cases, this is caused by corruption due to disks and VNX will mark the file system as corrupt. Root checkpoints will become unmounted also.

Resolution

Note: The backend issues need to be fixed first, like if any disks needs replacement, that need to be first. 

To delete affected replications:

If the sessions are directly attempted to delete, the delete task might appear as hung. Because, the deletion will try to update the root checkpoints and the update will hang because it is corrupted. Refer notes section for fixing that. This procedure is to fix in correct order and need to be done from control station via command prompt.

1) Login to the control station as nasadmin

2) Identify the replication session name by following this example: 
a) Find the full name of root checkpoints by:
[nasadmin@CS0 ~]$ server_mount ALL | grep unmount
root_rep_ckpt_28_242474_1 on /root_rep_ckpt_28_242474_1 ckpt,perm,ro,<unmounted>
root_rep_ckpt_28_242474_2 on /root_rep_ckpt_28_242474_2 ckpt,perm,ro,<unmounted>
b) For each checkpoint, issue the following command and note the file system name
[nasadmin@CS0 ~]$ /nas/sbin/rootnas_fs -info root_rep_ckpt_28_242474_1 | grep checkpt_of
checkpt_of= fs1 Mon Jun 15 16:51:54 EDT 2015

Repeat above as every FS will have 2 root checkpoints per replication session, so good to get all FS names first before proceeding to next step

2) Identify the replication checkpoints and delete, by following this example
  
a) Identify the file system name from the replications failing with critical errors by :
nas_replicate -list

Example: 
$ nas_replicate -i jsq-stx-mq
ID                             = 156_APM001_01F4_137_APM002_01F4
Name                           = rep_fs1
Source Status                  = Critical 8865448248: The replication session encountered an error that halted progres                                                                                          s.
Network Status                 = OK
Destination Status             = OK
Last Sync Time                 = Wed Jul 13 14:35:15 EDT 2016
Type                           = filesystem
Celerra Network Server         = CS01
Dart Interconnect              = Replication
Peer Dart Interconnect         = Replication
Replication Role               = source  <== note the role 
Source Filesystem              = fs1 <== this is the fs name if the role is source
Source Data Mover              = server_2
Source Interface               = 10.x.x.x
Source Control Port            = 0
Source Current Data Port       = 0
Destination Filesystem         = fs1-DR <== this is the fs name if the role is destination
Destination Data Mover         = server_2
Destination Interface          = 10.x.x.x
...

Match this name from names identified in step 2 to ensure they are same.
b) Check the replication checkpoint status by : fs_ckpt <fs_name> -list -all

Example: 
$ fs_ckpt fs1 -list -all
id    ckpt_name                creation_time           inuse fullmark   total_savvol_used  ckpt_usage_on_savvol
32    root_rep_ckpt_28_242474_ 06/15/2015-16:51:54-EDT   y   90%        INACTIVE           N/A
33    root_rep_ckpt_28_242474_ 06/15/2015-16:51:56-EDT   y   90%        INACTIVE           N/A
34    fs1_ckpt1                          06/17/2015-16:51:56-EDT   y   90%        INACTIVE           N/A
Info 26306752329: The value of ckpt_usage_on_savvol for read-only checkpoints may not be consistent with the total_savvol_used.

id    wckpt_name               inuse fullmark total_savvol_used  base  ckpt_usage_on_savvol

INACTIVE indicates, this is corrupted.
c) If the "inuse" value is "y", delete the root checkpoints using the following command:  /nas/sbin/rootnas_fs -delete id=<root_ckpt_id> -o umount=yes -ALLOW_REP_INT_CKPT_OP 
Rarely, root checkpoint may have, inuse" value as "n", then,  /nas/sbin/rootnas_fs -delete id=<root_ckpt_id> -ALLOW_REP_INT_CKPT_OP 

Example:
[nasadmin@CS0 ~]$ /nas/sbin/rootnas_fs -delete id=32  -o umount=yes -ALLOW_REP_INT_CKPT_OP 
id        = 32
name      = root_rep_ckpt_28_242474_1
acl       = 0
in_use    = True
type      = ckpt
worm      = off
..

d) Repeat above step and delete all the root checkpoints 
e) For non root checkpoints, delete using same command with out the last arument (Example: /nas/sbin/rootnas_fs -delete id=34 -o umount=yes)
3) Delete the replication session using following syntax: 
For the "source" replication role : nas_replicate -delete <replication_session_name> -mode source -background
For the "destination" replication role : nas_replicate -delete <replication_session_name> -mode destination -background

Above command will give a task number, which can be used to view the status, by "nas_task -i <task_number>

4) Delete the replication on the remote side too using just the above step 3 
 [ "nas_replicate -delete <session_name> -mode <mode> -background" ] 
If all above steps done, this completes the replication delete activity. A new replication can be reconfigured using Unisphere or command prompt.

Contact Dell EMC Technical Support if any of the above step fails.  Quote this Knowledgebase article ID.

Additional Information

If the sessions are directly attempted to delete on either Unisphere or via "nas_replicate -delete" command, the delete task might appear as hung. In most cases, this can be fixed without the need to reboot the data mover.

1) Identify the task via : 
nas_task -list | grep -i run

2) Find the detailed information (particularly the data mover name) by : 
nas_task -info <task_number>

3) Abort the task by : 
nas_task -abort <task_numer> -mover <data_mover_name>

Above also should delete the replication session and the root checkpoints. Any user checkpoints need to be deleted manually.

Affected Products

vVNX
Article Properties
Article Number: 000056557
Article Type: Solution
Last Modified: 29 Jul 2025
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.