Break and Resync Replication.

PURPOSE

This article lists steps on how to perform a replication resynchronization (resync) procedure on a system configured for directory or pool replication. For a more in-depth explanation of the purpose of a resync procedure, see Data Domain: Replication Resync Explanation.

• Directory Replication lag exceeds the data retention period: Data Domain: Measuring Replication Throughput
• Data Domain: Cannot Recover from NVRAM: Recently Written Data May be Lost
• Cleaning not recovering disk space on the source system.

• All Data Domain models configured for Directory or Pool Replication
• All Software Releases

SOLUTION

The steps listed in this solution assume a basic replication configuration where a source Data Domain is configured to write to a destination Data Domain. Replication configurations can be more complex consisting of bi-directional replication and chained configurations involving multiple systems.

If a context is broken, the partner system must also have the related context broken.

The break and resynchronization procedure requires that the file system to be temporarily unavailable for all systems involved.

This procedure can be performed using the Enterprise Manager UI (Option 1) or the command-line (Option 2).

The EM process is simpler, and can be completed with a few clicks.

Option 1: Break the existing replication and resync using the Enterprise Manager:

a. Break existing replication:

This removes the context from both the source and destination DDRs.

i. Go to the Enterprise Manager page for the source Data Domain, then choose "Replication"

ii. Select the context to break.

iii. Select "Delete Pair"

iv. Click "OK"

b. Resync the context:

This adds the context back to both the source and destination DDRs and start the resync process. The resync process can between several hours and several days, depending on the size of the system and current load factors.

i. From the same Replication tab in the Enterprise Manager, click "More", then choose "Start Resync".

ii. Select the Replication Type.

iii. Select the source and destination host names.

iv. Path out the replicated directory or mtree on both the source and destination DDRs

v. Click "OK"

-- Or --

Option 2. Command-line manual break and resync option:

a. Copy and save the existing replication configuration information from the source and destination systems.

At the command prompt on both the source and destination type:

# replication show config

CTX   Source                                                   Destination                                            Connection Host and Port                 Enabled
---   ------------------------------------------------------   ----------------------------------------------------   --------------------------------------   -------
1     dir://testdatadomain.company.com/backup/monthly          dir://test2datadomain.company.com/backup/monthlytest   testdatadomain.company.com   (default)   yes
2     dir://testdatadomain.company.com/backup/weekly           dir://test2datadomain.company.com/backup/weeklytest    testdatadomain.company.com   (default)   yes
---   ------------------------------------------------------   ----------------------------------------------------   --------------------------------------   -------

Copy both outputs temporarily to a text editor (such as notepad) and label each section appropriately. This information is needed when re-creating the replication contexts. Record the context numbers if targeting specific contexts.

(Ensure that backups have been paused (or any other systems using the Data Domain) and cleaning is not running.)

# system show stats

07/15 15:04:40

INTERVAL: 5 secs
--------------------------------------------------------------
CPU  FS    FS   Net MB/s    Disk KiB/s      Disk NVRAM  Repl
busy ops/s proc in    out   read    write   busy KiB/s  KB/s
---- ----- ---- ----- ----- ------- ------- ---- ------ ------
  0%     0   0%     0     0       0       0   0%      0      0

The system should show as idle.

# filesys clean status
Cleaning finished at 2012/04/10 06:01:30

Cleaning should indicate finished.

d. Disable the file system on the source system, if systems are running DDOS version 4.7 and below: This is an optional step for DDOS version 4.8 and above.

# filesys disable

e. Break the replication on the source system:

# replication break rctx://<context number>

Example:

# replication break rctx://2

All contexts can be broken with the following command:

# replication break context all

f. Disable the file system on the destination system, if systems are running DDOS version 4.7 and below: This is an optional step for DDOS version 4.8 and above.

# filesys disable

g. Break related contexts on the destination system.

If the destination system is the sole replication partner, follow the same procedure in the previous step (step 2).

If the destination system is configured with contexts involving other systems, break only the shared contexts using following command:

# replication break rctx://<context number>

# filesys enable

i. Re-create replication contexts on the source and destination systems. Use the previously saved configuration information to rebuild the deleted replication contexts:

# replication add source dir://<source directory> destination dir://<destination directory>

Tip: Copy the command from the source clipboard and paste it in the destination clipboard as same command is used on both source and destination systems.

 # replication resync rctx://<context number>

k. Confirm that replication resynchronization is working.

Here is a list of commands that show the status of the replication resynchronization:

# replication watch rctx://<context number>

# replication show performance [all | rctx://<context number>]

# iostat 2 (CTRL+C to quit)

If the source directory is empty, the resynchronization occurs quickly, and may not be represented in the output. In addition, if no data is being written to the source, directory throughput numbers will be low (or non-existent).

If problems persist after performing the steps in this article, Create a service request and Upload a Support Bundle (SUB) Data Domain: How to collect and upload a support bundle (SUB) from a Data Domain Restorer (DDR)

• Data Domain - Replication Resync Explanation
• Data Domain - Cannot Recover from NVRAM: Recently Written Data May be Lost