Data Domain: MTREE Replication Shows Incorrect Bytes_remaining During Initialization
Summary: MTREE replication shows incorrect bytes_remaining during initialization.
Symptoms
MTREE Replication shows incorrect bytes during the initialization process for old MTREEs which were never used during replication.
The following MTREE has been used [# mtree list]: /data/col1/support 5965.4 RW
Stats show 92 TB of remaining data after creating a new MTREE replication using the above steps:
# replication show detailed-stats rctx://1
CTX:1 Destination: mtree://dd-dr.support.com/data/col1/support Network bytes sent to destination: 2,287,577,928 Pre-compressed bytes written to source: 92,046,090,413,266 Pre-compressed bytes sent to destination: 2,860,770,559 Bytes after synthetic optimization: 2,860,770,559 Bytes after filtering by destination: 2,253,845,068 Bytes after low bandwidth optimization: 2,253,845,068 Bytes after local compression: 2,258,791,288 Pre-compressed bytes remaining: 92,043,229,642,707 <---- Compression ratio: 1.3 Sync'ed-as-of time: (initializing)
While underlying registry keys show more accurate bytes:
repl.001.src_host = dd-prod.support.com repl.001.src_path = /data/col1/support repl.001.dst_host = dd-dr.support.com repl.001.dst_path = /data/col1/support repl.001.progress.task_name = initializing 3/3 repl.001.progress.init_vbytes = 5991145719813 <---
From ddfs.info:
Initial virtual bytes estimation looks OK:
09/21 10:58:45.157 (tid 0x2b0e208796e0): repl ctx 1: mrepl_create_init_snap creating initial snapshot. 09/21 10:58:48.068 (tid 0x2b0e208796e0): repl ctx 1: estimating init vbytes as 5991145719813 09/21 10:58:48.995 (tid 0x2b0e208796e0): repl ctx 1: Using init_resync_snapid 1433317767:1886. Total snap count = 2, curent snap cnt = 0.
After couple of minutes, the normal course of mrepl_sends starts appearing:
09/21 11:19:04.136 (tid 0x327e760): repl ctx 1:3: mrepl_send_file: file 30:0:ac66:0:a30e9f46:556eb187:1d74 took 218 sec, net throughput 297 KB/s, virtual throughput 290 KB/s, refs_sent 7575, refs_features_sent 0, segs_sent 0, segs_features_sent 7575, size 63416832, vbytes 63416832, nbytes 64858812 (recid 361) (ch-xor 0x98f3c7e9) (hole_seen FALSE) 09/21 11:19:59.031 (tid 0x2b0e2083cde0): repl ctx 1:16: mrepl_send_file: file 30:0:ac68:0:fccc10a6:556eb187:1d74 took 237 sec, net throughput 286 KB/s, virtual throughput 280 KB/s, refs_sent 7908, refs_features_sent 0, segs_sent 0, segs_features_sent 7908, size 66419200, vbytes 66419200, nbytes 67938672 (recid 363) (ch-xor 0x462a1211) (hole_seen FALSE)
The next occurrence of vbytes_remain shows a different size:
09/21 11:28:02.246 (tid 0x2b0e096ba050): repl ctx 1: mrepl_stats_local: vbytes_remain: 92036838018199
Cause
Resolution
This is how replication is designed to calculate the logical bytes remaining. This mismatch is observed each time the user starts replicating some older MTREE. However, this does not indicate that MTREE replication replicates this of data, it only replicates the current size of the MTREE as indicated by init_vbytes_remain.
Once initialize phase is over, reset vbytes_remain to 0 and also update vbytes_written_and_replicated with the cum_raw_bytes so subsequent backups do not show unrealistic stats, rather reports only the new data which is ingested.