Data Domain: MTree Replication Fails to Synchronize After Upgrade to DDOS 8.7 or Later
Summary: After upgrading the Data Domain Operating System (DDOS) from a pre-8.7 release to DDOS 8.7 or later, MTree replication contexts may fail to synchronize. This failure results in continuous restarting due to changes in SSL handling. ...
Symptoms
Following an upgrade to DDOS 8.7.x, 8.8.x where MTree replication is configured with over-the-wire (OTW) encryption enabled, the following symptoms may be observed:
- 1. Replication contexts stuck restarting -
replicationshow output shows contexts repeatedly picking the same base snapshot and restarting:
repl ctx4 dest picked base snapid 32901 Restart
- 2. RPC errors in the
ddfs.infologs for the source DD - Repeated "RPC: Unable to send" and "RPC: Unable to receive" errors:
lrepl_send_segs_async_rpc: rpc error 3 (RPC:Unable to send)
do_lrepl_send_refs_async_rpc: rpc error 3 (RPC: Unable to send)
mrepl_send_file: failed to communicate with destination, error code 3
- 3. "
unexpected eof while reading" errors in theddfs.infologs for the destination DD - A high volume of OpenSSLunexpected eof while readingerrors:
error: 0: SSL routines:: unexpected eof while reading
grep -h "unexpected eof while reading" ddr/var/log/debug/ddfs.info* | wc -l
"unexpected eof while reading" in the ddfs.info log
Cause
OpenSSL 1.0.2 (pre-8.7): A peer TCP socket close or RST (for example, by an intermediate firewall or NAT timeout) before a TLS close_notify was treated as a clean EOF (SSL_read() returned SSL_ERROR_SYSCALL with errno==0). Replication recovered gracefully.
OpenSSL 3.x (8.7+): The same condition now returns SSL_ERROR_SSL with reason error:0A000126:SSL routines::unexpected eof while reading. The replication RPC layer (dd_rpc_ssl.c) interprets this as a fatal send or receive error (RPC_CANTSEND / RPC_CANTRECV), causing mrepl_send_file to fail and the replication context to restart in a loop.
This issue is more likely in environments with:
- Constrained or high-latency WAN links
- Replication throttling
- Firewalls or NAT devices that terminate idle or long-lived TCP sessions
Resolution
Workaround:
Disable encryption for the affected replication context.
If not using the user interface (UI), run the following command on both the source and destination systems:
replication modify <destination> encryption disabled
Permanent Fix
This issue is resolved in DDOS 8.9 and later.
The fix sets the SSL_OP_IGNORE_UNEXPECTED_EOF option on the replication SSL context, restoring the OpenSSL 1.x behavior so that an unexpected TCP EOF is treated as a clean SSL shutdown rather than a fatal error.
Action:
Upgrade to DDOS 8.9 or later to permanently resolve the issue.