Highlighted
Ross_
1 Copper

DD2500 - failed to communicate with destination, error code 4

We have two DD2500 in an MTree replication pair. Recently the replication stopped working and the only errors we see on them are:

Source DD error: "failed to communicate with destination, error code 4 (RPC: Unable to receive)"

clipboard_image_1.png

Destination DD error: "Error from remote host: failed to communicate with destination, error code 3"

clipboard_image_0.png

The backlog of data is now mounting. I tried recreating the pair and using resync, and for about the first 30 minutes the sync appears to be working and the backlog is reduced. But soon after it stops again with the same errors. It is now currently bouncing between the above errors and "initializing" every few minutes. 

Can anyone help and possibly decrypt these error messages? I can't find anything online about them.

Thanks.

EDIT:

In addition to the above, the messages log is filled with hundreds of the following:

 

Oct 30 06:11:38 dd01 ddfs[24695]: NOTICE: MSG-REPL-00809: repl ctx 1: Replication: replica connected.
Oct 30 06:11:45 dd01 ddfs[24695]: NOTICE: MSG-REPL-00801: repl ctx 1: Replication: replica disconnected
Oct 30 06:11:57 dd01 ddfs[24695]: NOTICE: MSG-REPL-00809: repl ctx 1: Replication: replica connected.
Oct 30 06:12:03 dd01 ddfs[24695]: NOTICE: MSG-REPL-00801: repl ctx 1: Replication: replica disconnected
Oct 30 06:12:14 dd01 ddfs[24695]: NOTICE: MSG-REPL-00809: repl ctx 1: Replication: replica connected.
Oct 30 06:12:20 dd01 ddfs[24695]: NOTICE: MSG-REPL-00801: repl ctx 1: Replication: replica disconnected
Oct 30 06:12:31 dd01 ddfs[24695]: NOTICE: MSG-REPL-00809: repl ctx 1: Replication: replica connected.
Oct 30 06:12:40 dd01 ddfs[24695]: NOTICE: MSG-REPL-00801: repl ctx 1: Replication: replica disconnected

 

EDIT 2:

Seems the disconnects are occurring in a repeating pattern of disconnecting for 10 seconds every minute (10 seconds off, 50 seconds on).

Doesn't appear to be resource related; CPU busy % not getting over 20%, network in about 15MB/s, disk busy % about 75%.

0 Kudos