NetWorker: Clone-Controlled Replication (CCR) fails with file copy operation failed (5009 I/O error).

NetWorker Clone Control Replication (CCR) is failing with filecopy errors. Data Domain error 5009 appears,

The following errors are observed in the NetWorker server's daemon.raw:

• Linux: /nsr/logs/daemon.raw
• Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw
• NetWorker: How to use nsr_render_log to render .raw log files

DDP LOG: [4A3E:2AAAAC001900] ddp_filecopy_status() failed, start_offset[0], length[10485760], Err: 5009-filecopy operation failed (nfs: I/O error)
nsrclone: Failed to obtain filecopy status for ssid=xxxxxxx, cloned 0 bytes out of 4: Retrieving the status of a file copy operation failed ([5009] I/O error).
DDP LOG: [4A3E:2AAAAC001900] ddp_filecopy_stop() failed, Err: 5009-nfs filecopy stop failed (nfs: I/O error)
nw_ddcl_filecopy_stop failed for ssid=xxxxxxxx: Stopping a file copy operation failed ([5009] I/O error).

The Data Domain's ddfs.info file shows the following error.

ddfs[31166]: ERROR: MSG-NFS-00005: Failed to do a filecopy, err = 5029

• NetWorker: How To Collect ddfs.info log from Data Domain

Port communication between NetWorker server and the source/target requires the following ports be open:

• 111 UDP/TCP
• 2049 TCP
• 2051 TCP
• 2052 TCP

This is also needed between the source and destination Data Domain servers.

On the Data Domain, use net rpcinfo -p <hostname> to confirm connectivity.

Communication must be open between the networker clients used for CCR and the source/target Data Domains.

It was found that the firewall was blocking the ports 111, 2049, 2051, and 2052 between the source and destination Data Domains. Once those ports were opened, the issue was resolved.

Example where ports are working:

ab@dd01## net rpcinfo -p dd02
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 901 status
100024 1 tcp 904 status
100011 1 udp 730 rquotad
100011 2 udp 730 rquotad
100011 1 tcp 733 rquotad
100011 2 tcp 733 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100021 1 udp 40985 nlockmgr
100021 3 udp 40985 nlockmgr
100021 4 udp 40985 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 tcp 34098 nlockmgr
100021 3 tcp 34098 nlockmgr
100021 4 tcp 34098 nlockmgr
100005 1 udp 882 mountd
100005 1 tcp 885 mountd
100005 2 udp 882 mountd
100005 2 tcp 885 mountd
100005 3 udp 882 mountd
100005 3 tcp 885 mountd
390436 1 tcp 9127
390435 1 tcp 8633
390113 1 tcp 7937
390103 2 tcp 8569
390109 2 tcp 8569
390110 1 tcp 8569
390120 1 tcp 8569
390109 2 udp 8071
390114 1 tcp 9211
390105 5 tcp 8776
390105 6 tcp 8776
390107 5 tcp 9565
390107 6 tcp 9565
390430 1 tcp 9719
390111 1 tcp 7986
390429 301 tcp 9201
390429 101 tcp 9598
390429 201 tcp 9342
390438 1 tcp 8098
390439 1 tcp 8640
390433 1 tcp 9322
390437 1 tcp 7946
390437 1 udp 9762
390104 2505 tcp 9398
390104 2805 tcp 8610
390104 1105 tcp 9380
390104 405 tcp 9896
390104 3905 tcp 8628
390104 3005 tcp 7952
390104 3105 tcp 9165
390104 3505 tcp 8072
390104 3205 tcp 8508
390104 205 tcp 8649
390104 2305 tcp 9832
390104 2405 tcp 9703
390104 5805 tcp 9893
390104 5605 tcp 8903
390104 1305 tcp 8370
390104 6405 tcp 8142
390104 5905 tcp 9811
390104 5105 tcp 8666
390104 3405 tcp 9849
390104 4205 tcp 8289
390104 2005 tcp 9068
390104 6005 tcp 8129
390104 7005 tcp 9910
390104 4505 tcp 9453
390104 3705 tcp 9062
390104 6105 tcp 8800
390104 4405 tcp 8207
390104 6505 tcp 9037
390104 5005 tcp 9019
390104 7105 tcp 8499
390104 7905 tcp 8938
390104 6305 tcp 9471
390104 7805 tcp 9644
390104 7305 tcp 8818
390104 8105 tcp 9635
390104 8205 tcp 8542
390104 8305 tcp 8237
390104 305 tcp 9113
390104 505 tcp 9309
390104 1205 tcp 8299
390104 2205 tcp 8522
390104 4605 tcp 9511
390104 1005 tcp 8756
390104 7705 tcp 9084
390104 905 tcp 9109
390104 605 tcp 9317
390104 1705 tcp 8017
390104 1405 tcp 8451
390104 2605 tcp 8469
390104 1605 tcp 8322
390104 2105 tcp 8808
390104 1805 tcp 9664
390104 1905 tcp 8925
390104 2705 tcp 8035
390104 3305 tcp 8908
390104 4905 tcp 9854
390104 3805 tcp 9497
390104 5405 tcp 9420
390104 2905 tcp 9247
390104 5305 tcp 8749
390104 4705 tcp 9183
390104 6705 tcp 7974
390104 4305 tcp 9522
390104 6605 tcp 8632
390104 6805 tcp 8211
390104 7605 tcp 8095
390104 6905 tcp 8530
390104 4005 tcp 9029
390104 5705 tcp 8091
390104 7405 tcp 8690
390104 8405 tcp 8650
390104 6205 tcp 8762
390104 8005 tcp 9355
390104 1505 tcp 7967
390104 7505 tcp 8204
390104 4805 tcp 9533
390104 805 tcp 9717
390104 705 tcp 8422
390104 3605 tcp 8271
390104 4105 tcp 9175
390104 5505 tcp 9890
390104 5205 tcp 9476

Example where ports are blocked:

ab@dd01## net rpcinfo -p dd03
rpcinfo: can't contact portmapper: RPC: Remote system error - Connection timed out

**** An error occurred while running an internal command.

ab@dd01## net ping dd03 count 3
PING dd14022 (172.xx.xx.xx) 56(84) bytes of data.
64 bytes from dd14022 (172.29.93.71): icmp_seq=0 ttl=59 time=1.15 ms
64 bytes from dd14022 (172.29.93.71): icmp_seq=1 ttl=59 time=1.50 ms
64 bytes from dd14022 (172.29.93.71): icmp_seq=2 ttl=59 time=1.11 ms

NetWorker: How To Troubleshoot Network Related Issues