Root2Root migration in a Av /DD setup

Question

Hi all, I'm testing a Root2Root replication in an Avamar / Data Domain setup and it doesn't seem that the backpus that reside on the Data Domain is being replicated. This is a fresh install of Avamar and just used for testing Source site: Avamar v. 7.1.1-145 DDOS v. 5.5.0.4 Dest site: Avamar v. 7.1.2-21 DDOS v. 5.5.0.8 This is the command that I'm using and I can see that everything that resides on Avamar is being replicated without problems nohup avrepl --operation=replicate --[replscript]dstaddr=lpa01.emcsw.loc --[replscript]dstid=root --dstpassword=xxxxxx --[avtar]id=root --[replscript]fullcopy=true --ap=xxxxxx --send-adhoc-request=false --max-streams=8 --[replscript]timeout=0 This is what I see in the nohup.out log avtar Error <0000>: Replication failed - id: 2, unexpected exception caught: ddr_replthread:2:replicateSlice: ddp_filecopy_status failed [25068] [140419191834368] Tue Sep 15 09:45:44 2015 ddp_filecopy_status() failed, start_offset[0], length[20971520], Err: 5009-filecopy operation failed (nfs:I/O error) [25068] [140419187623680] Tue Sep 15 09:45:43 2015 ddp_filecopy_status() failed, start_offset[0], length[20971520], Err: 5009-filecopy operation failed (nfs:I/O error) [25068] [140419188676352] Tue Sep 15 09:45:43 2015 ddp_filecopy_status() failed, start_offset[0], length[20971520], Err: 5009-filecopy operation failed (nfs:I/O error) [25068] [140419189729024] Tue Sep 15 09:45:43 2015 ddp_filecopy_status() failed, start_offset[0], length[20971520], Err: 5009-filecopy operation failed (nfs:I/O error) [25068] [140419192887040] Tue Sep 15 09:45:43 2015 ddp_filecopy_status() failed, start_offset[0], length[20971520], Err: 5009-filecopy operation failed (nfs:I/O error) [25068] [140419190781696] Tue Sep 15 09:45:42 2015 ddp_filecopy_status() failed, start_offset[0], length[20971520], Err: 5009-filecopy operation failed (nfs:I/O error) [25068] [140419187623680] Tue Sep 15 09:44:22 2015 ddp_filecopy_stop() failed, Err: 5004-nfs filecopy stop failed (nfs: No such file or directory) [25068] [140419192887040] Tue Sep 15 09:44:22 2015 ddp_filecopy_stop() failed, Err: 5004-nfs filecopy stop failed (nfs: No such file or directory) [25068] [140419190781696] Tue Sep 15 09:44:22 2015 ddp_filecopy_stop() failed, Err: 5004-nfs filecopy stop failed (nfs: No such file or directory) [25068] [140419188676352] Tue Sep 15 09:44:22 2015 ddp_filecopy_stop() failed, Err: 5004-nfs filecopy stop failed (nfs: No such file or directory) avtar Error <18797>: Replication failed - The replicator returned an error. avtar Error <18773>: MReplication failed -- backup: 1D0ECA656E313A2, from dd02.emcsw.loc(1):avamar-1441900151 to dd04.emcsw.loc(1):avamar-1442217989, DDR result code: 4912, desc: No error description available. avtar Warning <18125>: Calling DDR_REPLICATE returned result code:5009 message:I/O error avtar Error <10612>: Replication failed -- could not replicate file /cur/942c32b70715f427d2988ae8e86cc2c41e14496a/1D0ECA656E313A2/ddr_files.xml to /STAGING/942c32b70715f427d2988ae8e86cc2c41e14496a/BACKUP-5984104540515F95695C530B1C6CD980F3F7B51C-1D0ECA656E313A2/ddr_files.xml, LSU: avamar-1442217989, DDR result code: 5009, desc: I/O error avtar FATAL <12527>: Replication failed avtar FATAL <40009>: DDR encountered errors. They way I read the output from the log is that it doesn't seem like src site can send data to dst site ddboost device in Data Domain. The src site is of course created with a DD Boost device and some backup has been running to DD and some directly to Avamar. The dst site is created with a DD boost device but is an 'empty'. I'm pretty sure that you should be able to do a Root2Root in a combined enviroment but do the DDboost device need to be the same name on both src & dst? Do you need to rename the dst DD system to src DD name? Any feedback is much appreciated

broeste1 · Accepted Answer

Hi All,

I solve the issue. I'm afraid that is was as simple as a missing Replicator license on the dst Data Domain. Should have checked that as one of the first things but you learn by your mistakes :-)

After I added the Replicator license the replication starts with both data on Avamar and Data Domain.

I still see an issue with the MCS needs to be started on the dst site before any replication can start. Accordingly to the guide the MCS should be stopped but then I get the above error message

umichklewis · Answer

The DD system names do not need to match. Can you confirm the Avamar nodes on the destination side are intergrated with the destination side DD? You should be able to see the DD array on the destination side in the Avamar GUI on the destination side. I would check to make sure the DDBoost passwords are set on each device, and that a network route exists between the two DD arrays (you can check from the CLI of each DD).

Let us know if that helps!

broeste1 · Answer

Nope, didn't help. Everything is running on the same subnet in my test enviroment and no Firewall in between src and dst.

The dst DD system is integrated with the dst Avamar and everything looks fine in both GUI and CLI.

I'm following this guide for the migration EMC® Avamar® 7.0 System Migration Using Root-to-Root Replication. Accordingly to this guide the MCS has to be stopped on the src system. Not sure if this is related but when I stop the MCS on dst and run the avrepl command i get the following in the

2015/09/15-21:10:02.44024 [avagent] Debug output redirected

2015-09-15 23:10:02 avagent Info <5008>: Logging to /usr/local/avamar/var/client/NAH-1442351402392-1008-replicator-avagent.log

2015/09/15-21:10:02.44035 [avagent] Config: VARDIR=/usr/local/avamar/var, HOMEDIR=/root

2015/09/15-21:10:02.44038 [avagent] Looking for flag file "/usr/local/avamar/var/avamar.cmd"

2015/09/15-21:10:02.44040 [avagent] Looking for flag file "/usr/local/avamar/var/avagent.cmd"

2015-09-15 23:10:02 avagent Info <19803>: Ignoring the --service flag.

2015-09-15 23:10:02 avagent Info <5702>: Command Line: /usr/local/avamar/bin/avagent.bin --gencerts="true" --mcsaddr="lpa01.emcsw.loc" --mcsport="28001" --conntimeout="120" --logfile="/usr/local/avamar/var/client/NAH-1442351402392-1008-replicator-avagent.log" --debug="false"

2015-09-15 23:10:02 avagent Info <5703>: Parsed Flags: /usr/local/avamar/bin/avagent.bin --gencerts=true --mcsaddr=lpa01.emcsw.loc --mcsport=28001 --conntimeout=120 --logfile=/usr/local/avamar/var/client/NAH-1442351402392-1008-replicator-avagent.log --debug=false

2015-09-15 23:10:02 avagent Info <19807>: Creating certificates in '/usr/local/avamar/etc/172.30.17.116'

2015-09-15 23:10:02 avagent Info <18918>: Registration: Processing secure registration with the MCS.

2015-09-15 23:10:02 avagent Info <18921>: Registration: Requesting root CA from the MCS.

2015-09-15 23:10:02 avagent Error <5365>: Cannot connect to 172.30.17.116:28001.

2015-09-15 23:10:02 avagent Info <5059>: unable to connect, sleep(60) then retrying

2015-09-15 23:11:02 avagent Error <5365>: Cannot connect to 172.30.17.116:28001.

2015-09-15 23:11:02 avagent Info <5059>: unable to connect, sleep(60) then retrying

2015-09-15 23:12:02 avagent Error <5365>: Cannot connect to 172.30.17.116:28001.

2015-09-15 23:12:02 avagent Info <5059>: unable to connect, sleep(60) then retrying

2015-09-15 23:13:02 avagent Error <5365>: Cannot connect to 172.30.17.116:28001.

2015-09-15 23:13:02 avagent Info <5059>: unable to connect, sleep(60) then retrying

2015-09-15 23:14:02 avagent Error <5365>: Cannot connect to 172.30.17.116:28001.

2015-09-15 23:14:02 avagent Info <5059>: unable to connect, sleep(60) then retrying

Shouldn't the MCS be stopped on the dst system?

Avamar

Was this post helpful?