Hi Mark,

As you know the IRCCD default ports configured during the installation is 6728 (Control Port) and 6729 ( Data Port), So do you mean that there is another application uses those ports and you changed the ports of IRCCD for that conflict ?

If not , you can change back the default ports to the IRCCD by changing the replication manager port settings by doing the following steps:

1. From the command line,stop the rmagentps service.

net stop rmagentps

2. From the %Program Files%\Legato\nsr\rmagentps\client\bin directory, run the irccd command to change the port settings:

irccd –p  6728  -P 6729

3. From the  %ProgramFiles%\Legato\nsr\res directory, edit the control port and data port values in rmps.res :

type: RM Service Ports;

Control Port: 6728;

Data Port: 6729;

4. Start the rmagentps service:

net start rmagentps

After that you have to validate that there is an exception for the replication manager in the Windows Firewall:

In Windows Server 2008 firewall, click Allow a program through Windows Firewall.

In the Programs and Services list, verify that  EMC Replication Manager Client for RMAgentPS appears and is selected:

• If the checkbox is not selected, select it.

• If there is no entry for EMC Replication Manager Client for RMAgentPS, manually add it:

a. Click  Add Program .

b. Click  Browse.

c. Type  C:\Program Files\Legato\nsr\rmagentps\client\bin\irccd.exe.

If NMM was installed on an another drive letter other than C:\, specify the correct path to irccd.exe as needed.

d. Click  OK.

Hope this helps you to solve your problem.

Waiting your updates.

Thanks,

Ahmed Bahaa

Hi Ahmed

Just to give you a bit more information; there are 4 hosts in our Exchange 2010 DAG.

Checking connections on one of these shows 1000's of ports open between them. These are randomly generated too.

We cannot lock down exchange from using certain ports ie for NMM.

NMM does not seem to reserve the ports that we configure for it either.

So occassionally when our Legato backup kicks off it may find the ports already in use and therfore fail.

I then manually change the Replication Manager ports using the procedure you provided above, which may work for a while until that port is taken by another service again. So i then change the ports again and again and again.... It justs seems to me that NMM is a bit "unfinished"

Would be nice if the NMM app would reserve the ports so nothing else may use them or be able to use any available ports instead of hard coding them.

Thanks

Mark

Hi Mark,

NMM is using the IRCCD ports that has been specified during installation of when you change it as per the procedures above, So if you specify 6728 and 6729 , it will use these ports only , not a range of ports or any available ports. But what i would like to ask about is why there is another services using those ports ?

It is not a solution to change the IRCCD ports periodically to allow it to work. I have a lot of customers using Exchange 2010 DAG with 4 servers as well and i didnt see this issue before for other services in this environment uses the IRCCD ports. Are you using Windows firewall for communication between the servers ? and what are the services that uses the IRCCD ports ?

Thanks,

Ahmed Bahaa

Hi Mark,

I'm not aware of Exchange using any of these 2 ports (6728 and 6729), and bear in mind that these were chosen specifically to work with all Microsoft Applications, so I would suggest that instead of complaining about NMM, which has clearly defined the ports range (in fact RM, which uses 6728 and 6729), try to find out which application is using these ports and try to fix that instead, or else just change the RM ports as Ahmed was saying.

In the other hand that error message is pretty common and it's not related to any other application using those ports, I think that this is most likely caused by a previous backup failure, and some lock being left on the /nsr/tmp folder.

When you change the ports the lock is still applied for those ports, so changing it will be a temporary workaround, and backup completion should also release the lock for ports 6728 and 6729, or if it doesn't and you revert back to the same ports then you will be getting again the same error message so, in closing this is my advice:

1.- Stop NetWorker services including RM ones.

2.- Delete /nsr/tmp folder

3.- Start services again.

4.- Check the ports in use, in case there is any other application using 6728 or 6729

5.- Start a new backup, and that should work.

Thank you.

Carlos.

Hi Mark,

Yes indeed. Cleaning out the /nsr/tmp will "release" (in fact will remove) any lock files. Besides the temporary files created during the replica creation etc. are placed within that folder.

The tmp folder contains any temporary file for a backup process locally on each client, and in the case of NMM it also places the RM and PS lock and temporary files, besides the snapshot pointers etc.

Don't be pessimistic, backup should work fine by default

Thank you.

Carlos.

Ahmed / Carlos

Thanks for the replies.

Answering Ahmed, we do not use any host based firewalls, all are turned off.

I realise changing the ports periodically is not a solution, hence starting this thread. It just got the backups working again.

I think Carlos is probably right in that a previous failed backup has these ports locked. As for clearing these locks i'm not sure stopping the agent and clearing /nsr/tmp will do this - I have completely uninstalled/re-installed Networker client and NMM in the past only for  the backup to fail again. The fix then was to reboot the Exchange node, which in the case of our old CCR backups, of the passive node only was not a problem, but now with DAG we have to arrange this out of hours. Changing the IRCCD ports resolves the issue without scheduling downtime.

Currently the backups have been working for the past week, i know i'm being pessimistic, but when one of the nodes fails i will go through Carlos's reccommendations above and report back.

Thanks for your answers guys.

Mark

Hi Carlos

Just had a node fail, I knew it would…  The failure is the same that you helped Alex.Seryogin with. Wish I knew what caused it as over 1.8TB had backed up then fails at deporting the snapshot!! I have added the end on my NMM log to the end of this.

I went through your suggestion stopping all Networker and RM services, deleting /nsr/tmp then restarting. I also deleted all the shadows to re-run the backup.

Failed again complaining of the IRCCD ports already in-use, probably because of the previous failure, however the ports were not released with the procedure above.

I have manually changed the ports back to the default 6728,6729 and restarted the backup, now running again.

Do you have any more info for the backup failure, I’ve been assured that there has been no cluster failover or application role change with Exchange?

Thanks for any info you have on the failure.

Mark

End of NMM log:

03/04/2012 14:08:40  1 0 0 31752 NMM .. Delete the temporary snapshot as part of deport

03/04/2012 14:09:10  5 0 0 31752 PS: (CPSImportService::GetOperationStatus) ERR: Error for operation 2

03/04/2012 14:09:10  5 0 0 31752 NMM .. Error during deport of snapshot: PS .. Error completing snapshot deport.

03/04/2012 14:09:10  5 0 0 31752 NMM .. Registration of snapshot set failed -- PS .. Error completing snapshot deport 

03/04/2012 14:09:10  1 0 0 31752 NMM .. Delete the temporary snapshot as part of deport

03/04/2012 14:09:10  5 0 0 31752 PS: (CPSImportService::GetOperationStatus) ERR: Error for operation 2

03/04/2012 14:09:13  5 0 0 31752 NMM .. error caught calling RM to commit the replica. 

03/04/2012 14:09:13  5 0 0 31752 NMM.. Exception caught: Failed to commit replica.

03/04/2012 14:09:13  5 0 0 31752 NMM .. cluster failover or application role change after a replica created may have caused snapshot creation failure, try to restart the backup. 

03/04/2012 14:09:13  5 0 0 31752 NMM backup failed to complete successfully.

03/04/2012 14:09:13  0 0 0 31752 nsrsnap_vss_save: createInstanceBackup() failed

03/04/2012 14:09:13  2 0 0 31752 nsrsnap_vss_save: APPLICATIONS:\Microsoft Exchange 2010\D1DB04: failed 

03/04/2012 14:09:14  2 0 0 31752 nsrsnap_vss_save: APPLICATIONS:\Microsoft Exchange 2010\D1DB07: failed 

03/04/2012 14:09:14  2 0 0 31752 nsrsnap_vss_save: APPLICATIONS:\Microsoft Exchange 2010\D1DB09: failed 

03/04/2012 14:09:14  2 0 0 31752 nsrsnap_vss_save: APPLICATIONS:\Microsoft Exchange 2010\D1DB15: failed 

03/04/2012 14:09:14  2 0 0 31752 nsrsnap_vss_save: APPLICATIONS:\Microsoft Exchange 2010\D1DB17: failed 

Hi Mark,

This is interesting, however I would like to see the entire output of the nmmrendered) so I can see at what point this is failing.

Please check this:

1. Ensure the rmps.res file exists and is defined correctly. The following is an example of a default rmps.res file:

   type: RM Service Ports;
   Control Port: 6728;
   Data Port: 6729;

2. Stop and restart the 'Replication Manager Client for RMAgentPS' service

Can you please check the content of this file?:

C:\Program Files\Legato\nsr\res\servers

If there is wrong information here, or there is any name resolution issue then that could lead to a backup failure in the roll over process. Please comment any entry in there, stop NW services includinr RM and PS, delete /nsr/tmp and restart NW services again.

Also I would suggest you to install NMM Config Checker and run it in the 2 nodes. NMM Config Checker also install NMM Log Viewer which basically will render for you nmm.raw, PS logs and RM logs, so it will be easier for you to check, as I think we need to look now into the PS logs:

CPSImportService::GetOperationStatus

It is failing in operation 2, so I suppose that you are right and it's failing in the export (roll over) of the snapshot, that's why I would like to see the nmm.raw to check the snapshot policy you have set, the operations performed and at what point it fails, anyway I would suggest you to check Application and System event logs of the node failed ate the time of the bakcup:

03/04/2012 14:09:10

You should find some further information in there. Actually take a look at the daemon.raw just in case, and ensure that you have the TCP/IP best practices in place, in Exchange nodes and also in NetWorker server:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize=256000

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\GlobalMaxTcpWindowSize=16777216

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\KeepAliveInterval=1000

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\KeepAliveTime=600000

On Solaris:

#ndd -get /dev/tcp tcp_keepalive_interval

7200000

#ndd -set /dev/tcp tcp_keepalive_interval 3360000

And just in case please ensure again that since the last full backup there has been no changes in the status of the DB's (from passive to active or the other way around).

Thank you.

Carlos.

Thank you Mark,

I'll wait for your update.

Thank you.

Carlos.