UNSOLVED

stewartg1

updated

18 years ago

S

stewartg1

22 Posts

0

1079

May 28th, 2008 18:00

NetWorker Version 7.3.3 Network performance issue we are experiencing

The idea of having parallel threads to backup is to ensure that the server hosting the legato backup has enough bandwidth and memory to handle the parallel threads, in recent months we have been experiencing the following problems with our legato backup software even after adding an extra dedicated storage node.

The Network interface cards have been receiving an increase amount of ¿Receive Errors¿ during the backup time frames, this is on both the main node and the secondary storage node. The networking equipment from point to point has been analysed and no errors have been observed on the interfaces that connect the two storage node nor the path to get to the storage node.

These receive errors are causing RPC time outs from the main legato server and it means that one of our team members are having to restart this process when it fails on a weekly basis. We require some direction on how to tune the parallel threads or a way to debug this to rule out hardware.

Can anyone provide any details as to whether this is a known problem with our current legato version and how we might go about making a fix.
  • amediratta

    2 Intern

    •

    2014 Posts

    673

    0

    Posted May 28th, 2008 22:00

    Did you play around with parallelism to try & control this?
  • ble1

    6 Operator

    •

    14354 Posts

    •

    56186 Points

    673

    0

    Posted May 28th, 2008 23:00

    Can you explain a bit more "receive errors" and where and how do you see those?
  • stewartg1

    22 Posts

    673

    0

    Posted May 29th, 2008 15:00

    We are backing up some 200 plus client servers to this NetWorker backup server instance.
    Parellism on the backup server is set to 64
    We have 3 LT03 Tape drives each have Target and Max Sessions set to 21
    The NIC is set to Current Spreed / Duplex 1000 Full

    The error condition we are getting most often is RPC errors, see below:
    --- Unsuccessful Save Sets ---
    * Server:E:\ save: RPC error: RPC send operation failed.
    A network connection could not be established with the host.

    The error condition seems to state the obvious that the problem is NetWork related.
    We want to establish if there is anything at the backup server or client side that may be causing these errors. And would appreciate any help on how others who have had same, similar problems went about resolving.

    Thanks in advance to those who got back.

    Cheers GARY
  • stewartg1

    22 Posts

    673

    0

    Posted May 29th, 2008 15:00

    Forgot to include in my previous post that the NIC is showing some 34,000 Receive Errors?
    Whereas on other NetWorker backup server instances the NIC reflects no Receive Errors and no same, similar error conditions returned.

    Cheers GARY
  • stewartg1

    22 Posts

    673

    0

    Posted May 29th, 2008 17:00

    We have an associated second storage node that is dedicated to only backups and on this server the interface is clocking up recieve errrors also but not at such a high rate as the main backup server. I will look into the TCP offloading, is this done on the server end?

    Even if the NIC utilization is at 100% would networker not adjust to stream adequate windowing to not overflood the buffer on the NIC's? Understanding that overutlilization can cause these errors but is there a way to adjust networker to accomodate this?

    Thanks
  • ble1

    6 Operator

    •

    14354 Posts

    •

    56186 Points

    673

    0

    Posted May 29th, 2008 17:00

    Usually this sort of errors happens when either you have saturated network link or NIC port or simply misconfiguration on NIC mode or switch port to which your NIC is patched. I assume you already have gone through those... I didn't see what your backup server is, but I also recall having issues on Windows boxes with gigabit connection when TCP offloading was enabled - disabling it brought back performance and errors like were gone.

    Is it possible to replace NIC just to see if you get the same thing? At the time when errors occur do you see NIC utilization to hit 100%?
  • Biku1

    42 Posts

    672

    0

    Posted May 29th, 2008 21:00

    Hi Gary,


    There May be many reason for the RPC error in these type error we should go by the basic. As you explained that you have checked your network at every point .I think there is still some points to check like :-

    1) Is the client initiated backup is happening correctly
    2) Check the Alias field in the client resource . In this only put the client FQDN
    3)Put the server IP in the Server Network Interface in the each client resource
    4) You should also do the client service on manual mode and restart them.
    5) You can also check that is there any package drop in the network after some time
    6)If the RPC error come after some time then there amy also be some issue related to administrator rights and also the network is consistent or not means the network connectivity should remain consistent
  • ble1

    6 Operator

    •

    14354 Posts

    •

    56186 Points

    673

    0

    Posted May 30th, 2008 04:00

    No, NW client will push as much as it can. What you can do to balance this is to limit number of concurrent streams via parallelism settings on client, group and/or server.