NetWorker Version 7.3.3 Network performance issue we are experiencing
The idea of having parallel threads to backup is to ensure that the server hosting the legato backup has enough bandwidth and memory to handle the parallel threads, in recent months we have been experiencing the following problems with our legato backup software even after adding an extra dedicated storage node.
The Network interface cards have been receiving an increase amount of ¿Receive Errors¿ during the backup time frames, this is on both the main node and the secondary storage node. The networking equipment from point to point has been analysed and no errors have been observed on the interfaces that connect the two storage node nor the path to get to the storage node.
These receive errors are causing RPC time outs from the main legato server and it means that one of our team members are having to restart this process when it fails on a weekly basis. We require some direction on how to tune the parallel threads or a way to debug this to rule out hardware.
Can anyone provide any details as to whether this is a known problem with our current legato version and how we might go about making a fix.
We are backing up some 200 plus client servers to this NetWorker backup server instance. Parellism on the backup server is set to 64 We have 3 LT03 Tape drives each have Target and Max Sessions set to 21 The NIC is set to Current Spreed / Duplex 1000 Full
The error condition we are getting most often is RPC errors, see below: --- Unsuccessful Save Sets --- * Server:E:\ save: RPC error: RPC send operation failed. A network connection could not be established with the host.
The error condition seems to state the obvious that the problem is NetWork related. We want to establish if there is anything at the backup server or client side that may be causing these errors. And would appreciate any help on how others who have had same, similar problems went about resolving.
Forgot to include in my previous post that the NIC is showing some 34,000 Receive Errors? Whereas on other NetWorker backup server instances the NIC reflects no Receive Errors and no same, similar error conditions returned.
We have an associated second storage node that is dedicated to only backups and on this server the interface is clocking up recieve errrors also but not at such a high rate as the main backup server. I will look into the TCP offloading, is this done on the server end?
Even if the NIC utilization is at 100% would networker not adjust to stream adequate windowing to not overflood the buffer on the NIC's? Understanding that overutlilization can cause these errors but is there a way to adjust networker to accomodate this?
Usually this sort of errors happens when either you have saturated network link or NIC port or simply misconfiguration on NIC mode or switch port to which your NIC is patched. I assume you already have gone through those... I didn't see what your backup server is, but I also recall having issues on Windows boxes with gigabit connection when TCP offloading was enabled - disabling it brought back performance and errors like were gone.
Is it possible to replace NIC just to see if you get the same thing? At the time when errors occur do you see NIC utilization to hit 100%?
There May be many reason for the RPC error in these type error we should go by the basic. As you explained that you have checked your network at every point .I think there is still some points to check like :-
1) Is the client initiated backup is happening correctly 2) Check the Alias field in the client resource . In this only put the client FQDN 3)Put the server IP in the Server Network Interface in the each client resource 4) You should also do the client service on manual mode and restart them. 5) You can also check that is there any package drop in the network after some time 6)If the RPC error come after some time then there amy also be some issue related to administrator rights and also the network is consistent or not means the network connectivity should remain consistent
No, NW client will push as much as it can. What you can do to balance this is to limit number of concurrent streams via parallelism settings on client, group and/or server.
amediratta
2 Intern
•
2014 Posts
673
0
Posted May 28th, 2008 22:00