UNSOLVED

Ther_Max

updated

12 years ago

T

Ther_Max

8 Posts

0

790

December 8th, 2014 08:00

Networker server slowdown (8.1.2)?

Hi,

I saw a few threads on this but they weren't answered - I was wondering if anyone else is having this issue: we recently built a new Networker server 8.1.1.6 then upgraded it to 8.1.2 (uninstalled and installed binaries).  It's running on AIX.  Backups were running fast and quick in the beginning but as we added more clients we noticed that the server performance would slowly degrade over time.  Backups would take longer to run as time went on.  When we reboot the server the backup times decrease dramatically.  Almost as if it clears some kind of memory leak issue.  Then it would repeat and slowly degrade again until we reboot the server.

It's gotten so bad that now we are looking at weekly reboots.  A backup that would take 2 hours doubles by the end of the week.  We get alot of these errors: Unable to setup direct save with server : busy.  Then we know it's time to reboot.  Have about 850 clients configured and 6 storage nodes.   One thing to note - just restarting the networker services has no affect - we actually have to reboot the server to see any affect.  Interesting to note that we also had this problem on our other Networker server 7.6.3, that's why we decided to build a new server but now seeing the same issue.

Thanks for any input.

  • ble1

    6 Operator

    •

    14354 Posts

    •

    56186 Points

    315

    0

    Posted December 9th, 2014 04:00

    If there is memory leak with application, most likely you do not need to reboot server, but application only. Given that you have to reboot the box indicates an issue somewhere else potentially.  Next, you should be able to see memory leak via topas - try to isolate the issue.  Which AIX OS are you running (oslevel -s)?

  • Ther_Max

    8 Posts

    315

    0

    Posted December 9th, 2014 11:00

    7100-03.

    That's the weird thing - I don't see any memory issues, minimal paging, but Networker server starts to slow down, jobs fail with 'server busy' error or they take double or triple the time to complete.+

  • CarlosRojas

    1 Rookie

    •

    1693 Posts

    315

    0

    Posted December 10th, 2014 02:00

    Hello,

    Please make sure that you set nsrmmd's to static instead of dynamic.

    Also make sure you don't start too many groups/clients at the same time.

    Do you have DPA in place? Or maybe any other machine that could be issuing mminfo commands?

    Is there anything in the logs that gives you any clue?

    Thank you,

    Carlos

  • Ther_Max

    8 Posts

    315

    0

    Posted December 10th, 2014 08:00

    Hi Carlos,

    Yes, Dynamic nsrmmds are checked for all of our storage nodes.  It was set by default from the install - should it be set to static for performance reasons? I'll remove that option and see if it improves performance.

    We don't have many groups that start at the same time, majority of them are staggered to start every 5-10 minutes.

    No DPA running.  Nothing in logs.

    Thanks,

    Max

  • ble1

    6 Operator

    •

    14354 Posts

    •

    56186 Points

    315

    0

    Posted December 12th, 2014 03:00

    I run 7100-03-03-1415 and that seems to be ok (at least with 8.0.3.5).  From time to time I will get auth issues, but this is not so often and certainly I haven't seen any slowdown (but then again, different code tree than yours).  I would also be interested to hear more about static versus dynamic nsrmmd as dynamic is supposed to bring more gains, but this is the second time I see recommendation to switch to static with 8.1.x at least so I wonder what is the best practice there. Currently I run dynamic within all my datazones.