nsrjobd memory leak in NW8

I went to NW8 somewhere in September last year. Since then I wanted to write blog with all details I have to fight with since and steps I made, but I kept postponing it due to some ongoing issues. One of the things I noticed was SQL clients would have their nsrexecd die - well, it would stop responding to RPC calls.  I had following situation (always using VDI):

NW7 server + NW7 clients + NMSQL 5.2.2 -> OK

NW8 server + NW7 clients + NMSQL 5.2.2 -> NOK

NW8 server + NW8 clients + NMSQL 5.2.2 -> NOK

NW8 server + NW8.1 clients + NMM 3.0 -> NOK

The frequency of these freezes is irregular and I never really had a chance to spot any specific pattern so I just learned to live with a fact that I have to restart services on client and then all is back to normal for at least next 24 hours or so (ok, sometimes less).

In parallel to that, I have observed that some special jobs that I have, running over savepnpc every 15 minutes, would start to die too with hard time limit (most of the time due to savepnpc not completing post part, but not always). Increasing frequency to 20 minutes helped a bit, but then it was back to its usual nonsense. Real danger would happen when sporadically tmp file would not be removed from /nsr/tmp due to whatever was causing this.  As this looked as random pattern too (with its all possible outcomes within failure), I concluded that most likely same thing is causing it.  And since all started with simple upgrade of server and storage nodes, I isolate my thoughts to backup server to be major issue here.  As things started to look worse with time, it sounded as potential memory leak somewhere. And indeed, it looks as if nsrjobdb is to blame.

During course of several past month I have been monitoring this and all I can say is that either there is something really broken in nsrjobdb in general or on platform where I run backup server (HPUX ia64 11.31). Since I didn't see complains like these before here, I will guess that this might be isolated to ia64 platform for now.  When I look at my current run, I have backup server running since November 22nd (this is when 8.0.2.5 was applied which seems to be worse than 8.0.2.3 in that respect). My current nsrjobdb memory usage is 2.5G   And nsrjobd rarely likes to light on CPU nowadays (restart the server and it comes back to senses for a while).  I thought NW8 was supposed to fix nsrjobd, but in my case is driving me more crazy. 

If I check how log purge runs, I see nothing unusual - NW8 really does rock there. I keep records only for 12 hours so amount of records should be ok (it was ok for NW7 so I would expect NW8 to be very happy there too). However, if I check logs, I see all sorts of crazy messages.  In some random order and without any specific meaning, here are few:

nsrjobd SYSTEM critical User root does not have the privileges required to start the requested job. The required privilege is: Operate NetWorker. The job requested to be started is: savefs.

nsrjobd SYSTEM critical User root does not have the privileges required to start the requested job. The required privilege is: Operate NetWorker. The job requested to be started is: nsrsqlsv.

nsrjobd SYSTEM critical User root does not have the privileges required to start the requested job. The required privilege is: Operate NetWorker. The job requested to be started is: nsrdasv

etc...

Now, above shows savefs, NMM and NMDA affected, but I have seen it for save and NMSAP too.  To add further, I have seen that something when I start save from client or I restart group from CLI you get to see messages like:

Cannot create session with nsrjobd: Failed to register with server : Timed out

Retrying 4 more time(s)

Eventually, this may fail upon one or two retries, but at the end it always runs with success.  If you use GUI, you won't see it as this is masked (you may find entry in log indicating SSNCHNL error for group for example).

One thing I also noticed is that privilege message (Operate NetWorker) comes from physical node in logs while timeout shows virtual name.  Not sure if nsrjobd does not like clusters or this is irrelevant .

I do have ticket with support who opened this with EMC and I do have NW id for it (which at this stage is only indication that it has been pushed to engineer), but no progress at all.

My question is - did anyone who went to NW8 had this issue and did anyone observe nsrjobd being cookie monster when it comes to memory?  I have no doubts that my issue is related to nsrjobd as I have excluded pretty much anything else in all tests done so far, but I wonder how others have been dealing with this and what experience they might have.  Looking forward to see your memory usage

Some notes: I run pretty much same setup since 2006 (with HW refresh in 2009). We have very good experience with HPUX platform and system is tweaked and tuned as it should be (I also checked NW8 hints for performance optimization and we were there long time ago). The privilege message is bogus - let me assure you we have permission list in place and hasn't been changed for last 8 years - you get the same even with *@*.  Finally, even this is NW8, we run our setup with nsrauth disabled so any crazy thing which may occur by using nsrauth is not the case here.