Networker not starting after 7.4 upgrade

Question

I have had several failed attempts now to upgrade to 7.4.4.3 running on AIX.
nsrd appears to just hang when Networker is started following the upgrade.
Following various sessions with support, everything apart from nsrdb has been eliminated as being the cause. A vanilla nsrdb comes up just fine; our 7.3.4 nsrdb has the problem.
I ran nsrd in debug mode and established that it's stuck waiting for nsrmmdbd. If I start nsrmmdbd manually then nsrd 'wakes up' and Networker continues to come up as normal (albeit with nsrmmdbd outside of nsrd's control).
Whilst nsrd is stuck in it's loop, the only other process up is nsrexecd. Normally I'd expect nsrlmc to have kicked in by then, though my experience is only with 7.3.4 or earlier.
Has anyone seen anything like this ? Does anyone know by what mechanism nsrd is meant to fork off nsrmmdbd ?
Networker starts again just fine after reverting to 7.3.4

DavidHampson · Answer

JasonYou can delete:/nsr/res/jobsdbSee if that has an effect/nsr/res/nsrdb should not be deleted as that contains your configuration

Jason20 · Answer

Removing the jobsdb made no difference. We (re)moved everything and the presence or absense of our nsrdb is the trigger for Networker not starting properly.

DavidHampson · Answer

Try running nsrim -X and then starting Networker.

Jason20 · Answer

The problem will occur in the absense of any mm/ or index/ directories.A clean install of 7.4.4 with just our nsrdb dropped in will hang. I don't believe nsrim touches nsrdb ?

DavidHampson · Answer

All I can suggest you do is to do a DR from backups

ble1 · Answer

This was known issue for early 7.4.x and there is also regression in SP3 for certain platforms. However you said you use 7.4SP4.3 - it is strange support didn't give you SP4.5, but perhaps they went for first release which fixed the problem you originally received this fix for.

Not sure if it will help, but try following as test:
- stop NW
- start manually from CLI as root only /full_path/nsrexecd
- wait 2 minutes
- start manually from CLI as root only /full_path/nsrd

DavidHampson · Answer

Have you thought about trying upgrading to 7.5.1?

Jason20 · Answer

Thank you for the pointers. I've had a call open for just about a month now and this is the first time anyone has alluded to this being a known issue.
The exact version I have is 7.4.4.3, build 643
I'd already effectively tried what you suggested in order to run up nsrd with -D5, albeit with less than a two minute break after nsrexed was started.
It manages a
lgto_auth for `nsrmmdbd' succeeded
and then makes numerous calls into
Calling clnttcp_create function
punctuated with
create client handle for loncmxs1999, 390107, 6, NULL failed, sleeping...
connect to mmdb prog 390107 vers 6 on ` ' failed: Program not registered
lgto_mmdb_auth failed: Program not registered
and then sits forever (at least 1 hour plus I've left it) in endless
Calling clnttcp_create function
Presumably waiting on mmdb, hence me trying a manual start of it, which did get things moving.

Jason20 · Answer

I keep rolling back to 7.3.x but I should be able to rig up a test environment on AIX. This is the only datazone we have on AIX (and it's production too). The Linux hosted ones all upgraded without a hitch.

Jason20 · Answer

That's a good idea, it will be an interesting data point to see if it'll come up with 7.5

DavidHampson · Answer

At present the only way you have got to go is to reinstall the old version and do a DR so trying the newest version should not be too much of a problem...

ZENG1 · Answer

does your upgrade works ? Maybe it worth to try disable strong authentication before upgrade? just a thought ...

NetWorker

Networker not starting after 7.4 upgrade

Was this post helpful?