nsrpush and 8.2.1.2 @ Linux

After updating server from 8.2.1.1 to 8.2.1.2, nsrpush seems to be broken (argh!).

Any operation with it just hang.  For example:

# nsrpush -s -t

91512:nsrpush: Unable to authenticate with server ' ' during RAP bind operation: Timed out

91512:nsrpush: Unable to authenticate with server ' ' during RAP bind operation: Timed out

Trying to list products I have in inventory gives the same so this is pretty much server side issue.

I already tried to restart server with cleanup of /nsr/tmp and /nsr/res/jobsdb but result is the same.  So, next step was to remove cpdb as if corruption happens there then nsrpush won't run.  So, stop NW, remove NSR's tmp, jobdb and cpdb and start again.

Now nsrupush inventory worked, but many hosts reported error during coping scripts plus server itself started to have problems, eg:

savegrp RPC severe 2 %s 1 49 80 80720 9 %s%s - %s 3 24 0  11 23 352:Remote system error 11 18 Connection refused

savegrp RAP critical 94 Unable to return a list of resources from a database: the resource database for query is null. 0

savegrp RAP warning 39 Cannot find the %s resource named '%s'. 2 11 15 33243:NSR group 26 20

savegrp RPC severe 2 %s 1 49 80 80720 9 %s%s - %s 3 24 0  11 23 352:Remote system error 11 18 Connection refused

savegrp RAP critical 94 Unable to return a list of resources from a database: the resource database for query is null. 0

savegrp RAP warning 39 Cannot find the %s resource named '%s'. 2 11 15 33243:NSR group 26 20

nsrd NSR notice 58 Scheduled resource list: %s %s exited with return code %d. 3 0 9 NSR group 0 20 1 1 4

savegrp RPC severe 2 %s 1 49 80 80720 9 %s%s - %s 3 24 0  11 23 352:Remote system error 11 18 Connection refused

savegrp RAP critical 94 Unable to return a list of resources from a database: the resource database for query is null. 0

savegrp RAP warning 39 Cannot find the %s resource named '%s'. 2 11 15 33243:NSR group 26 20

nsrd NSR notice 58 Scheduled resource list: %s %s exited with return code %d. 3 0 9 NSR group 0 20 1 1 4

nsrd NSR notice 58 Scheduled resource list: %s %s exited with return code %d. 3 0 9 NSR group 0 20 1 1 4

I decided to restart server again since nsrwatch would not come back.  At this point, while nsrpush run it was able to blow away the server so perhaps I should do more careful inventory.  Since I had full list of nsrpush inventory output, I decided to extract names of hosts where it fails and run nsrpush individually for each with sleep 30 in between.

Meanwhile I decided to check core area and I found:

total 44740

-rw------- 1 root root 12718080 Apr 18 17:37 core.15294

-rw------- 1 root root 12713984 Apr 18 14:24 core.24283

-rw------- 1 root root 12722176 Apr 18 15:45 core.26429

-rw------- 1 root root 12722176 Apr 18 17:19 core.4171

This is all before - while trying to do inventory or list repository with cpdb from 8.2.1.2.  So it seems like each time this thing would core dump.  I can't say at which time this happened though - was it when I started nsrpush or when I did nsr_shutdown or killed the process with kill.  I just forgot to check it.

Anyway, out of 175 clients 21 failed and I run those with 30 sec delay in between.  In this run 4 failed where 3 I fixed by running again. One persisted again and again - it was backup server itself.  cpdb log shows:

04/18/2015 08:17:48 PM  srcpd SYSTEM critical Unable to open resource database file /nsr/res/cpdb/sd_products.res: No such file or directory

04/18/2015 08:17:48 PM  nsrcpd NSR severe Product resource file "/nsr/res/cpdb/sd_products.res" cannot be opened.

04/18/2015 08:17:48 PM  nsrcpd RAP critical Unable to return a list of resources from a database: the resource database for query is null.

04/18/2015 08:17:48 PM  nsrcpd NSR warning Using hard-coded product list.

04/18/2015 08:17:50 PM  nsrcpd NSR notice Starting Inventory Operation on selected clients

04/18/2015 08:18:00 PM  nsrcpd NSR notice Creating client type job for client .

04/18/2015 08:18:00 PM  nsrcpd NSR severe Copying inventory scripts to client .

04/18/2015 08:18:00 PM  nsrcpd RPC severe Unable to start command "nsrrcopy -t /tmp/cp_ /" on host ' ': RPC send operation failed; peer = 127.0.0.1:7937, errno = Broken pipe

04/18/2015 08:18:00 PM  nsrcpd RPC severe RPC send operation failed; peer = 127.0.0.1:7937, errno = Broken pipe

04/18/2015 08:18:00 PM  nsrcpd RPC severe RPC send operation failed; peer = 127.0.0.1:7937, errno = Broken pipe

04/18/2015 08:18:00 PM  nsrcpd NSR severe Error copying scripts to client .

04/18/2015 08:18:05 PM  nsrcpd NSR warning The operation failed for the following clients...

04/18/2015 08:18:05 PM  nsrcpd NSR warning

Two things I see above.  One is sd_product.res missing.  This one is supposed to be inside cpdb folder which I removed previously.  It is supposed to have file once it was created by nsrpush again so I have no idea why it is not there. However, by looking at the rest of the logs, I can see this is reported for all clients which were successful with nsrpush so that is not it.  Which brings us to error itself where we see nsrrcopy failing.

I have no idea what /tmp/cp_ / is, but I assume some temp scripts to run inventory or something.  For some reason that fails and without having those it is hard to try to reproduce this more in detail.  One may argue that this is not needed for backup since you won't use nsrpush to update backup server anyway.  Fair enough, but it worked so far plus I can always use nsrpush to inventory clients and keep my inventory up to date.

And then, out of the blue, it worked after running it once again during commercial break on TV (as I was watching this as I write this).  If you use nsrpush, you probably noticed that you had to use inventory twice to get modules listed as well (as first run seems to pick up only client's version - this is still the case).

My final test, and reason how I got myself in this in the first place, was to update all those Linux clients I have. This seems work fine as before.

So, if you find yourself using this, you might see (and use) above.