Unsolved
This post is more than 5 years old
45 Posts
0
5060
Authentication Error in networker backup , group remains in hung state
Authentication Error in networker backup , group remains in hung state , below are the daemon.raw logs for the conenction refused or hung error we are facing with AIX client.
Hrvoje Crvelin Carlos Rojas William Mason : Your Inputs Plz
backup stays in session state : queued
NSR Server : Win2008R2 , 8.2.4.9 Build 1416
Client : AIX 8.2.4.0 Build 1313
Backup : NMDA and Filesystem both exhibit same issue indication authenticating problem maybe
Logs below
101040 1508068297 1 3 0 1 65012182 0 networkerclinet_hostname nsrexecd RAP notice 63 Generated new '%s': %s and keys for nsrauth RPC authentication. 2 11 19 2375:NW instance ID 12 53 01e65051-00000004-3c48880f-59e34bc9-0001fa39-12618f03
0 1508068297 1 5 0 1 65012182 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 28 @(#) Product: NetWorker
0 1508068297 1 5 0 1 65012182 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 37 @(#) Release: 8.2.4.0.Build.1313
0 1508068297 1 5 0 1 65012182 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 23 @(#) Build number: 1313
0 1508068297 1 5 0 1 65012182 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 47 @(#) Build date: Tue Nov 8 15:52:13 PST 2016
0 1508068297 1 5 0 1 65012182 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 27 @(#) Build arch.: aixpower
0 1508068297 1 5 0 1 65012182 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 29 @(#) Build info: DBG=0,OPT=
78374 1508087611 5 3 13 1 65012182 0 networkerclinet_hostname nsrexecd RAP critical 150 An unsuccessful attempt to impersonate user %s was made by %s. If %s can be trusted, grant the user "Administrator" privileges on the NetWorker server 3 13 28 administrator@ 13 20 SYSTEM@networkerServer_hostname.fgb.ae 13 20 SYSTEM@networkerServer_hostname.fgb.ae
152095 1508087688 5 97 968 1297 65012182 0 networkerclinet_hostname nsrexecd NSR critical 86 Unable to authenticate user %s: %s. Consult %s's daemon log for additional information 3 12 33 SYSTEM/networkerServer_hostname.fgb.ae@NT AUTHORITY 49 237 80719 80 Unable to establish a connection with host %s for authentication credentials: %s 2 12 13 networkerServer_hostname.fgb.ae 49 118 80720 9 %s%s - %s 3 24 0 11 23 352:Remote system error 11 56 A remote host did not respond within the timeout period. 12 13 networkerServer_hostname.fgb.ae
152095 1508087763 5 97 968 1298 65012182 0 networkerclinet_hostname nsrexecd NSR critical 86 Unable to authenticate user %s: %s. Consult %s's daemon log for additional information 3 12 33 SYSTEM/networkerServer_hostname.fgb.ae@NT AUTHORITY 49 237 80719 80 Unable to establish a connection with host %s for authentication credentials: %s 2 12 13 networkerServer_hostname.fgb.ae 49 118 80720 9 %s%s - %s 3 24 0 11 23 352:Remote system error 11 56 A remote host did not respond within the timeout period. 12 13 networkerServer_hostname.fgb.ae
0 1508088788 1 5 0 1 64880902 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 28 @(#) Product: NetWorker
0 1508088788 1 5 0 1 64880902 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 37 @(#) Release: 8.2.4.0.Build.1313
0 1508088788 1 5 0 1 64880902 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 23 @(#) Build number: 1313
0 1508088788 1 5 0 1 64880902 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 47 @(#) Build date: Tue Nov 8 15:52:13 PST 2016
0 1508088788 1 5 0 1 64880902 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 27 @(#) Build arch.: aixpower
0 1508088788 1 5 0 1 64880902 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 29 @(#) Build info: DBG=0,OPT=
101040 1508144964 1 3 0 1 32637344 0 networkerclinet_hostname nsrexecd RAP notice 63 Generated new '%s': %s and keys for nsrauth RPC authentication. 2 11 19 2375:NW instance ID 12 53 46c0abe3-00000004-fa69262c-59e47744-0001fa39-12618f03
0 1508144964 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 28 @(#) Product: NetWorker
0 1508144964 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 37 @(#) Release: 8.2.4.0.Build.1313
0 1508144964 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 23 @(#) Build number: 1313
0 1508144964 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 47 @(#) Build date: Tue Nov 8 15:52:13 PST 2016
0 1508144964 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 27 @(#) Build arch.: aixpower
0 1508144964 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 29 @(#) Build info: DBG=0,OPT=
90471 1508148600 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
79345 1508149836 2 12 13 1315 32637344 0 networkerclinet_hostname nsrexecd GSS warning 137 User "%s" does not belong to group "%s" but claimed to belong to it. The group will be removed from the user's authentication credential. 2 13 6 oracle 0 5 staff
90471 1508152601 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508153069 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508153359 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508153372 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508156719 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508156738 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508157958 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508157980 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
84845 1508167651 5 2 12 258 32637344 0 networkerclinet_hostname nsrexecd RPC critical 39 Aborting client connection from %s: %s. 2 14 13 xx.xx.xx.xxx_IP_OF_A_RANDOM_CLIENT_IN_NETWORKER 24 56 A remote host did not respond within the timeout period.
84845 1508167913 5 2 12 1 32637344 0 networkerclinet_hostname nsrexecd RPC critical 39 Aborting client connection from %s: %s. 2 14 13 xx.xx.xx.xxx_IP_OF_A_RANDOM_CLIENT_IN_NETWORKER 24 56 A remote host did not respond within the timeout period.
84845 1508218945 5 2 12 515 32637344 0 networkerclinet_hostname nsrexecd RPC critical 39 Aborting client connection from %s: %s. 2 14 13 xx.xx.xx.xxx_IP_OF_A_RANDOM_CLIENT_IN_NETWORKER 24 56 A remote host did not respond within the timeout period.
84845 1508219208 5 2 12 1 32637344 0 networkerclinet_hostname nsrexecd RPC critical 39 Aborting client connection from %s: %s. 2 14 13 xx.xx.xx.xxx_IP_OF_A_RANDOM_CLIENT_IN_NETWORKER 24 56 A remote host did not respond within the timeout period.
84845 1508302601 5 2 12 515 32637344 0 networkerclinet_hostname nsrexecd RPC critical 39 Aborting client connection from %s: %s. 2 14 13 xx.xx.xx.xxx_IP_OF_A_RANDOM_CLIENT_IN_NETWORKER 24 56 A remote host did not respond within the timeout period.
84845 1508302864 5 2 12 1 32637344 0 networkerclinet_hostname nsrexecd RPC critical 39 Aborting client connection from %s: %s. 2 14 13 xx.xx.xx.xxx_IP_OF_A_RANDOM_CLIENT_IN_NETWORKER 24 56 A remote host did not respond within the timeout period.
90471 1508315329 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508315827 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508317310 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508318830 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508318922 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
90471 1508328907 1 5 0 1 32637344 0 networkerclinet_hostname nsrexecd NSR notice 104 User '%s@%s' is authorized by server '%s' through optional remote NetWorker server based authentication. 3 13 13 administrator 12 13 networkerServer_hostname.fgb.ae 12 13 networkerServer_hostname.fgb.ae
84845 1508331449 5 2 12 515 32637344 0 networkerclinet_hostname nsrexecd RPC critical 39 Aborting client connection from %s: %s. 2 14 13 xx.xx.xx.xxx_IP_OF_A_RANDOM_CLIENT_IN_NETWORKER 24 56 A remote host did not respond within the timeout period.
84845 1508331711 5 2 12 1 32637344 0 networkerclinet_hostname nsrexecd RPC critical 39 Aborting client connection from %s: %s. 2 14 13 xx.xx.xx.xxx_IP_OF_A_RANDOM_CLIENT_IN_NETWORKER 24 56 A remote host did not respond within the timeout period.
0 1508590712 1 5 0 1 43254346 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 28 @(#) Product: NetWorker
0 1508590712 1 5 0 1 43254346 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 37 @(#) Release: 8.2.4.0.Build.1313
0 1508590712 1 5 0 1 43254346 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 23 @(#) Build number: 1313
0 1508590712 1 5 0 1 43254346 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 47 @(#) Build date: Tue Nov 8 15:52:13 PST 2016
0 1508590712 1 5 0 1 43254346 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 27 @(#) Build arch.: aixpower
0 1508590712 1 5 0 1 43254346 0 networkerclinet_hostname nsrexecd NSR notice 2 %s 1 0 29 @(#) Build info: DBG=0,OPT=
79345 1508590872 2 12 13 1571 43254346 0 networkerclinet_hostname nsrexecd GSS warning 137 User "%s" does not belong to group "%s" but claimed to belong to it. The group will be removed from the user's authentication credential. 2 13 6 oracle 0 5 staff
networkerclinet_hostname[/nsr]#
ble1
2 Intern
2 Intern
•
14.3K Posts
0
October 25th, 2017 05:00
It is a bit strange to see random client in NW being mentioned in client side log, but this can happen for example if someone made a mistake and entered in DNS IP of some client to respond to localhost and/or localhost.domain (seen it couple of times). If this is not the case, I would stop nsr on client, remove nsrladb, remove auth key for the client on server and start daemons again on client. That should give clean picture as far as auth goes.
DataProtectxr
45 Posts
0
November 5th, 2017 05:00
Thanks Hrvoje , can u confirm the procedure to remove the auth key.
Also these are some errors seen on the client , which makes us believe something at the hostname / IP resolution messing with the smooth backups.
nsrexecd SYSTEM warning 42 Failed to start NSR portmapper service: %s 1 24 34 The socket name is already in use.
90304 1509814819 5 5 0 1 13632294 0 _nsr_clientname_ nsrexecd NSR critical 39 Unable to start the portmapper service. 0
The workaround that works currently is - Restart of Networker Services on the NSR server , but his has dependencies and we are still struggling with a permanent fix.
DataProtectxr
45 Posts
0
November 6th, 2017 06:00
Hrvoje Crvelin : Also as observed we have two clients exhibiting the same behavior , similarity being : both have filesys and NMDA backup configured in networker, AIX clients and both have the backups getting queued.
As mentioned above Networker server's NSR service refresh temporarily corrects the issue.
Peer Info have been cleared at both sides and tried without any positive signs.
Strangely , on the backup server opening the interactive jobkill displays , a number of NMDA save jobs that seem hung and are not terminating which maybe an indicator of the session being hung . Sample O/P below :
------------------------------------------------------
job id: 4289;
name: NetWorker Module for Databases and Applications;
type: save job;
command: ;
NW Client name/id: ;
start time: 1509673828;
------------------------------------------------------
job id: 4288;
name: NetWorker Module for Databases and Applications;
type: save job;
command: ;
NW Client name/id: ;
start time: 1509673828;
------------------------------------------------------
job id: 4286;
name: NetWorker Module for Databases and Applications;
type: save job;
command: ;
NW Client name/id: ;
start time: 1509673729;
Carlos RojasWilliam MasonAdam Kirkpatrick @Ian Anderson : Appreciate your inputs and assistance