Start a Conversation

Unsolved

This post is more than 5 years old

6161

February 5th, 2017 13:00

NetWorker "recover" not working: 53362:recover: Cannot start session with server Server: RPC server is unavailable

Starting the NetWorker CLI command on the client leads to the error message:

[root@Client ~]# recover -c Client -s Server

53362:recover: Cannot start session with server Server: RPC server is unavailable

76677:recover: RPC server is unavailable

NW Client release: 8.2.3.8 (RHEL 7.2)

NW Server release: 8.0.3.6 (RHEL 6.2)

Communication between NW client and NW server seams to be OK:

Client -> Server:

[root@Client ~]# nsradmin -s Server -p nsrexec

NetWorker administration program.

Use the "help" command for help, "visual" for full-screen mode.

nsradmin>

rpcinfo -p Server

   program vers proto   port  service

    100000    4   tcp    111  portmapper

    100000    3   tcp    111  portmapper

    100000    2   tcp    111  portmapper

    100000    4   udp    111  portmapper

    100000    3   udp    111  portmapper

    100000    2   udp    111  portmapper

[root@Client ~]# telnet Server 7937

Trying x.x.x.x...

Connected to Server.

Escape character is '^]'.

Server -> Client

root@Server:/root $ nsradmin -s Client  -p nsrexec

NetWorker administration program.

Use the "help" command for help, "visual" for full-screen mode.

nsradmin> ^C

root@Server:/root $ rpcinfo -p Client

   program vers proto   port  service

    100000    4   tcp    111  portmapper

    100000    3   tcp    111  portmapper

    100000    2   tcp    111  portmapper

    100000    4   udp    111  portmapper

    100000    3   udp    111  portmapper

    100000    2   udp    111  portmapper

root@Server:/root $ telnet Client 7937

Trying x.x.x.x...

Connected to Client.

Escape character is '^]'.

^]

telnet> Connection closed.

According to the EMS's NetWorker Compatibility Guide NetWorker server 8.0 and Client 8.2 should be compatible.

How to solve the problem?

Thanks and regards

Michael

25 Posts

February 5th, 2017 23:00

Please check for portmapper 7938 ports as well and next two ports, i.e 7937-7940

Check for peer information conflicts.

Update here and we'll provide suggestions accordingly.

February 6th, 2017 03:00

Hi Karthik,

via telnet I'm able to access port7938 and 7940, but not 7939.
Can  you please explain me how to check "check for portmapper 7938 ports"?

Thanks a lot for your help :-)

Regards

Michael

14.3K Posts

February 6th, 2017 05:00

On client running 8.2, you should use nsrrpcinfo and not rpcinfo.  Make sure also that iptables is not "blocking the view".

25 Posts

February 6th, 2017 05:00

You can try disabling firewall at both the endpoints, server and client.

Check output of nsrports, it should be 7937-9936, else logon to client and issue the below commands one by one.

nsradmin -p nsrexecd

p type: NSR system port ranges

And paste the output. Also, check peer information conflicts

February 6th, 2017 05:00

Hi Hrvoje,

thanks for your help. Following the nsrrpcinfo output (client -> server, client -> client and nsrrpcinfo with -n -t option) . Server is using NW 8.0.3, therefor no nsrrpcinfo test server -> client.

Between client and server a Firewall is in place. The tcp ports 22,111 and 7937-7990 have been requested to be opened bidirectional. Backups are working fin nrsports was used to configure the port range on the client (server is using the same port range).

[root@client ~]#nsrrpcinfo -p server

PROGRAM VERSION PROTOCOL PORT SERVICE    

100000  2      tcp      7938 nsrportmapper Port Mapper

100000  2      udp      7938 nsrportmapper Port Mapper

390436  1      tcp      7958 nsrexecd      GSS Authentication

390435  1      tcp      7980 nsrexecd      Resource Mirror

390113  1      tcp      7937 nsrexecd      Remote Execution

390103  2      tcp      7959 nsrd          Server

390109  2      tcp      7959 nsrd          Remote Monitoring

390110  1      tcp      7959 nsrd          Jukebox Management

390120  1      tcp      7959 nsrd          License Manager

390109  2      udp      7982 nsrd          Remote Monitoring

390105  5      tcp      7962 nsrindexd    Index Database

390105  6      tcp      7962 nsrindexd    Index Database

390438  1      tcp      7979 nsrlogd      Audit Logging

390107  5      tcp      7953 nsrmmdbd      Media Database

390107  6      tcp      7953 nsrmmdbd      Media Database

390111  1      tcp      7970 nsrsnmd      Storage Node Management

390104  605    tcp      7955 nsrmmd        Media Multiplexor

390104  205    tcp      7946 nsrmmd        Media Multiplexor

390433  1      tcp      7942 nsrjobd      Job Management

[root@client ~]# nsrrpcinfo -p client

PROGRAM VERSION PROTOCOL PORT SERVICE    

100000  2      tcp      7938 nsrportmapper Port Mapper

100000  2      udp      7938 nsrportmapper Port Mapper

390436  1      tcp      7961 nsrexecd      GSS Authentication

390435  1      tcp      7940 nsrexecd      Resource Mirror

390113  1      tcp      7937 nsrexecd      Remote Executio

[root@client~]# nsrrpcinfo  -n 7938 -t server 100000

Program 100000 version 2 ready and waiting

[root@client ~]# nsrrpcinfo  -n 7937 -t server 390113

Program 390113 version 1 ready and waiting

February 6th, 2017 06:00

Hi Hrvoje,

still one thing to mention: The tcp ports 22,111 and 7937-7990 have been requested, but not the udp ports.

Regards

Michael

February 6th, 2017 06:00

Hi Kathrink,

nsrports on cllient and server:

Service ports: 7937-7990

Connection ports: 0-0

The backup, "browse wizzard" and directed recovery are working. I've no messages in the daemon.raw file regarding peer information conflicts. I'm not sure what exactly what kind of "peer information conflicts" I should look for. Can you please explain?

Thanks and regards

Michael

14.3K Posts

February 7th, 2017 03:00

Port wise it looks to be ok.  I believe recover has debug switch so you can try something like following:

recover -D9 -s -a -d /tmp /etc/hosts

This will create bunch of data (just tested it), but it would be interesting to see where it breaks.  Also, when you run it, check netstat output on both client and server (and also check /etc/hosts itself to make sure that perhaps some faulty entry is not driving connection sideways).

February 7th, 2017 07:00

Hi Hrvoje,

following the output of the recover command and at the end some lines from the netstat output .

Thanks for your help.

Regards

Michael

[root@NW_client ~]# recover -D9 -s NW_server -a -d /tmp /etc/hosts

02/07/17 14:34:21.691215 Creating tcp RPC client handle with host NW_client (::1, prog 390113, vers 1)

02/07/17 14:34:21.691564 Creating TCP/IPv6 RPC client handle prog 390113 vers 1

02/07/17 14:34:21.692494 Found NetWorker portmapper port, setting to 7938.

02/07/17 14:34:21.692587 the service nsrrpc was set, ignoring the system portmapper02/07/17 14:34:21.692642 Creating TCP/IPv6 RPC client handle prog 100000 vers 2

02/07/17 14:34:21.692783 Attempting to bind IPv6 socket descriptor 3

02/07/17 14:34:21.692966 Socket bound to OS determined port

02/07/17 14:34:21.935177 Setting default RPC socket send buffer size to 262144

02/07/17 14:34:21.935224 Setting default RPC socket receive buffer size to 262144

02/07/17 14:34:21.935242 Setting RPC socket send buffer size to 262144

02/07/17 14:34:21.935262 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:21.935541 Bound TCP/IPv6 socket descriptor 3 to port 7938

02/07/17 14:34:21.935586 Auth methods not set, using default

02/07/17 14:34:21.935816 Attempting to bind IPv6 socket descriptor 3

02/07/17 14:34:21.935858 Socket bound to OS determined port

02/07/17 14:34:21.935875 Setting RPC socket send buffer size to 262144

02/07/17 14:34:21.935891 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:21.936000 Bound TCP/IPv6 socket descriptor 3 to port 7937

02/07/17 14:34:21.936043 Auth methods not set, using default

02/07/17 14:34:21.937577 User's total groups = 1, max groups set in environment or calculated = 512 and max groups buffer size = 10914

02/07/17 14:34:21.937670 Creating tcp RPC client handle with host localhost (::1, prog 390436, vers 1)

02/07/17 14:34:21.937712 Creating TCP/IPv6 RPC client handle prog 390436 vers 1

02/07/17 14:34:21.937773 Creating TCP/IPv6 RPC client handle prog 100000 vers 2

02/07/17 14:34:21.937850 Attempting to bind IPv6 socket descriptor 4

02/07/17 14:34:21.937902 Socket bound to OS determined port

02/07/17 14:34:21.937931 Setting RPC socket send buffer size to 262144

02/07/17 14:34:21.937964 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:21.938066 Bound TCP/IPv6 socket descriptor 4 to port 7938

02/07/17 14:34:21.938123 Auth methods not set, using default

02/07/17 14:34:21.938331 Attempting to bind IPv6 socket descriptor 4

02/07/17 14:34:21.938381 Socket bound to OS determined port

02/07/17 14:34:21.938404 Setting RPC socket send buffer size to 262144

02/07/17 14:34:21.938420 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:21.938489 Bound TCP/IPv6 socket descriptor 4 to port 7961

02/07/17 14:34:21.938527 Auth methods not set, using default

02/07/17 14:34:21.956871 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:21.957219 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:21.957518 Adding range 7937-7990 to "service" port type.

02/07/17 14:34:21.958170 Setting DNS cache TTLs to 1800 secs for positive lookups and 1800 secs for negative lookups

02/07/17 14:34:21.958363 clu_init:ENTRY

02/07/17 14:34:21.958404 calling clu_init_lc()

02/07/17 14:34:21.958440

clu_init_lc(): ENTRY...

02/07/17 14:34:21.958492

get_lc_fspath_vhost_map(): ENTRY ...

02/07/17 14:34:21.958654 No access to file /usr/sbin/lcmap: No such file or directory

02/07/17 14:34:21.958686

clu_init_lc(): Can't build fspath_vhost_map...

02/07/17 14:34:21.958703

dump_map_lc(): ENTRY ...

02/07/17 14:34:21.958742 Lc_use_local_vhost_list = FALSE

02/07/17 14:34:21.958758 MOUNTED filesystems

02/07/17 14:34:21.958772

dump_map_lc(): EXIT ...

02/07/17 14:34:21.958828 calling clu_is_cluster_host_lc()

02/07/17 14:34:21.958850

clu_is_cluster_host_lc(): ENTRY ...

02/07/17 14:34:21.958873

clu_init_hp(): ENTRY ...

02/07/17 14:34:21.958910 LGMutex and LGList creation succeeded

02/07/17 14:34:21.958930

get_user_delimiter_char(): ENTRY ...

02/07/17 14:34:21.958955 Searching for user-specified delimiter in /opt/cmcluster/conf/.nsr_cluster

02/07/17 14:34:21.958980

destroy_cmap_hp(): ENTRY ...

02/07/17 14:34:21.959011

destroy_pkglist(): ENTRY ...

02/07/17 14:34:21.959033

destroy_local_vost(): ENTRY ...

02/07/17 14:34:21.959072

destroy_local_vost(): EXIT ...

02/07/17 14:34:21.959090

destroy_cmap_hp(): EXIT ...

02/07/17 14:34:21.959138

get_cmap_hp(): ENTRY ...

02/07/17 14:34:21.959161

destroy_cmap_hp(): ENTRY ...

02/07/17 14:34:21.959206

destroy_pkglist(): ENTRY ...

02/07/17 14:34:21.959225

destroy_local_vost(): ENTRY ...

02/07/17 14:34:21.959249

destroy_local_vost(): EXIT ...

02/07/17 14:34:21.959266

destroy_cmap_hp(): EXIT ...

02/07/17 14:34:21.959306

dump_cmap_hp(): ENTRY ...

02/07/17 14:34:21.959327

dump_cmap_hp(): EXIT ...

02/07/17 14:34:21.959352 Entering build_local_vhost_tab function

02/07/17 14:34:21.959375 Exitting build_local_vhost_tab function

02/07/17 14:34:21.959396

dump_pkglist(): ENTRY ...

02/07/17 14:34:21.959419

dump_pkglist(): EXIT ...

02/07/17 14:34:21.959438 clu_init_hp(): returning void

02/07/17 14:34:21.959461 clu_is_native_cluster_host:ENTRY

02/07/17 14:34:21.959475

clu_is_cluster_host_lc(): ENTRY ...

02/07/17 14:34:21.959499

clu_is_cluster_host_hp(): ENTRY ...

02/07/17 14:34:21.959522 this_phostname=NW_client

02/07/17 14:34:21.959548 NetWorker is not configured for cluster system, cmviewcl will be bypassed ...

02/07/17 14:34:21.959580 clu_is_native_cluster_host:EXIT:returning value=0

02/07/17 14:34:21.959608 clu_init:EXIT

02/07/17 14:34:21.959649 Reverse DNS lookup failed for address fe80::250:56ff:fe01:a5e

02/07/17 14:34:21.959684 Reverse DNS lookup failed for address fe80::250:56ff:fe01:a5f

02/07/17 14:34:21.959700 Reverse DNS lookup failed for address fe80::250:56ff:fe01:a60

02/07/17 14:34:21.959748 Creating tcp RPC client handle with host NW_server (10.105.x.x, prog 390433, vers 1)

02/07/17 14:34:21.959773 Creating TCP/IPv4 RPC client handle prog 390433 vers 1

02/07/17 14:34:21.959790 Creating TCP/IPv4 RPC client handle prog 100000 vers 2

02/07/17 14:34:21.959854 Attempting to bind IPv4 socket descriptor 3

02/07/17 14:34:21.959908 Socket bound to OS determined port

02/07/17 14:34:21.959932 Setting RPC socket send buffer size to 262144

02/07/17 14:34:21.959947 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:21.960694 Bound TCP/IPv4 socket descriptor 3 to port 7938

02/07/17 14:34:21.961408 Attempting to bind IPv4 socket descriptor 3

02/07/17 14:34:21.961490 Socket bound to OS determined port

02/07/17 14:34:21.961521 Setting RPC socket send buffer size to 262144

02/07/17 14:34:21.961558 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:21.962036 Bound TCP/IPv4 socket descriptor 3 to port 7942

02/07/17 14:34:21.962167 User's total groups = 1, max groups set in environment or calculated = 512 and max groups buffer size = 10914

02/07/17 14:34:21.962241 Creating tcp RPC client handle with host localhost (::1, prog 390436, vers 1)

02/07/17 14:34:21.962285 Creating TCP/IPv6 RPC client handle prog 390436 vers 1

02/07/17 14:34:21.962360 Attempting to bind IPv6 socket descriptor 4

02/07/17 14:34:21.962410 Socket bound to OS determined port

02/07/17 14:34:21.962437 Setting RPC socket send buffer size to 262144

02/07/17 14:34:21.962467 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:21.962678 Bound TCP/IPv6 socket descriptor 4 to port 7961

02/07/17 14:34:21.979838 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.003916 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.004894 Registered session id:1, fd:3, idx:0, ssn_max_pollfd:1, 1(512)

02/07/17 14:34:22.030317 Creating tcp RPC client handle with host NW_server (10.105.x.x, prog 390109, vers 2)

02/07/17 14:34:22.030390 Creating TCP/IPv4 RPC client handle prog 390109 vers 2

02/07/17 14:34:22.030424 Creating TCP/IPv4 RPC client handle prog 100000 vers 2

02/07/17 14:34:22.030497 Attempting to bind IPv4 socket descriptor 4

02/07/17 14:34:22.030540 Socket bound to OS determined port

02/07/17 14:34:22.030584 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.030617 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.031162 Bound TCP/IPv4 socket descriptor 4 to port 7938

02/07/17 14:34:22.031844 Attempting to bind IPv4 socket descriptor 4

02/07/17 14:34:22.031940 Socket bound to OS determined port

02/07/17 14:34:22.031973 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.032007 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.032602 Bound TCP/IPv4 socket descriptor 4 to port 7959

02/07/17 14:34:22.032853 User's total groups = 1, max groups set in environment or calculated = 512 and max groups buffer size = 10914

02/07/17 14:34:22.033016 Creating tcp RPC client handle with host NW_server (10.105.x.x, prog 390109, vers 2)

02/07/17 14:34:22.033210 Creating TCP/IPv4 RPC client handle prog 390109 vers 2

02/07/17 14:34:22.033377 Creating TCP/IPv4 RPC client handle prog 100000 vers 2

02/07/17 14:34:22.033578 Attempting to bind IPv4 socket descriptor 5

02/07/17 14:34:22.033793 Socket bound to OS determined port

02/07/17 14:34:22.033955 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.034129 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.034811 Bound TCP/IPv4 socket descriptor 5 to port 7938

02/07/17 14:34:22.035538 Attempting to bind IPv4 socket descriptor 5

02/07/17 14:34:22.035620 Socket bound to OS determined port

02/07/17 14:34:22.035660 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.035705 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.036207 Bound TCP/IPv4 socket descriptor 5 to port 7959

02/07/17 14:34:22.036300 User's total groups = 1, max groups set in environment or calculated = 512 and max groups buffer size = 10914

02/07/17 14:34:22.036367 Creating tcp RPC client handle with host localhost (::1, prog 390436, vers 1)

02/07/17 14:34:22.036407 Creating TCP/IPv6 RPC client handle prog 390436 vers 1

02/07/17 14:34:22.036473 Attempting to bind IPv6 socket descriptor 6

02/07/17 14:34:22.036525 Socket bound to OS determined port

02/07/17 14:34:22.036553 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.036581 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.036793 Bound TCP/IPv6 socket descriptor 6 to port 7961

02/07/17 14:34:22.050673 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.053375 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.054438 lgto_auth for `nsrd' succeeded

02/07/17 14:34:22.054534 User's total groups = 1, max groups set in environment or calculated = 512 and max groups buffer size = 10914

02/07/17 14:34:22.054597 Creating tcp RPC client handle with host localhost (::1, prog 390436, vers 1)

02/07/17 14:34:22.054633 Creating TCP/IPv6 RPC client handle prog 390436 vers 1

02/07/17 14:34:22.054707 Attempting to bind IPv6 socket descriptor 5

02/07/17 14:34:22.054779 Socket bound to OS determined port

02/07/17 14:34:22.054812 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.054840 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.055037 Bound TCP/IPv6 socket descriptor 5 to port 7961

02/07/17 14:34:22.071776 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.076016 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.080877 Recovering files into /tmp

02/07/17 14:34:22.081030 read filename of /etc/hosts

02/07/17 14:34:22.081151

clu_is_cluster_host_lc(): ENTRY ...

02/07/17 14:34:22.081377 clu_pathl_gethost:ENTRY

02/07/17 14:34:22.081598 input path=/etc/hosts

02/07/17 14:34:22.081760 clu_pathl_gethost:EXIT:Clu_cluster_type is undefined

02/07/17 14:34:22.082598 clu_pathl_free:ENTRY

02/07/17 14:34:22.082796 clu_pathl_free:EXIT

02/07/17 14:34:22.082946 Creating tcp RPC client handle with host NW_server (10.105.x.x, prog 390103, vers 2)

02/07/17 14:34:22.083134 Creating TCP/IPv4 RPC client handle prog 390103 vers 2

02/07/17 14:34:22.083295 Creating TCP/IPv4 RPC client handle prog 100000 vers 2

02/07/17 14:34:22.083501 Attempting to bind IPv4 socket descriptor 4

02/07/17 14:34:22.083675 Socket bound to OS determined port

02/07/17 14:34:22.083845 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.084005 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.084653 Bound TCP/IPv4 socket descriptor 4 to port 7938

02/07/17 14:34:22.085327 Attempting to bind IPv4 socket descriptor 4

02/07/17 14:34:22.085475 Socket bound to OS determined port

02/07/17 14:34:22.085631 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.085798 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.086356 Bound TCP/IPv4 socket descriptor 4 to port 7959

02/07/17 14:34:22.086529 User's total groups = 1, max groups set in environment or calculated = 512 and max groups buffer size = 10914

02/07/17 14:34:22.086694 Creating tcp RPC client handle with host localhost (::1, prog 390436, vers 1)

02/07/17 14:34:22.086866 Creating TCP/IPv6 RPC client handle prog 390436 vers 1

02/07/17 14:34:22.087081 Attempting to bind IPv6 socket descriptor 5

02/07/17 14:34:22.087237 Socket bound to OS determined port

02/07/17 14:34:22.087395 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.087556 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.087868 Bound TCP/IPv6 socket descriptor 5 to port 7961

02/07/17 14:34:22.102232 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.104576 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.105554 User's total groups = 1, max groups set in environment or calculated = 512 and max groups buffer size = 10914

02/07/17 14:34:22.106740 lgto_auth for `nsrindexd' succeeded

02/07/17 14:34:22.106918 User's total groups = 1, max groups set in environment or calculated = 512 and max groups buffer size = 10914

02/07/17 14:34:22.107061 Creating tcp RPC client handle with host localhost (::1, prog 390436, vers 1)

02/07/17 14:34:22.107235 Creating TCP/IPv6 RPC client handle prog 390436 vers 1

02/07/17 14:34:22.107426 Attempting to bind IPv6 socket descriptor 5

02/07/17 14:34:22.107584 Socket bound to OS determined port

02/07/17 14:34:22.107742 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.107927 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.108259 Bound TCP/IPv6 socket descriptor 5 to port 7961

02/07/17 14:34:22.122861 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.123045 Creating tcp RPC client handle with host NW_server (10.105.x.x, prog 390105, vers 6)

02/07/17 14:34:22.123120 Creating TCP/IPv4 RPC client handle prog 390105 vers 6

02/07/17 14:34:22.123156 Creating TCP/IPv4 RPC client handle prog 100000 vers 2

02/07/17 14:34:22.123232 Attempting to bind IPv4 socket descriptor 5

02/07/17 14:34:22.123279 Socket bound to OS determined port

02/07/17 14:34:22.123308 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.123332 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.123877 Bound TCP/IPv4 socket descriptor 5 to port 7938

02/07/17 14:34:22.124521 Attempting to bind IPv4 socket descriptor 5

02/07/17 14:34:22.124593 Socket bound to OS determined port

02/07/17 14:34:22.124627 Setting RPC socket send buffer size to 262144

02/07/17 14:34:22.124652 Setting RPC socket recv buffer size to 262144

02/07/17 14:34:22.125128 Bound TCP/IPv4 socket descriptor 5 to port 7962

02/07/17 14:34:22.141196 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/07/17 14:34:22.363141 calling clntnsr_start_pools_2_2

53362:recover: Cannot start session with server NW_server: RPC server is unavailable

02/07/17 14:34:22.364943 Updating the Total Recover size: 0 and Total Files: 0 to jobdb

02/07/17 14:34:22.366123 Unregistered session id:1, fd:3, idx:0, ssn_max_pollfd:0, 0(512)

02/07/17 14:34:22.366273 indexbr_entry_done()

02/07/17 14:34:22.366426    exit indexbr_entry_done()

root@NW_server:/root $ netstat

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address              Foreign Address            State    

tcp        0      0 NW_server:7942    NW_client:57048          ESTABLISHED

tcp        0      0 NW_server:7942    NW_client:40047          TIME_WAIT

[admin@NW_client ~]$ netstat

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address          Foreign Address        State    

tcp        0      0 NW_client-br:43656  NW_server:nsrrpc      TIME_WAIT

tcp        0      0 NW_client-br:ssh    deratvbr02:45905        ESTABLISHED

tcp        0      0 NW_client-br:35796  NW_server:nsrrpc      TIME_WAIT

tcp        0      0 NW_client-br:37281  NW_server:nsrrpc      TIME_WAIT

tcp        0      0 NW_client-br:35046  NW_server:7959        TIME_WAIT

tcp        0      0 NW_client-br:37316  NW_server:7942        ESTABLISHED

tcp        0      0 NW_client:53461      ldap14:ldap            ESTABLISHED

tcp        0      0 NW_client-br:38978  NW_server:nsrrpc      TIME_WAIT

tcp        0      0 NW_client-br:43004  NW_server:7959        TIME_WAIT

tcp        0      0 NW_client-br:54392  NW_server:7959        ESTABLISHED

tcp        0      0 NW_client-br:51686  NW_server:nsrrpc      TIME_WAIT

tcp        0      0 NW_client-br:44438  NW_server:nsrrpc      TIME_WAIT

tcp        0      0 NW_client-br:40796  NW_server:nsrrpc      TIME_WAIT

tcp        0      0 NW_client-br:40062  NW_server:nsrrpc      TIME_WAIT

tcp        0      0 NW_client-br:55054  NW_server:nsrrpc      TIME_WAIT

tcp        0      0 NW_client-br:ssh    NW_server:34360        ESTABLISHED

tcp6      0      0 localhost:41265        localhost:7961          TIME_WAIT

tcp6      0      0 localhost:46364        localhost:7961          TIME_WAIT

tcp6      0      0 localhost:40040        localhost:7961          TIME_WAIT

tcp6      0      0 localhost:37759        localhost:7961          TIME_WAIT

tcp6      0      0 localhost:41995        localhost:7961          TIME_WAIT

tcp6      0      0 localhost:59891        localhost:7937          TIME_WAIT

tcp6      0      0 localhost:51083        localhost:7961          TIME_WAIT

February 7th, 2017 13:00

Hi Hrovje,

the hostname is NW_client. The server has two interfaces (two IPs). In the /etc/hosts file I've assigned two different IPs to two different hostnames: One IP for NW_client and one IP for NW_client-br (the dedicated BnR interface).

On the NetWorker server (and the NW storage node and NW console server) I've added the BnR IP address of the client with NW_client-bnr hostname and all host names used on the client.
(e. g. 10.192.17.4    NW_client-br      NW_client     NW_client.domain.com).

We've also several interfaces on our NW server, storage nodes and console server. Here we use the same approach: On the client we add IP-backup-interface, hostname, hostname-VLANid, hostname.VLANid.domain.com to the clients /etc/hosts file.

This is the approach we've been using in the past years without problems. From my point of view, I did this time the same tasks as I did in the past.

At the moment I would guess that NW server 8.0.3. and NW client 8.2.3. are not that compatible as they should be or maybe there is a "software glitch" on the NetWorker server. But recover works using other (older) NetWorker clients.

Thanks and regards

Michael

14.3K Posts

February 7th, 2017 13:00

I see NW_client and NW_client-br mentioned.  Do you also have NW_server-br?  (I would assume so if -br is separate segment on which traffic should go).

14.3K Posts

February 8th, 2017 04:00

I'm using the same (both SP2 and SP3) and also multihomed in pretty much similar way as you have it and it works just fine.  If I would replicate your setup to mine, it means I use clients defined in NW as NW_client-br and in each client server backup interface I place NW_server-br and for storage node NW_sn-br.  That way I make sure -br VLAN is only used.  I suspect some issue in translation happens somewhere due to some config issue most likely.

February 8th, 2017 04:00

Hi Hrvoje,

what kind of configuration issue could this be (beside of IP wrong hostnames / IP assignment)?

Thanks and regards

Michael

14.3K Posts

February 8th, 2017 05:00

I would split it to NW config issue and network config issue.  Both are broad in some terms.

I did my test again and compared debug outputs... after the point where you get failure I get:

[..]

02/08/17 14:39:27.408682 RPC Authentication: Client successfully obtained RPCSEC_GSS credentials

02/08/17 14:39:27.770639 calling clntnsr_start_pools_2_2

02/08/17 14:39:28.044155 createentries(/etc/hosts)

02/08/17 14:39:28.044334 fullcanon(name=/etc/hosts)

02/08/17 14:39:28.044396    exit fullcanon(fullname=/etc/hosts)

02/08/17 14:39:28.044497 initializing ROOT entry

02/08/17 14:39:28.044529 makeentry()

02/08/17 14:39:28.044563 lookupentry()

02/08/17 14:39:28.044597    exit lookupentry: name=/, savetime=0, ignoreflags=0 -> 0x0000000000000000

02/08/17 14:39:28.044681 lookupentry()

02/08/17 14:39:28.044769    exit lookupentry: name=/, savetime=0, ignoreflags=0 -> 0x0000000000000000

02/08/17 14:39:28.044800 lookupentry()

02/08/17 14:39:28.044829    exit lookupentry: name=/, savetime=0, ignoreflags=0 -> 0x0000000000000000

02/08/17 14:39:28.044871 direntname(): name=/

02/08/17 14:39:28.044900    exit direntname(): normal [/]

02/08/17 14:39:28.044999 makeentry: / not found e_name /

02/08/17 14:39:28.045032    exit makeentry()

02/08/17 14:39:28.045067 lookup `/', fsid = 0xffffffffffffffff, fid = 18446744073709551615

02/08/17 14:39:28.047466 expand_check(/)

02/08/17 14:39:28.047640 User's total groups = 7, max groups set in environment or calculated = 512 and max groups buffer size = 10914

02/08/17 14:39:28.049153 lgto_auth falling back: Operation not supported

[...]

That alone means nothing... so I suspect that block before where you have:

02/07/17 14:34:22.123045 Creating tcp RPC client handle with host NW_server (10.105.x.x, prog 390105, vers 6)


...things get hairy.  I assume when you called recover command you did by using -s NW_server-bk interface?

February 8th, 2017 08:00

Hi  Hrvoje,


yes, we use only the NW_server-bk interface and this also the only interface having IP connectivity between client and server (and storage node and console).


Regards

Michael

No Events found!

Top