NetWorker: Fixing inconsistent NSR peer information

Backups, recoveries and communications fail on client with the following errors:

Error: 'nsrexecd: SYSTEM error: There is already a machine using the name (NETWORKER_CLIENT_NAM). Either choose a different name for your machine, or delete the "NSR peer information" entry for (NETWORKER_CLIENT_NAM) on host: (HOST_NAME)'
Error: 'nsrexecd: SYSTEM error: Connection reset by peer'
Error: Could not get session key from NETWORKER_CLIENT_NAME for GSS authentication with SERVER_NAME: Authentication error; why = Server rejected credential

Similar errors also appear in the systems daemon.raw log file:

• Linux: /nsr/logs/daemon.raw
• Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw
• NetWorker: How to use nsr_render_log to render .raw log files

All NetWorker servers, storage nodes, and clients contain a nsrladb folder.

• Linux: /nsr/res/nsrladb
• Windows (Default): C:\Program Files\EMC NetWorker\nsr\res\nsrladb

At the first connection, a host will request and receive the certificate for the host it connects to, and cache that hosts certificate for future comparison. This folder contains the host's peering information, and cached certificates with other NetWorker systems that have successfully authenticated using nsrauth peering. These cached certificates are reflected in the Local Hosts branch of the Configuration tree.

Peer certificate mismatches can appear from a variety or issues. For example:

• Name resolution, addressing, or communication issues
• NetWorker upgrades
• Host level system changes that caused differences between system level information and the information stored in the nsraldb.

The NSR peer information is set at the client level, not at the server level. In other words: you must connect to the NSRLA not the NSR database. To perform this, you must connect using "nsradmin -p nsrexec" or "nsradmin -p nsrexecd." 

Delete the nonmatching and cached old certificate of the client (client_name) on the NetWorker host generating the error message (host_name). 

1. Delete the certificate for the client which has been updated from any host with a copy of the old one using Local Hosts in NMC
2. Delete the old certificate from the affected client using the command line

To clear the peer information of the client machine (from server)

nsradmin -s <host_name> -p nsrexec
nsradmin> delete type: nsr peer information; peer hostname: <client_name>

To clear the peer information from the client machine.

nsradmin -p nsrexec
nsradmin> print type: nsr peer information
delete

Peer issues may also appear if there is a mismatch in authentication methods allowed. In modern NetWorker implementations, only nsrauth should be used. Legacy (pre 19.4.x) NetWorker systems may have included "oldauth." Example of NetWorker system with oldauth fallback enabled:

[root@lnx-srvr01 ~]# nsradmin -p nsrexec
NetWorker administration program.
Use the "help" command for help, "visual" for full-screen mode.
nsradmin> p type: nsrla
                        type: NSRLA;
                        name: lnx-srvr01.networker.lan;
           reverse DNS state: uncached;
              nsrmmd version: 19.14.0.0.Build.12;
             nsrsnmd version: 19.14.0.0.Build.12;
          Multi-subnet state: Disabled;
                   TCP Retry: 5;
 NW instance info operations: ;
       NW instance info file: ;
          installed products: ;
                auth methods: "0.0.0.0/0,nsrauth/oldauth";
           max auth attempts: 8;
...
...

Using oldauth as a communication fallback may work around some communication or peering failures; however, oldauth should not be used. oldauth is considered not secure.
nsrauth is the default authentication method used by all supported NetWorker releases. Clients deployed on versions later than 19.4.x do not have oldauth enabled. Upgrading a client with oldauth enabled does not automatically remove this setting. The setting must be removed manually.

[root@lnx-srvr01 ~]# nsradmin -p nsrexec
NetWorker administration program.
Use the "help" command for help, "visual" for full-screen mode.
nsradmin> . type: nsrla
Current query set
nsradmin> update auth methods: "0.0.0.0/0,nsrauth"
                auth methods: "0.0.0.0/0,nsrauth";
Update? y
updated resource id 3.0.243.13.0.0.0.0.101.15.41.103.192.168.0.6(517)
nsradmin> show auth methods
nsradmin> print
                auth methods: "0.0.0.0/0,nsrauth";
nsradmin>

Restart NetWorker services:

• Linux: systemctl restart networker
• Windows (PowerShell): net stop nsrexecd ; net start nsrexecd

Clear peer information from NetWorker systems that communicate with the host where oldauth was disabled.