NetWorker: Linux Clustered Services Fail to Start "Cannot start nsrd because file is local"

NetWorker is 19.8.x or earlier
Operating system: Red Hat Enterprise Linux 8.2, 8.3, 8.4, or 8.6 

NetWorker Server cluster:

In a clustered environment, where the NetWorker Server software is configured as a HA application, after upgrading the cluster node's operating system to above Red Hat versions, NetWorker services failed to start.

NetWorker Client cluster:

In a clustered environment, where the NetWorker Client software is configured, backing up the shared file system path using Virtual Client has issues. Though backup goes through fine, it is observed that indexes are created using the Physical client and not with the Virtual client, which impacts the recovery using the Virtual client.

Error Message:

When the service fails to start, the following error message is observed in the daemon.raw log file:

73248 MM/DD/YYYY HH:MM:SS AM/PM  5 5 0 1700550464 48537 0 NODE_HOSTNAME nsrd NSR critical Can't start nsrd because /nsr/res (/nsr) is local, and NetWorker is configured as a cluster server. Use cluster manager to check NetWorker service status.

NetWorker uses Red Hat command pcs in the script /usr/sbin/lcmap. This script is used to identify the cluster logical IP address and shared path of the NetWorker Server cluster resource. Running the command lcmap should return owned paths with the shared /nsr location; however in this case it does not:

[root@NODE_HOSTNAME ~]# lcmap
type: NSR_CLU_TYPE;
clu_type: NSR_LC_TYPE;
interface version: 1.0;

There are changes to current versions of pcs commands used. As a result, the NetWorker script that uses the command was unable to determine the path-ownership in a cluster which causes the failures.

This issue was addressed in NetWorker 19.9.0.0; however NetWorker 19.9.x is now End of Support Life (EOSL). If the NetWorker server is 19.8.x or earlier, upgrade to a supported release.

NetWorker software is available for download at the Dell NetWorker product page. 

Workaround:

Modify the /usr/sbin/lcmap file on each node to use the correct command syntax.

1. Open the lcmap file: sudo vi /usr/sbin/lcmap

a. Modify the syntax of the GRPS line:

GRPS=`pcs resource show 2>&1 | egrep "^\s*Resource Group:" | awk '{print $3}'`

with

GRPS=`pcs resource group list 2>&1 | awk '{print $1}' | tr -d ":"`

b. Replace each entry that says pcs resource show with pcs resource config. This appears on six different lines in the file:

Replace:

RESOURCES=`pcs resource show ${grp} | egrep "^\s*Resource:" | awk '{print $2}'`
      TMP=`pcs resource show ${res} | egrep "^\s*Attributes:.*ip="`
      TMP=`pcs resource show ${res} | egrep "^\s*Attributes:.*device="`
                        
RESOURCES=`pcs resource show ${Linkedres} | egrep "^\s*Resource:" | awk '{print $2}'`
      TMP=`pcs resource show ${res} | egrep "^\s*Attributes:.*ip="`
      TMP=`pcs resource show ${res} | egrep "^\s*Attributes:.*device="`

With:

RESOURCES=`pcs resource config ${grp} | egrep "^\s*Resource:" | awk '{print $2}'`
      TMP=`pcs resource config ${res} | egrep "^\s*Attributes:.*ip="`
      TMP=`pcs resource config ${res} | egrep "^\s*Attributes:.*device="`
                        
RESOURCES=`pcs resource config ${Linkedres} | egrep "^\s*Resource:" | awk '{print $2}'`
      TMP=`pcs resource config ${res} | egrep "^\s*Attributes:.*ip="`
      TMP=`pcs resource config ${res} | egrep "^\s*Attributes:.*device="`

2. Save the file.
3. Confirm that the shared location is returned by running: sudo lcmap

[root@NODE_HOSTNAME ~]# lcmap
type: NSR_CLU_TYPE;
clu_type: NSR_LC_TYPE;
interface version: 1.0;

type: NSR_CLU_VIRTHOST;
hostname: IP_ADDRESS;
local: TRUE;
owned paths: /nsr_share;<

4. Clean up the pcs resource: sudo pcs resource cleanup nws
5. Confirm all resources start up: sudo pcs resource status

[root@NODE_HOSTNAME ~]# pcs resource status
  * Resource Group: NW_group:
    * fs        (ocf::heartbeat:Filesystem):     Started NODE_HOSTNAME
    * ip        (ocf::heartbeat:IPaddr):         Started NODE_HOSTNAME
    * nws       (ocf::EMC_NetWorker:Server):     Started NODE_HOSTNAME

• NetWorker: Bootstrap Backup Fails on Clustered Linux Server After Upgrade "nsrauthcasm died with signal 13"