NetWorker: nws resource on Red Hat pacemaker cluster fails to start "nsrd NSR critical Cannot start nsrd"

• The /nsr_share/nsr/logs/daemon.raw logs the following error during service startup:

73248 01/31/2023 12:57:48 PM  5 5 0 926312256 966299 0 nwrhelnodef.emclab.local nsrd NSR critical Can't start nsrd because /nsr/res (/nsr) is local, and NetWorker is configured as a cluster server. Use cluster manager to check NetWorker service status.
144354 01/31/2023 12:57:48 PM  1 5 0 130900160 963225 0 nwrhelnodef.emclab.local nsrctld NSR notice Daemon nsrd terminated.

• The node can see the cluster resources with: lcmap

root@NWrhelNodeF:~# lcmap
type: NSR_CLU_TYPE;
clu_type: NSR_LC_TYPE;
interface version: 1.0;

type: NSR_CLU_VIRTHOST;
hostname: 192.168.25.28;
local: FALSE;
owned paths: /nsr_share;

• The lcmap output matches the pcs resource configuration: pcs resource config

root@NWrhelNodeF:~# pcs resource config
 Group: NW_group
  Resource: fs (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/sdb1 directory=/nsr_share fstype=ext4
   Operations: monitor interval=20 timeout=300 (fs-monitor-interval-20)
               start interval=0s timeout=60s (fs-start-interval-0s)
               stop interval=0s timeout=60s (fs-stop-interval-0s)
  Resource: ip (class=ocf provider=heartbeat type=IPaddr)
   Attributes: cidr_netmask=24 ip=192.168.25.28 nic=ens192
   Operations: monitor interval=15 timeout=120 (ip-monitor-interval-15)
               start interval=0s timeout=20s (ip-start-interval-0s)
               stop interval=0s timeout=20s (ip-stop-interval-0s)
  Resource: nws (class=ocf provider=EMC_NetWorker type=Server)
   Meta Attrs: is-managed=true
   Operations: meta-data interval=0 timeout=10 (nws-meta-data-interval-0)
               migrate_from interval=0 timeout=120 (nws-migrate_from-interval-0)
               migrate_to interval=0 timeout=60 (nws-migrate_to-interval-0)
               monitor interval=100 timeout=1000 (nws-monitor-interval-100)
               start interval=0 timeout=300 (nws-start-interval-0)
               stop interval=0 timeout=300 (nws-stop-interval-0)

The IP address is not resolving to the name used by the NetWorker cluster configuration:

root@NWrhelNodeF:~# nslookup 192.168.25.28
** server can't find 28.25.168.192.in-addr.arpa: NXDOMAIN

The IP should resolve to the NSR_SERVERHOST value in /usr/lib/ocf/resource.d/EMC_NetWorker/Server file:

root@NWrhelNodeF:~# cat /usr/lib/ocf/resource.d/EMC_NetWorker/Server | grep SERVERHOST
                    echo "q" | nsradmin -s ${NSR_SERVERHOST} -i - > /dev/null 2>&1
                        echo "q" | nsradmin -s ${NSR_SERVERHOST} -i - > /dev/null 2>&1
NSR_SERVERHOST=NWrhelClus.amer.lan

This value is set when running the /usr/sbin/networker.cluster script.

Fix name resolution for the VIP. This can be corrected by AD admin in Domain Name System (DNS) configuration or by using /etc/hosts file entries on each node involved in the cluster.

root@NWrhelNodeF:~# nslookup 192.168.25.28
28.25.168.192.in-addr.arpa      name = NWrhelClus.amer.lan

Once the name is resolved correctly, NetWorker services can be started:

root@NWrhelNodeF:~# pcs resource cleanup nws
Cleaned up fs on NWrhelNodeF.amer.lan
Cleaned up fs on NWrhelNodeE.amer.lan
Cleaned up ip on NWrhelNodeF.amer.lan
Cleaned up ip on NWrhelNodeE.amer.lan
Cleaned up nws on NWrhelNodeF.amer.lan
Cleaned up nws on NWrhelNodeE.amer.lan

root@NWrhelNodeF:~# pcs resource
  * Resource Group: NW_group:
    * fs        (ocf::heartbeat:Filesystem):     Started NWrhelNodeF.amer.lan
    * ip        (ocf::heartbeat:IPaddr):         Started NWrhelNodeF.amer.lan
    * nws       (ocf::EMC_NetWorker:Server):     Started NWrhelNodeF.amer.lan

If lcmap does not return the hostname, or owned path values see: 

• NetWorker: Server is configured on Red Hat 8.x failover cluster fails to start
• NetWorker Server's or Client's service disruption in HA (High Availability) cluster nodes that are installed with Red Hat 8.2, 8.3, 8.4, 8.6 operating system versions.