NetWorker: Red Hat Pacemaker Cluster How to Upgrade NetWorker Server and Best Practices

Prerequisites:

1. Complete a Server Protection backup job from the NetWorker Management Console (NMC) or NetWorker Web User Interface (NWUI). This ensures that the NetWorker server resource database (nsrdb), media database (mm), and client file indexes (index) are backed up and available in the event a disaster recovery is needed. Disaster recovery events are not expected; however, preparing for one is always recommended.
2. Collect bootstrap backup details and make note of them in another location:

mminfo -B

3. Confirm the shared storage location and logical hostname used by the cluster:

For NetWorker 19.11 and later:

1. 1. Run the command: lcmap
   2. The "hostname" and shared "owned" path identify the logical cluster name and shared storage mountpoint. 

Example:

NWrhelNode1:~# lcmap
type: NSR_CLU_TYPE;
clu_type: NSR_LC_TYPE;
interface version: 1.0;

type: NSR_CLU_VIRTHOST;
hostname: NWrhelClusA.emclab.local;
local: TRUE;
owned paths: /nsr_share;

clu_nodes: NWrhelNodeA.emclab.local NWrhelNodeB.emclab.local;

For NetWorker 19.10 and earlier:

1. 1. Use the lcmap command to get the IP address of the logical cluster resource:

NWrhelNode1:~# lcmap
type: NSR_CLU_TYPE;
clu_type: NSR_LC_TYPE;
interface version: 1.0;

type: NSR_CLU_VIRTHOST;
hostname: 192.168.25.26;
local: TRUE;
owned paths: /nsr_share;

clu_nodes: NWrhelNodeA.emclab.local NWrhelNodeB.emclab.local;

b: Use the nslookup command to confirm the logical cluster name associated with the cluster's ip resource:

root@NWrhelNode1:~# nslookup 192.168.25.26
20.25.168.192.in-addr.arpa      name = NWrhelClusA.emclab.local.

4. Download the NetWorker Linux installation package from: 

https://www.dell.com/support/home/product-support/product/networker/drivers

5. Copy the installation package to each of the nodes in the cluster. For more information, see: 

NetWorker (Linux): How to copy files to/from a Linux NetWorker server.

6. Extract the installation package to a location of your choosing: 

tar xvf nwVERSION_linux_x86_64.tar.gz

Upgrade Procedure:

On one of the nodes, perform the following:

1. Stop NetWorker server services:

pcs resource disable nws

2. Wait for the NetWorker resource (nws) to show as "stopped (disabled)"

watch -n 5 pcs resource

On each of the nodes in the cluster, perform the following.

3. Stop the client service:

nsr_shutdown

4. Get a list of the installed NetWorker packages.

rpm -qa | grep lgto

5. Change directories into the extracted installation package (see Prerequisite step 5):

cd linux_x86_64

6. Upgrade the existing NetWorker packages with the new installation packages:

rpm -Uvh PACKAGE_1 PACKAGE_2 PACKAGE_3 ...

7. Run the authentication service configuration script:

/opt/nsr/authc-server/scripts/authc_configure.sh

Reconfigure the cluster to use any changes in the Server Open Cluster Framework (OCF) file or lcmap settings.

8. On each node, run:

pcs cluster stop --force

9. On each node, run the following command to remove the cluster configuration files (these settings are readded later)

/usr/sbin/networker.cluster -r

10. Readd the cluster configuration:

/usr/sbin/networker.cluster

Example:

[root@NWrhelNodeA: ~]# /usr/sbin/networker.cluster
Pacemaker (a CRM solution)  is detected.
Would you like to configure NetWorker for it [Yes]?
-----------------------------------------------------------------
Pacemaker is a high availability product for LINUX.
It allows administrators to connect separate systems together to
create failover clusters that ensure application availability and
data integrity under several failure conditions. Administrators
can use OS's Cluster Manager with database applications, file
sharing services, web servers, and more.
-----------------------------------------------------------------
Do you wish to continue? [Yes]?
Shutting down NetWorker services...

Enter directory where local NetWorker database is installed [/nsr]?

Do you wish to configure for both NetWorker server and client?
Yes or No [Yes]?

Creating /usr/lib/ocf/resource.d/EMC_NetWorker directory ...

The mountpoint of the shared NetWorker directory (the nsr
directory) must be the same for all the nodes where NetWorker
server can failover to.

In what path will the shared nsr directory be created/located? /nsr_share

Enter the Logical Hostname to be used for NetWorker? NWrhelClusA.emclab.local

Creating /usr/lib/ocf/resource.d/EMC_NetWorker/Server file ...
-rwxr-xr-x. 1 root root 5689 Oct  1 15:41 /usr/lib/ocf/resource.d/EMC_NetWorker/Server
Finished creating /usr/lib/ocf/resource.d/EMC_NetWorker/Server file ...

Creating /usr/sbin/lcmap file ...
-rwxrwxr-x. 1 root root 9259 Oct  1 15:41 /usr/sbin/lcmap
Finished creating /usr/sbin/lcmap file.


NetWorker has been successfully cluster-configured.

11. Start the cluster services:

pcs cluster start --all

12. On each node, ensure that the client service is started:

ps -ef | grep nsrexecd

If the client is not started, start it by running:

/usr/sbin/nsrexecd

On one of the nodes, perform the following:

13. Re-enable the NetWorker resource, this process starts up the NetWorker server services:

pcs resource enable nws

14. Wait for the NetWorker resource to show as "started":

watch -n 5 pcs resource

The cluster NetWorker server has been upgraded.

For any issues during server startup see:
NetWorker: Troubleshooting Guide for Red Hat Cluster Service Issue

If the server package fails to install:

   5:lgtoserv-19.11.0.1-1             error: failed to open dir nsr of /nsr/lic/: File exists
################################# [ 42%]
error: unpacking of archive failed on file /nsr/lic/EMCLicenseFilev7.xsd;66fc3bd7: cpio: open failed - No such file or directory
error: lgtoserv-19.11.0.1-1.x86_64: install failed
   6:lgtoman-19.11.0.1-1              ################################# [ 50%]
error: lgtoserv-19.10.0.5-1.x86_64: erase skipped

With the nws pcs resource disabled, the NetWorker server directory is not mounted. The upgrade cannot find the /nsr/lic file path.

Perform the following on one of the nodes.
 

1. Mount the shared /nsr partition to the cluster shared disk:

mount /dev/sdb1 /nsr_share

2. The mountpoint should show the /nsr server contents:

[root@NWrhelNodeA ~]# ls -l /nsr_share
total 0
drwxr-xr-x. 17 root root 194 Sep 30 15:24 nsr
[root@NWrhelNodeA ~]# ls -l /nsr_share/nsr/
total 12
drwxrwxrwt.  2 root      root         6 Sep 30 15:22 applogs
drwxr-xr-x. 11 nsrtomcat nsrtomcat  117 Oct  1 13:55 authc
drwxr-x--t.  3 root      root        24 Sep 30 15:22 cache
drwx------. 27 root      root      4096 Oct  1 13:55 cores
drwxr-xr-x.  2 root      root         6 Sep 30 15:22 debug
drwxr-xr-x.  8 root      root       170 Sep 30 15:45 index
drwxr-x--x.  2 root      root        32 Sep 30 15:23 lic
drwx------.  3 root      root        34 Sep 30 15:23 lockbox
drwxr-x--x.  8 root      root      4096 Oct  1 12:03 logs
drwxr-xr-x.  3 root      root        83 Oct  1 10:00 mm
drwxr-x--x.  2 root      root        28 Sep 30 15:23 rabbitmq
drwxr-xr-x.  6 root      root        78 Oct  1 10:00 res
drwxr-x--x.  2 root      root        18 Oct  1 13:56 run
drw-------.  3 root      root        23 Sep 30 15:24 sec
drwxr-xr-x.  7 root      root      4096 Oct  1 13:56 tmp

3. Update the NetWorker server package.

rpm -Uvh lgtoserv_package

4. Unmount the shared /nsr partition:

umount /nsr_share

5. Return to step 7 of the upgrade procedure.