NetWorker: NetWorker server deployed on RHEL Pacemaker failover cluster has no method to tune the monitoring interval

• The NetWorker server is installed on a Red Hat Enterprise Linux (RHEL) Pacemaker (pcs) failover cluster.
• There are intermittent outages in NetWorker due to brief interruptions to the Pacemaker (pcs) resource for NetWorker server (default is nws).

The cause of cluster outages can vary. This section defines what is used to perform NetWorker cluster monitor functions. 

By default the Pacemaker resource for NetWorker has a "monitor" operation. The operation has "interval" and "timeout" settings which are configured by the cluster administrator during initial NetWorker cluster configuration.

[root@rhel-node02 ~]# pcs resource
  * Resource Group: NW_group:
    * fs        (ocf:heartbeat:Filesystem):      Started rhel-node02.amer.lan
    * ip        (ocf:heartbeat:IPaddr2):         Started rhel-node02.amer.lan
    * nws       (ocf:EMC_NetWorker:Server):      Started rhel-node02.amer.lan
[root@rhel-node02 ~]# pcs resource config nws
Resource: nws (class=ocf provider=EMC_NetWorker type=Server)
  Meta Attributes: nws-meta_attributes
    is-managed=true
  Operations:
    meta-data: nws-meta-data-interval-0
      interval=0 timeout=10
    migrate_from: nws-migrate_from-interval-0
      interval=0 timeout=120
    migrate_to: nws-migrate_to-interval-0
      interval=0 timeout=120
    monitor: nws-monitor-interval-100
      interval=100 timeout=600
    start: nws-start-interval-0
      interval=0 timeout=900
    stop: nws-stop-interval-0
      interval=0 timeout=300
    validate-all: nws-validate-all-interval-0
      interval=0 timeout=10

NetWorker is configured to use Open Cluster Framework (OCF). The monitoring function is defined in /usr/lib/ocf/resource.d/EMC_NetWorker/Server:

NWServer_monitor() {
        local count

        # exit immediately if configuration is not valid
        NWServer_validate_all || exit $?

        quick_monitor
        if [ $? -eq 0 ]; then
                count=0
                while [ $count -lt 3 ]; do
                        echo "q" | nsradmin -s ${NSR_SERVERHOST} -i - > /dev/null 2>&1
                        if [ $? -eq 0 ]; then
                                return $OCF_SUCCESS
                        else
                                count=`expr ${count} + 1`
                                sleep 1
                        fi
                done
        else
                return $OCF_NOT_RUNNING
        fi

        return $OCF_NOT_RUNNING
}

Monitor failure should technically never occur under normal circumstances and indicates an unrecoverable error. However, some large environments may observe intermittent issues where nsradmin can fail on monitor test of a Pacemaker even if only temporarily, and that results in the Pacemaker taking a full outage.

The cluster administrator should investigate all cluster outage issues. The cluster logs can be reviewed for any details about interruptions:

• /var/log/pcsd/pcsd.log
• /var/log/pacemaker/pacemaker.log
• /var/log/messages

NetWorker server logs can also be reviewed. The NetWorker server's daemon.log is located on the shared disk (for example, /nsr_share).

• /nsr_share/nsr/logs/daemon.log

If real-time rendering is not enabled, the .raw log can be rendered into an .log file with the following command:

nsr_render_log /nsr_share/nsr/logs/daemon.raw > /nsr_share/nsr/logs/daemon_`date -I`.log

The cluster administrator can increase the monitor interval and timeout values for the NetWorker server pcs resource. See Red Hat Pacemaker documentation for directions about changing the timeout values as Pacemaker commands may change across Pacemaker versions.

Alternatively, modify the NWServer_monitor retry account in the /usr/lib/ocf/resource.d/EMC_NetWorker/Server file.

1. Stop NetWorker services: pcs resource disable nws
2. On each node make a copy of the original /usr/lib/ocf/resource.d/EMC_NetWorker/Server file: cp /usr/lib/ocf/resource.d/EMC_NetWorker/Server /usr/lib/ocf/resource.d/EMC_NetWorker/Server.orig
3. Edit the file in a text editor: vi /usr/lib/ocf/resource.d/EMC_NetWorker/Server
4. Replace the NWServer_monitor block with:

NWServer_monitor() {
    local retries=10
    local attempt

    # exit immediately if configuration is not valid
        NWServer_validate_all || exit $?

    quick_monitor || return $OCF_NOT_RUNNING

    for ((attempt=1; attempt<=retries; attempt++)); do
        if echo "q" | nsradmin -s "${NSR_SERVERHOST}" -i - >/dev/null 2>&1; then
            return $OCF_SUCCESS
        fi

        # Optional log to journalctl (/var/log/messages)
        logger -t NWServer_monitor "nsradmin check failed (attempt ${attempt}/${retries})"
        sleep 1
    done

    return $OCF_NOT_RUNNING
}

In the above example, the retry count is set to 10. This count affects how long the monitor is willing to retry and wait before returning before returning OCF_NOT_RUNNING and considering NetWorker in a failed state. In some scenarios (particularly busy servers), the server may momentarily fail to respond to nsradmin query but return shortly after. Table 1 contains more information.

Table 1. Query Retry Times

The Total Wait Time does not factor in the execution time of each nsradmin command. The retry count you choose to use is dependent on your environment and real vs. transient communication issues that do not require a failover to occur. You must coordinate with the cluster manager and carefully consider tuning the NWServer_monitor function versus investigating other underlying communication issues.

5. Save the file.
6. Start NetWorker services: pcs resource enable nws

If NWServer_monitor fails, the following is logged by journactl:

[root@rhel-node02 ~]# journalctl -b | grep -i NWServer
Jul 28 10:12:41 rhel-node02 NWServer_monitor[2871414]: nsradmin check failed (attempt 1/10)
Jul 28 10:12:52 rhel-node02 NWServer_monitor[2871722]: nsradmin check failed (attempt 2/10)
Jul 28 10:13:04 rhel-node02 NWServer_monitor[2871917]: nsradmin check failed (attempt 3/10)
Jul 28 10:13:15 rhel-node02 NWServer_monitor[2872161]: nsradmin check failed (attempt 4/10)

These errors are also logged to /var/log/messages on the node.

If any issues are observed, revert to the default Server file.