How to implement SBD STONITH in Linux clusters

STONITH: "Shoot The Other Node In The Head" or "Shoot The Offending Node In The Head"
Or
STONITH: "Shoot The Other Member or Machine In The Head"

STONITH is a technique for fencing in computer clusters. Fencing is the isolation of a failed node so that it does not cause disruption to a computer cluster.

In SBD STONITH, the nodes in the Linux cluster keep each other updated by using the Heartbeat mechanism. If something goes wrong with a node in the cluster, a poison pill is written for that node to the shared storage device. The node has to eat (accept) the poison pill and terminate itself. After which a file system resource can be safely failed over to another node in the Linux cluster.

In a DELL environment, all the disks are multipathed. This includes the SBD STONITH device. Consequentially, when creating the SBD STONITH device, the timeouts SBD uses must be adjusted as the following can cause delays; Multiple Path I/O's (MPIO) path down detection, PowerPath path down detection or NMP path down detection. If you have multiple devices, transient timeouts of a single device do not negatively affect SBD. However, if they all go through the same FC switches, you must do this. The watchdog timeout is the time required to detect a path failure and to switch to another path. After the msgwait timeout, the message is assumed to have been delivered to the node. For multipath, this should be the time required for the multipathing to detect a path failure and switch to the next path, added to the time between two loops to access the SBD device (the default interval between two loops is 1 s).

The watchdog timeout must be shorter than the msgwait timeout - half the value is a good value to use.

The default value of the msgwait timeout is 10 s. If the msgwait timeout is not adjusted, this is too short to give time to a host which is having an issue on a path when accessing the SBD device to detect that the path is bad and to failover to another path. If the host cannot update the watchdog timer fast enough, a poison pill is written to the host. This causes the host to commit suicide when it can access the SBD device again, which is not the purpose of the STONITH mechanism.

You would set these values by adding -4 msgwait -1 watchdogtimeout to the create command.
Here is an example for an ESX environment that is using NMP:
/usr/sbin/sbd -d /dev/sbd -4 40 -1 20 create
Here is an example for a Linux environment using PowerPath or MPIO (with devloss tmo set to 30):
/usr/sbin/sbd -d /dev/sbd -4 72 -1 36 create
(All timeouts are in seconds.)

This issue can incur significant delays to fail over. For more information about SBD STONITH, check SUSE Storage Protection and SBD (External Link)