With the inclusion
of support for watchdog
can now perform the function of a watchdog daemon Linux. On Dell PowerEdge systems, this hardware could either be the chipset watchdog timer built into the platform’s chipset (like Intel ICH9) or Dell iDRAC’s IPMI compliant BMC watchdog timer.
Dell iDRAC provides Automated System Recovery
which, in addition to recovering from OS lock-ups, can capture a screen shot for analysis at a later time. It was necessary to install additional software
on the OS to enable this. With newer distributions supporting systemd, this feature will work with software available natively in a distribution, eliminating the need for add-on software.
It was however possible to use the watchdogd daemon on Linux, but there was a probability where the daemon itself could lock-up while the rest of the system was operational. systemd acts as the software watchdog for all system services and the BMC watchdog timer acts as the hardware watchdog for systemd itself. So if systemd is non-operational, there is good chance that the system is unusable in general. So we now have a more reliable method for all system services, the manager of the services (systemd) to be ‘watched’ by the BMC’s watchdog timer.
The glue between systemd and Dell iDRAC’s BMC watchdog is the ipmi_watchdog kernel module, which provides Linux watchdog API
access to the BMC watchdog via /dev/watchdog. Systemd uses this interface to kick the watchdog periodically.
Setting up systemd with ipmi_watchdog
Systemd can be configured to use iDRAC BMC watchdog with these steps (on Fedora 19):
- Since the system has two watchdog timers (chipset and BMC), we can use either of them. For this example, we will disable the chipset watchdog. The chipset watchdog can be disabled by setting the "OS Watchdog Timer" option in the System BIOS to "Disabled" (default).
- Arrive at a timeout value for the watchdog, say 180 seconds.
- Enable the ipmi_watchdog kernel module to load at system startup with the timeout from above:
- Method1: Create /etc/modules-load.d/ipmi_watchdog with the following content
- options ipmi_watchdog timeout=180
- blacklist iTCO_wdt # Optional. If chipset watchdog is not disabled in BIOS setup.
- Install OpenIPMI rpm
- $ sudo yum install OpenIPMI
- Set IPMI_WATCHDOG=yes and IPMI_WATCHDOG_OPTIONS with the timeout in /etc/sysconfig/ipmi.
- Enable the ipmi service to startup automatically
- $ sudo systemctl enable ipmi
- Enable systemd’s watchdog:
- Uncomment and set RuntimeWatchdogSec=180 in /etc/systemd/system.conf
- Restart systemd
- # systemctl daemon-reexec
Test if this works:
- Check if watchdog is active
- $ sudo journalctl |grep -i 'hardware watchdog' # should show that systemd is setup to use IPMI watchdog.
- $ sudo ipmitool mc watchdog get # check if the "Watchdog Timer Is: Started/Running".
- Test by simulating a Kernel Panic (do not do this on a production system). Ensure kdump is disabled.
- $ sudo echo c > /proc/sysrq-trigger
- After system reset, verify that a picture of the failure screen is available in the iDRAC
- Login to iDRAC web GUI
- Overview -> Server -> Troubleshooting -> Last Crash Screen.