Using Systemd for Automated System Recovery

Using Systemd for Automated System Recovery

With the inclusion of support for watchdog hardware, systemd can now perform the function of a watchdog daemon Linux. On Dell PowerEdge systems, this hardware could either be the chipset watchdog timer built into the platform’s chipset (like Intel ICH9) or Dell iDRAC’s IPMI compliant BMC watchdog timer.

Dell iDRAC provides Automated System Recovery which, in addition to recovering from OS lock-ups, can capture a screen shot for analysis at a later time. It was necessary to install additional software on the OS to enable this. With newer distributions supporting systemd, this feature will work with software available natively in a distribution, eliminating the need for add-on software.

It was however possible to use the watchdogd daemon on Linux, but there was a probability where the daemon itself could lock-up while the rest of the system was operational. systemd acts as the software watchdog for all system services and the BMC watchdog timer acts as the hardware watchdog for systemd itself. So if systemd is non-operational, there is good chance that the system is unusable in general. So we now have a more reliable method for all system services, the manager of the services (systemd) to be ‘watched’ by the BMC’s watchdog timer.

The glue between systemd and Dell iDRAC’s BMC watchdog is the ipmi_watchdog kernel module, which provides Linux watchdog API access to the BMC watchdog via /dev/watchdog. Systemd uses this interface to kick the watchdog periodically.

Setting up systemd with ipmi_watchdog

Systemd can be configured to use iDRAC BMC watchdog with these steps (on Fedora 19):
  1. Since the system has two watchdog timers (chipset and BMC), we can use either of them. For this example, we will disable the chipset watchdog. The chipset watchdog can be disabled by setting the "OS Watchdog Timer" option in the System BIOS to "Disabled" (default).
  2. Arrive at a timeout value for the watchdog, say 180 seconds.
  3. Enable the ipmi_watchdog kernel module to load at system startup with the timeout from above:
  • Method1: Create /etc/modules-load.d/ipmi_watchdog with the following content
    • options ipmi_watchdog timeout=180
    • blacklist iTCO_wdt # Optional. If chipset watchdog is not disabled in BIOS setup.
  • Method2:
    • Install OpenIPMI rpm
      • $ sudo yum install OpenIPMI
    • Set IPMI_WATCHDOG=yes and IPMI_WATCHDOG_OPTIONS with the timeout in /etc/sysconfig/ipmi.
    • Enable the ipmi service to startup automatically
      • $ sudo systemctl enable ipmi
  • Enable systemd’s watchdog:
    • Uncomment and set RuntimeWatchdogSec=180 in /etc/systemd/system.conf
  • Restart systemd
    • # systemctl daemon-reexec

Test if this works:

  1. Check if watchdog is active
    • $ sudo journalctl |grep -i 'hardware watchdog' # should show that systemd is setup to use IPMI watchdog.
    • $ sudo ipmitool mc watchdog get # check if the "Watchdog Timer Is: Started/Running".
  2. Test by simulating a Kernel Panic (do not do this on a production system). Ensure kdump is disabled.
    • $ sudo echo c > /proc/sysrq-trigger
  3. After system reset, verify that a picture of the failure screen is available in the iDRAC
    • Login to iDRAC web GUI
    • Overview -> Server -> Troubleshooting -> Last Crash Screen.
Note: that this feature is unsupported by Dell at this time and is shared here with the intent of soliciting feedback from the community at

Article ID: SLN312339

Last Date Modified: 08/17/2018 04:11 AM

Rate this article

Easy to understand
Was this article helpful?
Yes No
Send us feedback
Comments cannot contain these special characters: <>()\
Sorry, our feedback system is currently down. Please try again later.

Thank you for your feedback.