Isilon: OneFS-How to interpret watchdogs errors

Summary: The software watchdog is a process that monitors the kernel and prints a stack or reboots the node when the node is unresponsive. This protects the cluster from symptoms of major CPU-starvation and helps Dell Technical Support identify issues in order to correct them. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Instructions

Introduction

This Knowledge article describes how to read and interpret a stack created by the swatchdog process. The software watchdog is also known as swatchdog or softwatch.

 

Details

Sometimes a node writes a stack into the /var/log/messages file or reboots itself with an error similar to the following:

**********************************************
Software Watchdog failed (userspace is starved!)
**********************************************

**********************************************
Software Watchdog failed on CPU 0 (6353: kt: gmp-split [-])
0x80bda7b9 -> 0x80bda5dc (fp=0xf734bb78): lk_fail_create_entry_and_owner
0x80bbe950 -> 0x80bbe7e0 (fp=0xf734bbf0): lkf_group_change_save_locks
0x80aa251c -> 0x80aa2268 (fp=0xf734bc2c): rtxn_sync_locks_prepare
0x80aa447d -> 0x80aa4304 (fp=0xf734bcdc): rtxn_split
0x80aac9cf -> 0x80aac8ec (fp=0xf734bcfc): kt_main
0x802a9d43 -> 0x802a9ca8 (fp=0xf734bd14): fork_exit

intr counts:
irq3: 1382 irq4: 1164845 irq14: 19331 irq17: 10672321 irq18: 11 stray: 1 irq24: 22011026 irq48: 46902637
**********************************************

panic @ time 1257444527.664: Software watchdog timed out

Stack: -------------------------------------------------

0x802e24f0 -> 0x802e24e4 (fp=0xf734ba78): isi_swatchdog_panic
0x802e27d7 -> 0x802e26ac (fp=0xf734ba8c): isi_swatchdog_hardclock
0x80295187 -> 0x80295068 (fp=0xf734bab0): hardclock_process
0x802951ba -> 0x802951a8 (fp=0xf734bac4): hardclock
0x8041d608 -> 0x8041d5b8 (fp=0xf734bad4): lapic_handle_timer
0x804281c3 -> 0x804281a4 (fp=0xf734bb78): bcmp
0x80bbe950 -> 0x80bbe7e0 (fp=0xf734bbf0): lkf_group_change_save_locks
0x80aa251c -> 0x80aa2268 (fp=0xf734bc2c): rtxn_sync_locks_prepare
0x80aa447d -> 0x80aa4304 (fp=0xf734bcdc): rtxn_split
0x80aac9cf -> 0x80aac8ec (fp=0xf734bcfc): kt_main
0x802a9d43 -> 0x802a9ca8 (fp=0xf734bd14): fork_exit

---------------------------------------------------------

The swatchdog is built as follows:

  • Low-level timer interrupt triggers every 10 s.
  • High-level user-space code tries to set a mailbox for the timer interrupt every 5 s.

Action is taken when the low-level timer interrupt fails to find a mailbox note from user-space, then the stack is dumped. After four failures in a row, the cluster is rebooted.
For assistance interpreting an error stack or swatchdog triggered reboot, contact Dell Technical Support.

Affected Products

Isilon

Products

Isilon, PowerScale OneFS
Article Properties
Article Number: 000018976
Article Type: How To
Last Modified: 10 Jun 2025
Version:  6
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.