CSM: CSI Drivers: Panic in external-health-monitor-controller container causes CSI Driver Controller to restart

Summary: The external-health-monitor-controller container provided by Kubernetes can cause a panic and Container Storage Interface (CSI) Driver Controller pods to restart.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Issue seen with:
Dell CSI Drivers: v2.1.0 and v2.2.0
csi-external-health-monitor-controller: v0.4.0

The controller shows 12 restarts:

$ kubectl get pod -n isilon
 NAME                          READY   STATUS    RESTARTS        AGE
 isilon-controller-xxxx-xxxx   5/5     Running   12 (141m ago)   32h
...

 

Using the -p to get the previous logs, the csi-external-health-monitor-controller container shows the following panic:


E0629 18:48:41.494845       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 200 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x184d660, 0x27951e0)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86
panic(0x184d660, 0x27951e0)
    /usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/kubernetes-csi/external-health-monitor/pkg/controller.(*NodeWatcher).deleteNode(0xc000612500, 0xc002fa3b90, 0x28, 0x0)
    /workspace/pkg/controller/node_watcher.go:275 +0x29
github.com/kubernetes-csi/external-health-monitor/pkg/controller.(*NodeWatcher).WatchNodes.func1(0xb28012007187300)
    /workspace/pkg/controller/node_watcher.go:181 +0x647
github.com/kubernetes-csi/external-health-monitor/pkg/controller.(*NodeWatcher).WatchNodes(0xc000612500)
    /workspace/pkg/controller/node_watcher.go:185 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000499d90)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000499d90, 0x1c820c0, 0xc002b10450, 0x1, 0xc000126060)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000499d90, 0xdf8475800, 0x0, 0x1, 0xc000126060)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000499d90, 0xdf8475800, 0xc000126060)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/kubernetes-csi/external-health-monitor/pkg/controller.(*NodeWatcher).Run
    /workspace/pkg/controller/node_watcher.go:147 +0x194
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x168e749]

goroutine 200 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x109
panic(0x184d660, 0x27951e0)
    /usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/kubernetes-csi/external-health-monitor/pkg/controller.(*NodeWatcher).deleteNode(0xc000612500, 0xc002fa3b90, 0x28, 0x0)
    /workspace/pkg/controller/node_watcher.go:275 +0x29
github.com/kubernetes-csi/external-health-monitor/pkg/controller.(*NodeWatcher).WatchNodes.func1(0xb28012007187300)
    /workspace/pkg/controller/node_watcher.go:181 +0x647
github.com/kubernetes-csi/external-health-monitor/pkg/controller.(*NodeWatcher).WatchNodes(0xc000612500)
    /workspace/pkg/controller/node_watcher.go:185 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000499d90)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000499d90, 0x1c820c0, 0xc002b10450, 0x1, 0xc000126060)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000499d90, 0xdf8475800, 0x0, 0x1, 0xc000126060)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000499d90, 0xdf8475800, 0xc000126060)
    /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/kubernetes-csi/external-health-monitor/pkg/controller.(*NodeWatcher).Run
    /workspace/pkg/controller/node_watcher.go:147 +0x194 

Cause

This is a known issue in the external-health-monitor sidecar version 0.4.0 that Kubernetes provides that CSI Driver v2.1 and v2.2 uses:
https://github.com/kubernetes-csi/external-health-monitor/issues/100 Third party link
https://github.com/kubernetes-csi/external-health-monitor/pull/101 Third party link

This issue was fixed and merged into the next version of the external-health-monitor sidecar version 0.5.0. This was released on March 4, 2022.

Resolution

Resolution:
Upgrade to CSI Driver version 2.3.0 or greater that uses external-health-monitor sidecar version 0.5.0 per the Dell Technologies Container Storage Modules documentation:
https://dell.github.io/csm-docs/ Third party link icon

 

Article Properties
Article Number: 000201147
Article Type: Solution
Last Modified: 30 Jan 2026
Version:  8
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.