CSI Driver Shows that Only One Controller is Running and the Others are in the CrashLoopBackoff State

Summary: In Container Storage Interface (CSI) Driver 2.9.0, when the driver is installed with multiple controller replicas, only one controller is running while the others are in the "CrashLoopBackoff" state. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

The user can install the driver but, if multiple controller replicas are deployed, only one replica is running while the others are not responding. 
level=info msg="Attempting driver connection at: unix:/var/run/csi/csi.sock"
level=debug msg="grpc.Dial returned context deadline exceeded"
level=error msg="Waiting on connection to driver csi.sock: context deadline exceeded"
level=debug msg="grpc.Dial returned context deadline exceeded"
level=error msg="Waiting on connection to driver csi.sock: context deadline exceeded"

Cause

A timeout added to the Kubernetes sidecars connections causes the leader election process in the non-leader driver controller to fail. Without the timeout, non-leader driver controllers attempt to establish a connection indefinitely, avoiding the unresponsive state, see https://github.com/kubernetes-csi/csi-lib-utils/commit/bbcd13252ca60814a2e5c31255db660de4bed957This hyperlink is taking you to a website outside of Dell Technologies.

This issue occurs in the Helm installation of the driver where the number of replicas for the controller is 2, by default.

Resolution

Workaround

Reduce the number of controller replicas to one in driver values file, set controller.controllerCount to 1.

See the Upgrade documentation appropriate for your driver type to upgrade the driver. This is not an "upgrade" in the traditional sense as this is more aligned to modifying the configuration.
https://dell.github.io/csm-docs/docs/supportmatrix/ This hyperlink is taking you to a website outside of Dell Technologies.

Resolution

Engineering has provided a patch for Dell Container Storage Interface Driver for all driver types, v2.9.1, where leader-election is disabled.

Article Properties
Article Number: 000221763
Article Type: Solution
Last Modified: 13 Feb 2026
Version:  5
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.