CSM object associated with a driver deployment stays in a failed state even when the deployment succeeds

Summary: In CSM Operator v1.4.1, the CSM object associated with a driver deployment stays in a failed state even when the deployment succeeds.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

The system is unable to correctly display the status of the driver deployment when pods are successful. 
[root@master-1-Zaglt7mQUY8Wg e2e]# k describe csm -n test-vxflexos test-vxflexos
Name: test-vxflexos
Namespace: test-vxflexos
…
Status:
Controller Status:
Available: 0
Desired: 1
Failed: 1
Node Status:
Available: 2
Desired: 2
Failed: 0
State: Failed
Events:
Type Reason Age From Message
Normal Updated 3m12s csm Object finalizer is added
Normal Completed 3m11s (x2 over 3m11s) csm install/update storage component: test-vxflexos completed OK
Normal Completed 3m11s csm Driver deployment running OK
Warning Updated 3m11s csm at 1707491918292836118 Pod error details error message for default-source-cluster
PodInitializing=
Warning Updated 3m11s csm at 1707491918387122708 Pod error details error message for default-source-cluster
PodInitializing=
Warning Updated 3m11s (x3 over 3m11s) csm Failed install: Operation cannot be fulfilled on containerstoragemodules.storage.dell.com "test-vxflexos": the object has been modified; please apply your changes to the latest version and try again
Warning Updated 3m11s csm at 1707491918502243242 Pod error details error message for default-source-cluster
PodInitializing=
Warning Updated 3m11s csm at 1707491918603289132 Pod error details error message for default-source-cluster
PodInitializing=
Warning Updated 3m11s csm at 1707491918633307657 Pod error details error message for default-source-cluster
PodInitializing=
Warning Updated 3m11s csm at 1707491918702738940 Pod error details error message for default-source-cluster
PodInitializing=
Warning Updated 3m10s csm at 1707491919356615788 Pod error details error message for default-source-cluster
PodInitializing=
Warning Updated 3m10s csm at 1707491919773768318 Pod error details error message for default-source-cluster
PodInitializing=
Warning Updated 3m9s (x3 over 3m9s) csm (combined from similar events): at 1707491920766553739 Pod error details error message for default-source-cluster
PodInitializing=
Normal Completed 3m8s csm at 1707491921372865165 Driver pods running OK
Normal Completed 3m8s csm Driver daemonset running OK

[root@master-1-Zaglt7mQUY8Wg e2e]# k get pods -n test-vxflexos
NAMESPACE NAME READY STATUS RESTARTS AGE
test-vxflexos test-vxflexos-controller-797f95f7c7-xfs7r 5/5 Running 0 3m24s
test-vxflexos test-vxflexos-node-js29j 2/2 Running 0 3m24s
test-vxflexos test-vxflexos-node-pg75s 2/2 Running 0 3m24s

[root@master-1-Zaglt7mQUY8Wg e2e]# kubectl get csm -A
NAMESPACE NAME CREATIONTIME CSIDRIVERTYPE CONFIGVERSION STATE
test-vxflexos test-vxflexos 4m19s powerflex v2.9.1 Failed

Cause

There are update handlers for the daemonset (node pods) and deployment (controller pods) objects. These handlers both try to update the status simultaneously. For example, if the controller pod went into Ready state while a node pod went from Init into ContainerCreating. If the deployment handler got the status update mutex first, it would write in its updated status. But when the daemonset handler got the mutex afterward, it would overwrite the controller status as failed.

Resolution

Resolution:
Engineering has resolved this issue in CSM Operator patch 1.4.2. The immediate fix has the update routine to double check the controller status before updating.

Products

Container Storage Modules Family, Container Storage Modules
Article Properties
Article Number: 000222163
Article Type: Solution
Last Modified: 03 سبتمبر 2025
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.