Containers

[Nishit1]

Environment Details:-

Master node =1

Workers =2

Host OS = SLES 15 SP2

Kubernetes version = v1.21.1

Dell Unity CSI Version = v1.6

To reproduce the issue , follow the steps:-

1. Create a PVC with RWO (Block) using the Dell Unity CSI. The FS created is EXT4 .
2. Cordon 1 worker node and leave the second one uncordon, so that all the PODs get scheduled on the same node. (OR use the POD Affinity to make sure they land on the same node)
3. Create two different PODs (using Deployment Controller) using the same PVC. Wait for both pods to come into “Running” state. After running state we can see that there is one “globalmount” and 2 bind mounts exists on the worker node.
4. There is a one volumeAttachment object created for the same PV and the worker node where PODs got scheduled.
5. Now stop the “kubelet” on the same worker node.
6. After few seconds , worker node becomes “NotReady”. And after a while both PODs gets into Terminating state.
7. Delete the PODs forcefully to clear the Terminating PODs.
8. Delete the volumeAttachment object.
9. Start the “Kubelet”
10. PODs gets scheduled on the same node.
11. New VolumeAttachment object gets created for the same worker node.
12. PODs fails to start with EXT4 FS corruption. Logs of corruption exits in the “/var/log/messages” file.
13. Below error is observed:

MountVolume.MountDevice failed for volume "csivol-4562e8f2a0" : kubernetes.io/csi: attacher.MountDevice failed to create dir "/var/lib/kubelet/plugins/k
ubernetes.io/csi/pv/csivol-4562e8f2a0/globalmount": mkdir /var/lib/kubelet/plugins/kubernetes.io/csi/pv/csivol-4562e8f2a0/globalmount: file exists

Also, we notice a number of failed multipath devices on the node even though volumeattachment is deleted.

Observation: - Kubelet on startup only clears one bind mount and unable to clear the other bind mount and “globalmount” point. Once volumeAttachment becomes “true” the “globalmount” point gets into “RO” mode.

Note:- We delete volumeAttachment because, we are singleton application and want to move application PODs to the other worker node as quickly as possible in the event of worker node getting into “NotReady” state. When moving to the other node is successful but coming back to the first node again causes FS corruption due to dangling mount points.

[Flo_csI]

Hi,

Under normal operation, I did not reproduce the issue on Unity iSCSI.

That is to say, I executed the exact same steps as you have from 1 to 10 without an error.

On step 7. I run kubectl delete pod [podname] --grace-period=0 --force as per : https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/

On step 8., I did nothing as Kubernetes scheduler takes care of it after 6 minutes : https://github.com/kubernetes/kubernetes/blob/5e31799701123c50025567b8534e1a62dbc0e9f6/pkg/controlle...

Have you try to not manually delete the volumeAttachment and let Kubernetes do ?

Are you running the Unity driver on top of FiberChannel, iSCSI or NFS ?

Rgds.

[Nishit1]

Hi,

We have tried both, manual deletion and kubernetes deletion as well. However, we noticed that our application continues to do I/O even when kubelet is down, which I suppose is normal behavior since pods on the failed node continue to run. The application dies with I/O error once the volumeattachment is deleted.

So I guess to reproduce the exact scenerio, an I/O workload might be needed inside the pod. 

Regards,

[Nishit1]

@Flo_csI  : and we are using unity ontop of Fiber channel.

Regards,

[rbo54]

A question to clarify:  You are not using our CSM-Resiliency product in this test (with the podmon container)? You do not mention it.

[Nishit1]

No, we are not using CSM Resiliency product in this test.