Dell CSI not removing globalmount and multipath devices when kubelet is restarted with delay

Environment Details:-

Master node =1

Workers =2

Host OS = SLES 15 SP2

Kubernetes version = v1.21.1

Dell Unity CSI Version = v1.6

To reproduce the issue , follow the steps:-

1. Create a PVC with RWO (Block) using the Dell Unity CSI. The FS created is EXT4 .
2. Cordon 1 worker node and leave the second one uncordon, so that all the PODs get scheduled on the same node. (OR use the POD Affinity to make sure they land on the same node)
3. Create two different PODs (using Deployment Controller) using the same PVC. Wait for both pods to come into “Running” state. After running state we can see that there is one “globalmount” and 2 bind mounts exists on the worker node.
4. There is a one volumeAttachment object created for the same PV and the worker node where PODs got scheduled.
5. Now stop the “kubelet” on the same worker node.
6. After few seconds , worker node becomes “NotReady”. And after a while both PODs gets into Terminating state.
7. Delete the PODs forcefully to clear the Terminating PODs.
8. Delete the volumeAttachment object.
9. Start the “Kubelet”
10. PODs gets scheduled on the same node.
11. New VolumeAttachment object gets created for the same worker node.
12. PODs fails to start with EXT4 FS corruption. Logs of corruption exits in the “/var/log/messages” file.
13. Below error is observed:

MountVolume.MountDevice failed for volume "csivol-4562e8f2a0" : kubernetes.io/csi: attacher.MountDevice failed to create dir "/var/lib/kubelet/plugins/k
ubernetes.io/csi/pv/csivol-4562e8f2a0/globalmount": mkdir /var/lib/kubelet/plugins/kubernetes.io/csi/pv/csivol-4562e8f2a0/globalmount: file exists

Also, we notice a number of failed multipath devices on the node even though volumeattachment is deleted.

Observation: - Kubelet on startup only clears one bind mount and unable to clear the other bind mount and “globalmount” point. Once volumeAttachment becomes “true” the “globalmount” point gets into “RO” mode.

Note:- We delete volumeAttachment because, we are singleton application and want to move application PODs to the other worker node as quickly as possible in the event of worker node getting into “NotReady” state. When moving to the other node is successful but coming back to the first node again causes FS corruption due to dangling mount points.