Solved!
Go to Solutioncsi-powerstore cannot support RollingUpdate ?
when i change a statefulset yaml and submit , pods start rolling update. if one of pods container terminated and recreate in the same node, this container cannot run.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
serviceName: "nginx"
replicas: 3 # by default is 1
template:
metadata:
labels:
app: nginx # has to match .spec.selector.matchLabels
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx
resources:
limits:
cpu: 2
memory: 16000Mi
requests:
cpu: 2
memory: 8000Mi
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "powerstore"
resources:
requests:
storage: 10Gi
edit cpu: 2 to cpu: 4
kubectl apply -f nginx.yaml
kubectl describe po nginx-2
Warning FailedMount 2m6s kubelet Unable to attach or mount volumes: unmounted volumes=[my-persistent-storage], unattached volumes=[my-persistent-storage default-token-2mzjs]: timed out waiting for the condition
Warning FailedMount 2s (x9 over 2m10s) kubelet MountVolume.SetUp failed for volume "csi-pstore-60831e62c4" : rpc error: code = Internal desc = error while trying to detect fs for staging path /var/lib/kubelet/plugins/kubernetes.io/csi/pv/csi-pstore-60831e62c4/globalmount/c846493b-108c-45d7-8e80-a97668b34415: exit status 32
kubernetes version v1.20.9
csi-powerstore version v1.4
node:
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.9.2009 (Core)
Release: 7.9.2009
Codename: Core
We created a StorageClass with powerstore-ext4.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: powerstore
provisioner: csi-powerstore.dellemc.com
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate
parameters:
arrayID: "0"and just use volumeClaimTemplates to mount the storage volume, without any mount options.
We checked the powerstore-node log and found an abnormal problem
{"ID":"9c30da54-7bd7-4573-8db2-ee1a83500b83","StagingPath":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/csi-pstore-2188ee9d93/globalmount/9c30da54-7bd7-4573-8db2-ee1a83500b83","level":"info","msg":"multipath command: chroot args: /noderoot multipath -f /dev/dm-0","time":"2021-08-18T04:59:43.793728527Z"}
Stuck here for 120 seconds
{"ID":"9c30da54-7bd7-4573-8db2-ee1a83500b83","StagingPath":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/csi-pstore-2188ee9d93/globalmount/9c30da54-7bd7-4573-8db2-ee1a83500b83","level":"debug","msg":"multipath command output: ","time":"2021-08-18T05:01:43.794298901Z"}
{"ID":"9c30da54-7bd7-4573-8db2-ee1a83500b83","StagingPath":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/csi-pstore-2188ee9d93/globalmount/9c30da54-7bd7-4573-8db2-ee1a83500b83","level":"error","msg":"failed to flush multipath device: signal: killed","time":"2021-08-18T05:01:43.794691707Z"}Why can't flush multipath be executed ?
The 2 minutes are the default timeout value to try to delete a path.
In the next version, this timeout value will be lowered and possibly made configurable.
Do you think the rollout fails because of the multipathing ? Do you use Fiber Channel or iSCSI here ?
If you need a session with support please send me a private message to organize it.
Yes, we believe that multipath is the key to the problem, and we are using the Iscsi environment. We compared FC and ISCSI scenarios, and there is really no problem in FC mode.
Thank you very much！ we need some professional guidance for using csi-powerstore in iscsi scenarios, such as network card mode and system settings.
oisk
5 Posts
1530
0
Posted August 22nd, 2021 18:00
Before, we unanimously installed csi-powerstore via dell csi operator. Later, after we switched to the helm, everything was normal.