If you do a lsblk on the node that is supposed to mount the volume do you see it ?
Also, can you run kubectl get pv,pvc,volumeattachment -o yaml and attach the logs (if possible clean it so we can see only the related volume details and not all the objects )
It's the host was not able initiator rescan was not working, and the lsblk was not displaying anything. I actually did reboot the worker, it thought it was going to do a re-scan, just to save time. But I got pay back on this. After I can the rescan, I can mount the volume now, and pod is running ok.
echo "- - -"> /sys/class/scsi_host/host4/scan
echo "- - -"> /sys/class/scsi_host/host3/scan
[root@worker containerd]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 20G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 19G 0 part ├─centos-root 253:0 0 17G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm sdb 8:16 0 10G 0 disk sdc 8:32 0 10G 0 disk /var/lib/kubelet/pods/0fe82c78-da1d-4893-b8e5-220968d0a8a5/volumes/kubernetes.io~csi/jij8-csivol-d
Flo_csI
2 Intern
•
167 Posts
0
August 6th, 2020 02:00
If you do a lsblk on the node that is supposed to mount the volume do you see it ?
Also, can you run kubectl get pv,pvc,volumeattachment -o yaml and attach the logs (if possible clean it so we can see only the related volume details and not all the objects )
143033
23 Posts
2
August 6th, 2020 07:00
Bingo!
It's the host was not able initiator rescan was not working, and the lsblk was not displaying anything. I actually did reboot the worker, it thought it was going to do a re-scan, just to save time. But I got pay back on this. After I can the rescan, I can mount the volume now, and pod is running ok.
echo "- - -"> /sys/class/scsi_host/host4/scan
echo "- - -"> /sys/class/scsi_host/host3/scan
[root@worker containerd]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root
253:0 0 17G 0 lvm /
└─centos-swap
253:1 0 2G 0 lvm
sdb 8:16 0 10G 0 disk
sdc 8:32 0 10G 0 disk /var/lib/kubelet/pods/0fe82c78-da1d-4893-b8e5-220968d0a8a5/volumes/kubernetes.io~csi/jij8-csivol-d
[root@master helm]# kubectl exec task-pv-pod -- df -h
Filesystem Size Used Avail Use% Mounted on
overlay 17G 2.6G 15G 15% /
tmpfs 64M 0 64M 0% /dev
tmpfs 910M 0 910M 0% /sys/fs/cgroup
/dev/mapper/centos-root 17G 2.6G 15G 15% /etc/hosts
shm 64M 0 64M 0% /dev/shm
/dev/sdc 9.8G 37M 9.2G 1% /usr/share/nginx/html >>>>>
Flo_csI
2 Intern
•
167 Posts
0
August 4th, 2020 05:00
Hi,
Can you confirm on the array side that the volume is present ?
Can you have a look at the status of iscsi on the node (systemctl status iscsid ; iscsiadm -m session) ?
Also, if there is nothing useful can you enable the csiDebug on the node thanks to kubectl edit -n unity deployment unity-node-xxx
Rgds
143033
23 Posts
0
August 5th, 2020 05:00
Hi,
Yes, I can confirm that on array side, once I created the pvc, the new storage lun can be created ok, and presenting to the worker node.
The worker node iSCSI session is also looking fine. Paths are showing active on Storage Unisphere as well.
[root@worker ~]# iscsiadm -m session
tcp: [1] 10.241.185.142:3260,1 iqn.1992-04.com.emc:cx.apm00194717505.a2 (non-flash)
tcp: [2] 10.241.185.143:3260,2 iqn.1992-04.com.emc:cx.apm00194717505.b2 (non-flash)
And about the CSI debug, I think I have enabled it by specified myvalues.yaml file:
[root@master helm]# cat myvalues.yaml
certSecretCount: 1
syncNodeInfoInterval: 15
volumeNamePrefix: jij8-csivol
snapNamePrefix: jij8-csi-snap
csiDebug: "true"
kubectl edit -n unity po unity-node-8g78s
value: unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock
- name: X_CSI_MODE
value: node
- name: X_CSI_DEBUG
value: "true"