CSM: CSI Driver for PowerStore "Cannot match block device with provided HCTL"
Summary: Container Storage Module (CSM) Container Storage Interface (CSI) Driver shows that it cannot match block device with provided Host Channel Target LUN (HCTL). This occurs with PowerStore array over Fiber Channel (FC) on CoreOS hosts in OpenShift. This occurs when the logical unit number (LUN) identification numbers (IDs) are greater than 255 due to a CoreOS limitation. ...
Symptoms
The user is unable to mount after the LUN ID is over 255 with the following errors:
Kubernetes Pod error:
>MountVolume.MountDevice failed for volume "csivol-837d8735a7" : rpc error: code = Internal desc = unable to find device after multiple discovery attempts: wwn for FC device not found
dmesg/Journal logs:
kernel: scsi: host X channel Y id Z lun<nnn> has a LUN larger than allowed by the host adapter
CSI Driver Node log
{"ID":"36634f4a-ff50-40f5-8c6d-384aef677f4e","Lun":"288","StagingPath":"/var/lib/kubelet/plugins/kubernetes.io/csi/csi-powerstore.dellemc.com/53bb9c3360084159a88e3325b93e3635f525b9fb528c00e391aceb56fa9a5a17/globalmount/36634f4a-ff50-40f5-8c6d-384aef677f4e","Targets":null,"WWN":"68ccf0980011601b6037197dd0b3248d","level":"error","msg":"can't match block device with provided HCTL, 11 0 0 288","time":"2023-12-14T15:29:45.070156092Z"}
Cause
The CSI Driver for PowerStore uses Representational State Transfer (REST) Application Programming Interface (API) calls to get the LUN ID from the PowerStore array.
The LUN ID continues to increment up to the maximum LUN ID for the host type. Once this is reached, the array starts reusing the lower numbers.
The default LUN ID limit in Red Hat CoreOS (RHCOS) and Red Hat Enterprise Linux is 255 (0-255). The LUN ID provided by the PowerStore exceeded this number.
Resolution
Resolution:
The maximum LUN could not be changed on CoreOS with typical Linux procedures because it is an immutable operating system.
The hosts are not Storage Area Network (SAN) and do not use iDRAC.
This allowed for the maximum number of LUN IDs allowed by the Host Bus Adapter (HBA) to be increased in RHCOS.
Red Hat provided the following article (KB 26017) to increase the maximum LUN using a MachineConfig:
How to increase the number of LUN IDs allowed by my HBA in RHCOS? (External Link)
Workaround:
- Cordon and drain the problematic Kubernetes node
- In the PowerStore UI, remove the node, assuming there are no volumes masked.
- Reboot the Kubernetes node
- The Kubernetes node
autoregistersin the PowerStore UI again on startup - Uncordon the Kubernetes node and remount the PVCs again to the same node