PowerFlex: Kubernetes CSI Driver deployment errors

Incorrect Gateway URL

# kubectl logs -n vxflexos vxflexos-node-28gh2 driver
time="2021-10-06T10:31:16Z" level=debug msg="enabled context injector"
time="2021-10-06T10:31:16Z" level=debug msg="init req & rep validation" withSpec=false
time="2021-10-06T10:31:16Z" level=debug msg="init implicit req validation" withSpecReq=false
time="2021-10-06T10:31:16Z" level=debug msg="init req validation" withSpecReq=true
time="2021-10-06T10:31:16Z" level=debug msg="enabled request ID injector"
time="2021-10-06T10:31:16Z" level=debug msg="enabled request logging"
time="2021-10-06T10:31:16Z" level=debug msg="enabled response logging"
time="2021-10-06T10:31:16Z" level=debug msg="enabled spec validator opt: request validation"
time="2021-10-06T10:31:16Z" level=debug msg="enabled serial volume access"
time="2021-10-06T10:31:16Z" level=info msg="configured 55b832655862adb7" allSystemNames= endpoint="http://10.10.10.19:28443" isDefault=true password="********" skipCertificateValidation=true systemID=55b832655862adb7 user=K8S_user
time="2021-10-06T10:31:16Z" level=info msg="driver configuration file " file=/vxflexos-config-params/driver-config-params.yaml
time="2021-10-06T10:31:16Z" level=info msg="Read CSI_LOG_FORMAT from log configuration file" format=text
time="2021-10-06T10:31:16Z" level=info msg="Read CSI_LOG_LEVEL from log configuration file" fields.level=debug
time="2021-10-06T10:31:16Z" level=info msg="array configuration file" file=/vxflexos-config/config
time="2021-10-06T10:31:16Z" level=info msg="Probing all arrays. Number of arrays: 1"
time="2021-10-06T10:31:16Z" level=error msg="array 55b832655862adb7 probe failed: rpc error: code = FailedPrecondition desc = unable to login to VxFlexOS Gateway: invalid character 'B' looking for beginning of value"
time="2021-10-06T10:31:16Z" level=info msg="configured csi-vxflexos.dellemc.com" allowRWOMultiPodAccess=false autoprobe=true mode=node privatedir=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks sdcGUID= thickprovision=false
time="2021-10-06T10:31:16Z" level=info msg="removed sock file" path=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/csi_sock
time="2021-10-06T10:31:16Z" level=fatal msg="grpc failed" error="rpc error: code = FailedPrecondition desc = All arrays are not working. Could not proceed further: map[55b832655862adb7:rpc error: code = FailedPrecondition desc = unable to login to VxFlexOS Gateway: invalid character 'B' looking for beginning of value]"

The issue below is with the config.yaml file - the endpoint should point to the IM Gateway via "https:", not "http":

config.yaml:

username: "K8S_user"
password: "password"
systemID: "55b832655862adb7"
endpoint: "http://10.10.10.19:28443" <<<
skipCertificateValidation: true
isDefault: true
mdm: "10.10.10.20,10.10.10.21,10.10.10.22"

User password needs to be changed.

# kubectl logs -n vxflexos vxflexos-node-xkhmv driver
time="2021-10-06T13:24:00Z" level=debug msg="enabled context injector"
time="2021-10-06T13:24:00Z" level=debug msg="init req & rep validation" withSpec=false
time="2021-10-06T13:24:00Z" level=debug msg="init implicit req validation" withSpecReq=false
time="2021-10-06T13:24:00Z" level=debug msg="init req validation" withSpecReq=true
time="2021-10-06T13:24:00Z" level=debug msg="enabled request ID injector"
time="2021-10-06T13:24:00Z" level=debug msg="enabled request logging"
time="2021-10-06T13:24:00Z" level=debug msg="enabled response logging"
time="2021-10-06T13:24:00Z" level=debug msg="enabled spec validator opt: request validation"
time="2021-10-06T13:24:00Z" level=debug msg="enabled serial volume access"
time="2021-10-06T13:24:00Z" level=info msg="configured 55b832655862adb7" allSystemNames= endpoint="https://10.10.10.19:28443" isDefault=true password="********" skipCertificateValidation=true systemID=55b832655862adb7 user=K8S_user
time="2021-10-06T13:24:00Z" level=info msg="driver configuration file " file=/vxflexos-config-params/driver-config-params.yaml
time="2021-10-06T13:24:00Z" level=info msg="Read CSI_LOG_FORMAT from log configuration file" format=text
time="2021-10-06T13:24:00Z" level=info msg="Read CSI_LOG_LEVEL from log configuration file" fields.level=debug
time="2021-10-06T13:24:00Z" level=info msg="array configuration file" file=/vxflexos-config/config
time="2021-10-06T13:24:00Z" level=info msg="Probing all arrays. Number of arrays: 1"
time="2021-10-06T13:24:00Z" level=error msg="array 55b832655862adb7 probe failed: rpc error: code = FailedPrecondition desc = unable to find matching VxFlexOS system name: err: problem getting instances: Password change required. Please change your password."
time="2021-10-06T13:24:00Z" level=info msg="configured csi-vxflexos.dellemc.com" allowRWOMultiPodAccess=false autoprobe=true mode=node privatedir=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks sdcGUID= thickprovision=false
time="2021-10-06T13:24:00Z" level=info msg="removed sock file" path=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/csi_sock
time="2021-10-06T13:24:00Z" level=fatal msg="grpc failed" error="rpc error: code = FailedPrecondition desc = All arrays are not working. Could not proceed further: map[55b832655862adb7:rpc error: code = FailedPrecondition desc = unable to find matching VxFlexOS system name: err: problem getting instances: Password change required. Please change your password.]"

After logging in via SCLI and changing the password, the deployment will proceed:

# scli --login --username K8S_user --password password
Logged in. User role is Administrator. System ID is 55b832655862adb7
Current password must be changed. Please use --set_password command

# scli --set_password --old_password password --new_password newpassword

# scli --login --username K8S_user --password newpassword
Logged in. User role is Administrator. System ID is 55b832655862adb7

Incorrect system IDs in SDC configuration

# kubectl logs -n vxflexos vxflexos-node-7r8df driver
time="2021-10-06T14:17:05Z" level=debug msg="enabled context injector"
time="2021-10-06T14:17:05Z" level=debug msg="init req & rep validation" withSpec=false
time="2021-10-06T14:17:05Z" level=debug msg="init implicit req validation" withSpecReq=false
time="2021-10-06T14:17:05Z" level=debug msg="init req validation" withSpecReq=true
time="2021-10-06T14:17:05Z" level=debug msg="enabled request ID injector"
time="2021-10-06T14:17:05Z" level=debug msg="enabled request logging"
time="2021-10-06T14:17:05Z" level=debug msg="enabled response logging"
time="2021-10-06T14:17:05Z" level=debug msg="enabled spec validator opt: request validation"
time="2021-10-06T14:17:05Z" level=debug msg="enabled serial volume access"
time="2021-10-06T14:17:05Z" level=info msg="configured 55b832655862adb7" allSystemNames= endpoint="https://10.10.10.19:28443" isDefault=true password="********" skipCertificateValidation=true systemID=55b832655862adb7 user=K8S_user
time="2021-10-06T14:17:05Z" level=info msg="driver configuration file " file=/vxflexos-config-params/driver-config-params.yaml
time="2021-10-06T14:17:05Z" level=info msg="Read CSI_LOG_FORMAT from log configuration file" format=text
time="2021-10-06T14:17:05Z" level=info msg="Read CSI_LOG_LEVEL from log configuration file" fields.level=debug
time="2021-10-06T14:17:05Z" level=info msg="array configuration file" file=/vxflexos-config/config
time="2021-10-06T14:17:05Z" level=info msg="Probing all arrays. Number of arrays: 1"
time="2021-10-06T14:17:05Z" level=info msg="default array is set to array ID: 55b832655862adb7"
time="2021-10-06T14:17:05Z" level=info msg="55b832655862adb7 is the default array, skipping VolumePrefixToSystems map update. \n"
time="2021-10-06T14:17:05Z" level=info msg="array 55b832655862adb7 probed successfully"
time="2021-10-06T14:17:05Z" level=info msg="set SDC GUID" guid=3F3C5F0A-9307-42ED-91FC-67CCE89770FA
time="2021-10-06T14:17:05Z" level=info msg="Found connected system" ID=0000000000000000
time="2021-10-06T14:17:05Z" level=info msg="configured csi-vxflexos.dellemc.com" allowRWOMultiPodAccess=false autoprobe=true mode=node privatedir=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/disks sdcGUID=3F3C5F0A-9307-42ED-91FC-67CCE89770FA thickprovision=false
time="2021-10-06T14:17:05Z" level=info msg="removed sock file" path=/var/lib/kubelet/plugins/vxflexos.emc.dell.com/csi_sock
time="2021-10-06T14:17:05Z" level=fatal msg="grpc failed" error="rpc error: code = FailedPrecondition desc = duplicate systems found that are known to SDC: 0000000000000000"

The SDC configuration (/etc/emc/scaleio/drv_cfg.txt) contains multiple incorrect system entries (MDM-ID) - "0000000000000000." Removing incorrect system IDs with "drv_cfg" fixes the issue.

Impact

Unable to install the PowerFlex CSI Driver.

User misconfigurations

See steps to resolve each issue in the Symptoms section above. 

Impacted Versions

Not a PowerFlex issue