Powerscale CSI driver initialization fails with 'Authorization required'

Question

Hi, We have a strange authorisation issue with Powerscale CSI driver that I am not sure where it comes from. So, we have the following setup: Isilon - OneFS 9.2.0 Separate access zone for CSI - zone-CSI (for security reasons don't want to allow System AZ access for CSI). AZ zone-CSI has base directory set to /ifs/CSI. Also a separate IP pool pool-CSI is created for AZ zone-CSI with range 192.168.10.51-53. Pool-CSI is in different groupnest and subnet than System zone .In zone-CSI created custom Role role-CSI with following privileges as per CSI documentation:ISI_PRIV_LOGIN_PAPI Read OnlyISI_PRIV_NFS Read WriteISI_PRIV_IFS_RESTORE Read OnlyISI_PRIV_NS_IFS_ACCESS Read OnlyISI_PRIV_IFS_BACKUP Read Only ISI_PRIV_QUOTA and ISI_PRIV_SNAPSHOT privileges are not set since zRBAC in 9.2.0 does not allow to add them. However, for the time being we don't plan to set quotas and snapshots in Kubernetes so these 2 privileges should not be an issue. In the AD authentication provider for zone-CSI created an user account isicsi and added it to the role-CSI to get listed above privileges. Our aim with that setup is to separate isicsi account in its own AZ and to prevent it to have privileges in all other access zones which will be the case if it is defined in system zone. We use Kubernetes v1.19.6CSI driver version 1.6 with following parameters (parameters with default values are not listed): myvalues.yaml# 'isiPort' defines the HTTPs port number of the PowerScale OneFS API serverisiPort: '8080' # The name of the access zone a volume can be created inisiAccessZone: 'zone-CSI' # 'volumeNamePrefix' defines a string prepended to each volume created by the CSI driver.volumeNamePrefix: k8s # The default base path for the volumes to be created, this will be used if a storage class does not have the IsiPath parameter specified# Ensure that this path exists on PowerScale.isiPath: '/ifs/CSI' secret.yamlisilonClusters:- clusterName: 'Isilon'             # logical name of PowerScale Clusterusername: 'isicsi'                    # username for connecting to PowerScale OneFS API serverpassword: 'xxxxxxx'                # password for connecting to PowerScale OneFS API serverendpoint: '192.168.10.51'       # HTTPS endpoint of the PowerScale OneFS API serverisDefault: true                          # default cluster (would be used by storage classes without ClusterName parameter)skipCertificateValidation: true  # indicates if client side validation of server's SSL certificate can be skippedisiPath: '/ifs/CSI'                      # base path for the volume(directory) to be created on PowerScaleEndpoint 192.168.10.51 is different than the endpoint for system zone. But as per Powerscale documentation the privilege ISI_PRIV_LOGIN_PAPI gives option to login or call API on the specific access zone with zone aware RBAC/.   With above setup when we try to initialize CSI driver Isilon returns 401 error msg='init client failed for isilon cluster 'Isilon': 'Authorization required''.At the same time when we try to login to Isilon with browser on same endpoint 192.168.10.51:8080 with same user isicsi it is successful and it shows management GUI interface with appropriate permissions defined by the role role-CSI (i.e. almost everything is disabled but full permission to create, delete and manage NFS exports). Also from K8s linux box: doing curl -kv https://192.168.10.51:8080 -uisicsi is successful doing curl -kv https://192.168.10.51:8080/platform/latest/ -uisicsi fails with Authorization required just the same as CSI driver initialisation So could anyone help me to understand what could be wrong and why API calls from CSI driver fail with 'Authorization required'? I found the following article, but don't think it is relevant in the case with CSI:https://www.dell.com/community/Isilon/API-Auth-Changes-with-Recent-Security-Patches-anti-CSRF-token/td-p/7170612 Any support will be much appreciated.

Flo_csI · Accepted Answer

Hi @ctodorov & @fdimatteo,

Isilon API 9.2 switched to a session-based authentication by default.

CSI 1.6 only supports basic authentication.

You can either switch the Isilon to enable basic authentication or wait for the next version of the driver which implements it and is just a few days away from a release.

HTH

fdimatteo123 · Answer

Hello ctodorov,

Is your Isilon cluster brand new deployed with version 9.2 or upgraded from a 8.x or 9.x?

Reason I ask is because I'm facing the exact same API authentication error with brand new 9.2 clusters. I tested the same in 2 clusters that had been upgraded to 9.2 from previous versions, and there API authentication works fine...

I haven't found a cause yet, but it would be nice to see if you have the same pattern.

ctodorov · Answer

Hello fdimatteo,My cluster is brand new deployed with version 9.2.0. So it confirms your pattern. The same K8s cluster works perfectly fine with CSI ver1.6 and another Isilon with OneFS 8.1.2. Unfortunately it is Isilon-SD and could not be upgraded to newer versions of OneFS (product goes discontinued).I was thinking that the issue might be the fact I want to use different AZ with zRBAC for CSI instead of system AZ. But now with your question that you have same issue only with new deployed 9.2.x clusters, while it works fine with clusters upgraded from 8.x to 9.x the issue could be else. I plan to upgrade to 9.2.1 to see if the issue would persist and will post here once it is done.

ctodorov · Answer

Thank you so much for that information! It helps a lot!

Do you know if we could expect next version of CSI to be released this week or not? We are little bit on a rush here to verify this setup before deploying it into production. I don't want to enable basic authentication if it is matter of days.

ctodorov · Answer

That's awesome news! We downloaded CSI 2.0.0 and start testing it. Will update here once we have some results. Thank you so much for your prompt responses and support!

Flo_csI · Answer

It is GA for a couple of hours: https://github.com/dell/csi-powerscale  The doc site is on its way but if you are familiar with existing install process it shouldn't be a pb for you.

ctodorov · Answer

Hi again, I confirm that after upgrading to CSI 2.0.0, session based authentication worked and CSI driver was able to authenticate to Isilon on 9.2.0. Thank you again!

I want to use the opportunity to ask how CSI should be configured to use for authentication AD account? When we upgraded to CSI 2.0 we configured it to use (as per above configuration) "isicsi" and got reply "unknown user or password".
I realised that "isicsi" is a user from AD domain and I should put domain prefix to the user in CSI. So tried to put "DEV\\isicsi" (put 2 \\ to escape DEV\isicsi) in helm values.yaml to indicate the domain. But it returned unknown escape character.

At that moment I created local user iscsi on Isilon in the AZ, added it to the CSI role and removed from role AD account. Then tried with CSI and it worked instantly.

So my question is what should be the syntax to use AD account for CSI to authenticate successfully in Isilon API? DOMAIN\\user didn't work, never tried DOMAIN@user though.

We have very strong security requirements and that's the reason to configure CSI to use its own dedicated AZ on Isilon and also need to use AD provided service account to authenticate in Isilon.

Sean_Zhan · Answer

Hi Ctodorov,

The AD user works well in my env. I'm running CSI 1.6 though, but the CSI behavior on this point should be same as 2.0. I noticed you were using double slash "\\" in your configuration. Probably you have to change it to "\", i.e. "DOMAIN\user".

ctodorov · Answer

Hi Sean_Zan,

thank you for your reply. I had no enough time to test it with slashes but it instantly worked with DOMAIN@user

So now my instance authenticates using AD domain provided service account towards separate dedicated access zone for CSI on Isilon. I am using custom role created there and assigned domain service account to that role as per CSI documentation.

The only thing I still miss is that in custom roles it is not possible to give privileges about quota and snapshot. But I am still on 9.2.0. I have to upgrade to 9.2.1 since someone mentioned there it will be also possible to assign these privileges to a custom role.

Sean_Zhan · Answer

Hi ctodorov

Roles in non-system zone, cannot have privileges of quota and snapshot. Whether or not the user is an AD user or local user doesn't matter.

Can you check if the user you are using for CSI is in system zone?

Containers

Was this post helpful?