8 Posts
0
4848
Powerscale CSI driver initialization fails with "Authorization required"
Hi,
We have a strange authorisation issue with Powerscale CSI driver that I am not sure where it comes from.
So, we have the following setup:
Isilon - OneFS 9.2.0
Separate access zone for CSI - zone-CSI (for security reasons don't want to allow System AZ access for CSI). AZ zone-CSI has base directory set to /ifs/CSI. Also a separate IP pool pool-CSI is created for AZ zone-CSI with range 192.168.10.51-53. Pool-CSI is in different groupnest and subnet than System zone
.
In zone-CSI created custom Role role-CSI with following privileges as per CSI documentation:
ISI_PRIV_LOGIN_PAPI Read Only
ISI_PRIV_NFS Read Write
ISI_PRIV_IFS_RESTORE Read Only
ISI_PRIV_NS_IFS_ACCESS Read Only
ISI_PRIV_IFS_BACKUP Read Only
ISI_PRIV_QUOTA and ISI_PRIV_SNAPSHOT privileges are not set since zRBAC in 9.2.0 does not allow to add them. However, for the time being we don't plan to set quotas and snapshots in Kubernetes so these 2 privileges should not be an issue.
In the AD authentication provider for zone-CSI created an user account isicsi and added it to the role-CSI to get listed above privileges.
Our aim with that setup is to separate isicsi account in its own AZ and to prevent it to have privileges in all other access zones which will be the case if it is defined in system zone.
We use Kubernetes v1.19.6
CSI driver version 1.6 with following parameters (parameters with default values are not listed):
myvalues.yaml
# "isiPort" defines the HTTPs port number of the PowerScale OneFS API server
isiPort: "8080"
# The name of the access zone a volume can be created in
isiAccessZone: "zone-CSI"
# "volumeNamePrefix" defines a string prepended to each volume created by the CSI driver.
volumeNamePrefix: k8s
# The default base path for the volumes to be created, this will be used if a storage class does not have the IsiPath parameter specified
# Ensure that this path exists on PowerScale.
isiPath: "/ifs/CSI"
secret.yaml
isilonClusters:
- clusterName: "Isilon" # logical name of PowerScale Cluster
username: "isicsi" # username for connecting to PowerScale OneFS API server
password: "xxxxxxx" # password for connecting to PowerScale OneFS API server
endpoint: "192.168.10.51" # HTTPS endpoint of the PowerScale OneFS API server
isDefault: true # default cluster (would be used by storage classes without ClusterName parameter)
skipCertificateValidation: true # indicates if client side validation of server's SSL certificate can be skipped
isiPath: "/ifs/CSI" # base path for the volume(directory) to be created on PowerScale
Endpoint 192.168.10.51 is different than the endpoint for system zone. But as per Powerscale documentation the privilege ISI_PRIV_LOGIN_PAPI gives option to login or call API on the specific access zone with zone aware RBAC/.
With above setup when we try to initialize CSI driver Isilon returns 401 error msg="init client failed for isilon cluster 'Isilon': 'Authorization required'".
At the same time when we try to login to Isilon with browser on same endpoint 192.168.10.51:8080 with same user isicsi it is successful and it shows management GUI interface with appropriate permissions defined by the role role-CSI (i.e. almost everything is disabled but full permission to create, delete and manage NFS exports).
Also from K8s linux box:
doing curl -kv https://192.168.10.51:8080 -uisicsi is successful
doing curl -kv https://192.168.10.51:8080/platform/latest/ -uisicsi fails with Authorization required just the same as CSI driver initialisation
So could anyone help me to understand what could be wrong and why API calls from CSI driver fail with "Authorization required"?
I found the following article, but don't think it is relevant in the case with CSI:
https://www.dell.com/community/Isilon/API-Auth-Changes-with-Recent-Security-Patches-anti-CSRF-token/td-p/7170612
Any support will be much appreciated.
Flo_csI
166 Posts
0
September 20th, 2021 06:00
Hi @ctodorov & @fdimatteo,
Isilon API 9.2 switched to a session-based authentication by default.
CSI 1.6 only supports basic authentication.
You can either switch the Isilon to enable basic authentication or wait for the next version of the driver which implements it and is just a few days away from a release.
HTH
fdimatteo123
1 Message
0
September 17th, 2021 12:00
Hello ctodorov,
Is your Isilon cluster brand new deployed with version 9.2 or upgraded from a 8.x or 9.x?
Reason I ask is because I'm facing the exact same API authentication error with brand new 9.2 clusters. I tested the same in 2 clusters that had been upgraded to 9.2 from previous versions, and there API authentication works fine...
I haven't found a cause yet, but it would be nice to see if you have the same pattern.
ctodorov
8 Posts
0
September 20th, 2021 00:00
Hello fdimatteo,
My cluster is brand new deployed with version 9.2.0. So it confirms your pattern. The same K8s cluster works perfectly fine with CSI ver1.6 and another Isilon with OneFS 8.1.2. Unfortunately it is Isilon-SD and could not be upgraded to newer versions of OneFS (product goes discontinued).
I was thinking that the issue might be the fact I want to use different AZ with zRBAC for CSI instead of system AZ. But now with your question that you have same issue only with new deployed 9.2.x clusters, while it works fine with clusters upgraded from 8.x to 9.x the issue could be else.
I plan to upgrade to 9.2.1 to see if the issue would persist and will post here once it is done.
ctodorov
8 Posts
0
September 20th, 2021 08:00
Thank you so much for that information! It helps a lot!
Do you know if we could expect next version of CSI to be released this week or not? We are little bit on a rush here to verify this setup before deploying it into production. I don't want to enable basic authentication if it is matter of days.
ctodorov
8 Posts
0
September 21st, 2021 00:00
That's awesome news! We downloaded CSI 2.0.0 and start testing it. Will update here once we have some results. Thank you so much for your prompt responses and support!
Flo_csI
166 Posts
0
September 21st, 2021 00:00
It is GA for a couple of hours: https://github.com/dell/csi-powerscale
The doc site is on its way but if you are familiar with existing install process it shouldn't be a pb for you.
ctodorov
8 Posts
0
September 21st, 2021 07:00
Hi again, I confirm that after upgrading to CSI 2.0.0, session based authentication worked and CSI driver was able to authenticate to Isilon on 9.2.0. Thank you again!
I want to use the opportunity to ask how CSI should be configured to use for authentication AD account? When we upgraded to CSI 2.0 we configured it to use (as per above configuration) "isicsi" and got reply "unknown user or password".
I realised that "isicsi" is a user from AD domain and I should put domain prefix to the user in CSI. So tried to put "DEV\\isicsi" (put 2 \\ to escape DEV\isicsi) in helm values.yaml to indicate the domain. But it returned unknown escape character.
At that moment I created local user iscsi on Isilon in the AZ, added it to the CSI role and removed from role AD account. Then tried with CSI and it worked instantly.
So my question is what should be the syntax to use AD account for CSI to authenticate successfully in Isilon API? DOMAIN\\user didn't work, never tried DOMAIN@user though.
We have very strong security requirements and that's the reason to configure CSI to use its own dedicated AZ on Isilon and also need to use AD provided service account to authenticate in Isilon.
Sean_Zhan
4 Posts
0
October 5th, 2021 20:00
Hi Ctodorov,
The AD user works well in my env. I'm running CSI 1.6 though, but the CSI behavior on this point should be same as 2.0. I noticed you were using double slash "\\" in your configuration. Probably you have to change it to "\", i.e. "DOMAIN\user".
ctodorov
8 Posts
0
October 6th, 2021 00:00
Hi Sean_Zan,
thank you for your reply. I had no enough time to test it with slashes but it instantly worked with DOMAIN@user
So now my instance authenticates using AD domain provided service account towards separate dedicated access zone for CSI on Isilon. I am using custom role created there and assigned domain service account to that role as per CSI documentation.
The only thing I still miss is that in custom roles it is not possible to give privileges about quota and snapshot. But I am still on 9.2.0. I have to upgrade to 9.2.1 since someone mentioned there it will be also possible to assign these privileges to a custom role.
Sean_Zhan
4 Posts
0
October 7th, 2021 23:00
Hi ctodorov
Roles in non-system zone, cannot have privileges of quota and snapshot. Whether or not the user is an AD user or local user doesn't matter.
Can you check if the user you are using for CSI is in system zone?