I am starting to think that it's a mix of the way the pod is setup and the isilon permissions. I am curious if you could add AddExportRootClients to the csi. I was thinking I could create a second storage class for pods that would need the root client set. 

Yeah, we have not verified with rancher (and not supported) but this may be related to user privileges. (userid used by CSI driver)

Is this with root userid (secret created)? If not, does user have enough privileges to perform the actions as mentioned in the Product Guide?

We created a user and set the permissoins per the product guide. The directory gets created with the default k8s prefix and is owned by the isi user. It also gets exported and the smart quota is applied. The pod, postgres in this case, creates a data directory under the exported share. It's owned by root. The pod tried to change the mode and ownership. It fails at this step. I am deploying postgres via helm using the stable chart. If I attach to the pod, I can mount the share, but as root I cannot change the ownership, etc.

I changed the mode and ownership via the isi cli, but that did not resolve the issue. As far as the CSI goes, it seems to work well with rancher running on OL7. Just need to figure out why the pod cannot change ownership and mode. 

Just re-reading the product guide. I see that we did not follow it to the 't'. We created the isi user in another zone. Per the guide, "The username must be from the authentication providers in the System zone of Isilon." Wondering if that is the issue.

I take that back. The user is in the system zone.

I think your observation is correct.

The user has been translated from root to nobody because your Isilon cluster has root squashing enabled.

However, I don't think this is the crux of the issue. As you have suspected, csi-isilon does not add the node to the "root clients" field of the NFS export. That's why chmod run on a directory on the volume fails, even if the command is run by the owner of the directory.

If you have access to your Isilon cluster's OneFS UI (or REST API), as a workaround, you can manually edit the NFS export backing the volume and add the node ip/FQDN to its "root clients" field.

I did try that. The changes did not help. It's seems like once the pod is up, it just goes into a crashloop trying to run the commands. If I delete the pod the whole deployment fails. I am thinking about creating a new access zone with differing defaults for shares. That way when the csi creates new shares, it should get the defaults and possibly allow access. Now I just need more IPs.

I have tested it myself, after adding node IP to the root clients field of the NFS export, chmod does work now.

So my test shows that the root clients field does matter.

Having said that, the root cause could be manifold, there might be some other issues at play.  But adding node IP to root clients is one of the things that must be done for chmod to work.

Yep. for example a postgres pod will not work using this CSI. Trying to work around the issue is nearly impossible.

For postgres - I had tested with bitnami helm chart and that worked fine for my tests at that time. Is this any specific use case that you were trying?