I think I am understanding the issue better now. The pods are running as root. I just ran the starttest.sh with 7vols as the chart. It created all the vols and the container has access. I ran a kubectl exec to get a shell to the pod. And I am root in the pod.
Forgive my ignorance, but I think because you developed for open shift. It may be that crio does not run pods as root. Rancher uses docker as the CRI. The created containers run as root. The issue makes more sense now.
Thanks for the additional info.
When I was reviewing this post I totally ignored "chown", as the title only has "chmod".
I think chown and chmod errors should be looked at separately as the root causes could be different. This is because chown must require root privilege to run while chmod does not. Thus for chown, whether it's the root running the container does matter (for chmod it might not matter as much, as long as the user is the owner of the directory being operated on).
So why don't we focus on the chown error you posted for the mysql helm chart installation first:
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
So it's clear that this error is here because the user is not root. Why isn't the user root? There could be multiple reasons:
1. Root squashing is enabled on Isilon which translates root to nobody on Isilon side. Adding the k8s nodes to rootClients field of the NFS export should solve the problem. Or this could be achieved by directly disabling root squashing on Isilon.
2. As you have mentioned, OpenShift will run pods using a randomly generated non-root user, in order to run as root, one should explicitly set "privileged:true" in the securityContext section of the manifest. In order for this to work, the "privileged" scc needs to be added to the service account first. Having said that, since you mentioned that when you manually log into a pod you can see that you're root, then I assume you're not on OpenShift and this would be a non-issue for you.
3. Other reasons why root is translated to a non-root user when running "chown" command.
So let me double confirm this first: for the mysql helm chart installation case specifically, did adding node IPs to "rootClients" work?
I am going to work through this one more time. I will be deploying the stable/postgresql helm chart. I'll document all the findings. If you can PM me your e-mail, I will send you the doc. Hopefully that will help with troubleshooting the issue.