I talked to Isilon team and they think it's worth filing a ticket via the official Isilon customer support channel so that they can look at this issue from NFS perspective.
At the end of the day, it all boils down to how permission management is handled on the NFS export from Isilon, so Isilon folks might be able to provide some useful information.
Also, I'm still not very sure how Isilon is leveraged in your use case, could you please give more details in this regard?
For example, when you install postgre/mysql helm chart, how do you create the PVC and what path do you mount it to? Can you please paste the corresponding sections of the yaml files so we can have a better idea about your specific use case?
OK. We are trying to use isilon for persistent storage in k8s. I found the csi and decided to test it out. Our current environment is VMs and it made the most sense to use isilon as it exports as NFS. We have the device, so why not use it. The CSI is great because it will provision storage as pvc's are created. The alternative is doing a lot of manual work and then having people use and exisitng pv or pvc. I am not sure how else to explain this, the point of the CSI is to create pv's behind the scenes.
The CSI does what it's supposed to do. But when deploying postgresql, or mysql, via standard helm charts, the pods that get created end up in a crashloop because they cannot change ownership, or mode, of directories under the exported share. I am not a developer by training, I am a sysadmin. I am just using the default helm charts. The stable/postgresql helm chart be default will use persistent storage. I have the CSI set as the default storage class. It's creating the k8s prefixed folder, the export, adding the host to the export, and creating a quota based on the requested size. That part works great. When the share is mounted on the worker node and a root account creates a directory, it gets the owner/group of nobody:nobody. Which is the expected result since the worker node is not in the root client list for the exported share.
The CSI works as expected. It's either the way the helm chart works, the way the image for the containers work, or something in between. From what I can gather in the code, there is nothing in the CSI for adding worker nodes as root clients. This may have been by design. But my guess is that there may be use cases where this is necessary. If there was a way to deploy the CSI as a storage class that added root clients, a user could then choose the storage class when doing a deployment, replicaset, or pod.
Overall, my use case is that I would like to use the CSI for persistent storage. I don't know that is should matter what I am trying to deploy. There are 100s of apps out there being thrusted into the k8s world. I don't know what any developer may choose to use with persistent storage. I just want that when they request it via a pvc, it works without a lot of manual intervention.
I think I am understanding the issue better now. The pods are running as root. I just ran the starttest.sh with 7vols as the chart. It created all the vols and the container has access. I ran a kubectl exec to get a shell to the pod. And I am root in the pod.
Forgive my ignorance, but I think because you developed for open shift. It may be that crio does not run pods as root. Rancher uses docker as the CRI. The created containers run as root. The issue makes more sense now.
When I was reviewing this post I totally ignored "chown", as the title only has "chmod".
I think chown and chmod errors should be looked at separately as the root causes could be different. This is because chown must require root privilege to run while chmod does not. Thus for chown, whether it's the root running the container does matter (for chmod it might not matter as much, as long as the user is the owner of the directory being operated on).
So why don't we focus on the chown error you posted for the mysql helm chart installation first:
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
So it's clear that this error is here because the user is not root. Why isn't the user root? There could be multiple reasons: 1. Root squashing is enabled on Isilon which translates root to nobody on Isilon side. Adding the k8s nodes to rootClients field of the NFS export should solve the problem. Or this could be achieved by directly disabling root squashing on Isilon. 2. As you have mentioned, OpenShift will run pods using a randomly generated non-root user, in order to run as root, one should explicitly set "privileged:true" in the securityContext section of the manifest. In order for this to work, the "privileged" scc needs to be added to the service account first. Having said that, since you mentioned that when you manually log into a pod you can see that you're root, then I assume you're not on OpenShift and this would be a non-issue for you. 3. Other reasons why root is translated to a non-root user when running "chown" command.
So let me double confirm this first: for the mysql helm chart installation case specifically, did adding node IPs to "rootClients" work?
I am going to work through this one more time. I will be deploying the stable/postgresql helm chart. I'll document all the findings. If you can PM me your e-mail, I will send you the doc. Hopefully that will help with troubleshooting the issue.
Our use case is a bit different as we enforce runAsNonRoot by PodSecurityPolicies, it's impossible to start a pod like postgres using a csi isilon pvc except if we can add a subPath (which is normally the case usally).
But for instance on tomcat helm chart baked by Bitnami, it is impossible.....
I'm surprised that this issue has not been addressed as running with privileged: true is not an option at all.
cocampbe
36 Posts
0
January 15th, 2020 14:00
Seems I am out of luck. Been so focused on postgres. I just tried a mysql helm chart and same issue.
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
It's interesting how the the pod deployed using the starttest.sh script works fine.
Zack.Zhang
44 Posts
0
January 15th, 2020 17:00
I talked to Isilon team and they think it's worth filing a ticket via the official Isilon customer support channel so that they can look at this issue from NFS perspective.
At the end of the day, it all boils down to how permission management is handled on the NFS export from Isilon, so Isilon folks might be able to provide some useful information.
Zack.Zhang
44 Posts
0
January 15th, 2020 19:00
Also, I'm still not very sure how Isilon is leveraged in your use case, could you please give more details in this regard?
For example, when you install postgre/mysql helm chart, how do you create the PVC and what path do you mount it to? Can you please paste the corresponding sections of the yaml files so we can have a better idea about your specific use case?
cocampbe
36 Posts
0
January 16th, 2020 06:00
OK. We are trying to use isilon for persistent storage in k8s. I found the csi and decided to test it out. Our current environment is VMs and it made the most sense to use isilon as it exports as NFS. We have the device, so why not use it. The CSI is great because it will provision storage as pvc's are created. The alternative is doing a lot of manual work and then having people use and exisitng pv or pvc. I am not sure how else to explain this, the point of the CSI is to create pv's behind the scenes.
The CSI does what it's supposed to do. But when deploying postgresql, or mysql, via standard helm charts, the pods that get created end up in a crashloop because they cannot change ownership, or mode, of directories under the exported share. I am not a developer by training, I am a sysadmin. I am just using the default helm charts. The stable/postgresql helm chart be default will use persistent storage. I have the CSI set as the default storage class. It's creating the k8s prefixed folder, the export, adding the host to the export, and creating a quota based on the requested size. That part works great. When the share is mounted on the worker node and a root account creates a directory, it gets the owner/group of nobody:nobody. Which is the expected result since the worker node is not in the root client list for the exported share.
The CSI works as expected. It's either the way the helm chart works, the way the image for the containers work, or something in between. From what I can gather in the code, there is nothing in the CSI for adding worker nodes as root clients. This may have been by design. But my guess is that there may be use cases where this is necessary. If there was a way to deploy the CSI as a storage class that added root clients, a user could then choose the storage class when doing a deployment, replicaset, or pod.
Overall, my use case is that I would like to use the CSI for persistent storage. I don't know that is should matter what I am trying to deploy. There are 100s of apps out there being thrusted into the k8s world. I don't know what any developer may choose to use with persistent storage. I just want that when they request it via a pvc, it works without a lot of manual intervention.
cocampbe
36 Posts
0
January 16th, 2020 06:00
I think I am understanding the issue better now. The pods are running as root. I just ran the starttest.sh with 7vols as the chart. It created all the vols and the container has access. I ran a kubectl exec to get a shell to the pod. And I am root in the pod.
Forgive my ignorance, but I think because you developed for open shift. It may be that crio does not run pods as root. Rancher uses docker as the CRI. The created containers run as root. The issue makes more sense now.
Zack.Zhang
44 Posts
0
January 16th, 2020 17:00
Thanks for the additional info.
When I was reviewing this post I totally ignored "chown", as the title only has "chmod".
I think chown and chmod errors should be looked at separately as the root causes could be different. This is because chown must require root privilege to run while chmod does not. Thus for chown, whether it's the root running the container does matter (for chmod it might not matter as much, as long as the user is the owner of the directory being operated on).
So why don't we focus on the chown error you posted for the mysql helm chart installation first:
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
So it's clear that this error is here because the user is not root. Why isn't the user root? There could be multiple reasons:
1. Root squashing is enabled on Isilon which translates root to nobody on Isilon side. Adding the k8s nodes to rootClients field of the NFS export should solve the problem. Or this could be achieved by directly disabling root squashing on Isilon.
2. As you have mentioned, OpenShift will run pods using a randomly generated non-root user, in order to run as root, one should explicitly set "privileged:true" in the securityContext section of the manifest. In order for this to work, the "privileged" scc needs to be added to the service account first. Having said that, since you mentioned that when you manually log into a pod you can see that you're root, then I assume you're not on OpenShift and this would be a non-issue for you.
3. Other reasons why root is translated to a non-root user when running "chown" command.
So let me double confirm this first: for the mysql helm chart installation case specifically, did adding node IPs to "rootClients" work?
cocampbe
36 Posts
0
January 17th, 2020 06:00
I am going to work through this one more time. I will be deploying the stable/postgresql helm chart. I'll document all the findings. If you can PM me your e-mail, I will send you the doc. Hopefully that will help with troubleshooting the issue.
Zack.Zhang
44 Posts
0
January 20th, 2020 17:00
Sure @cocampbe , I've PMed my email to you, please feel free to send info over.
thomas.menard
3 Posts
0
May 4th, 2021 09:00
Hi
We encounter also such situation.
Our use case is a bit different as we enforce runAsNonRoot by PodSecurityPolicies, it's impossible to start a pod like postgres using a csi isilon pvc except if we can add a subPath (which is normally the case usally).
But for instance on tomcat helm chart baked by Bitnami, it is impossible.....
I'm surprised that this issue has not been addressed as running with privileged: true is not an option at all.