sparky102
1 Copper

CSI Driver for Unity VSA - unity-node problems

I have a working Unity VSA running in VM Ware Workstation (test platform). It is successfully providing storage to pods running in CoreOS based Kubernetes cluster.

I have download and installed the CSI drivers from the git repo (https://github.com/dell/csi-unity/tree/v1.0.0) following the instructions and using Helm to create the necessary unity-controller-0 and unity-node-<xxxxx> pods.

The unity-controller-0 pod is reliably running, authenticates with the VSA and the logs contain no errors.

The unity-node pods initally come up as Running, but within seconds their state changes to Error and then CrashLoopBackOff. This cycle repeats continually.

Looking at the logs with 'kubectl logs unity-node-k5k5h -n unity -c driver' I see the following:

Endpoint unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock
ls: cannot access /var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock: No such file or directory
gounity logger initiated. This should be called only once.
csi-unity logger initiated. This should be called only once.
time="2020-03-02T23:41:18Z" level=info msg="X_CSI_MODE:node"
time="2020-03-02T23:41:18Z" level=info msg="X_CSI_UNITY_NODENAME:node05"
time="2020-03-02T23:41:18Z" level=info msg="configured csi-unity.dellemc.com" autoprobe=false endpoint="https://10.0.1.50" insecure=true mode=node nodename=node05 password="******" pvtMountDir=/var/lib/kubelet/plugins/unity.emc.dell.com/disks systemname= user=admin
time="2020-03-02T23:41:18Z" level=info msg="identity service registered"
time="2020-03-02T23:41:18Z" level=info msg="node service registered"
time="2020-03-02T23:41:18Z" level=info msg=serving endpoint="unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock"
time="2020-03-02T23:41:18Z" level=info msg="Executing NodeGetInfo. Host name:node05"
time="2020-03-02T23:41:18Z" level=info msg="AutoProbe has not been called. Executing manual probe"
time="2020-03-02T23:41:18Z" level=info msg="Executing nodeProbe"
time="2020-03-02T23:41:18Z" level=info msg="Executing Authenticate REST client"
time="2020-03-02T23:41:21Z" level=info msg="Response code:200 for url: /api/types/loginSessionInfo"
time="2020-03-02T23:41:21Z" level=info msg="Authentication response code:200"
time="2020-03-02T23:41:21Z" level=info msg="Authentication successful"
time="2020-03-02T23:41:21Z" level=error msg="Cannot read directory: /sys/class/fc_host Error: open /sys/class/fc_host: no such file or directory"
time="2020-03-02T23:41:22Z" level=info msg="Executing NodeGetInfo. Host name:node05"

The last two lines repeats many times.

So, it looks like it may be something to do with the lack of fibre channel adapter on my worker nodes, but as I'm using the virtual storage appliance I should not need (and can not configure) FC adapters.

Is this currently a limitation of the CSI drivers? Is there a way round this? I'd appreciate any help / guidance anyone can give.

 

Many thanks

Carl

 

0 Kudos
7 Replies
bmcfeeters
3 Argentum

Re: CSI Driver for Unity VSA - unity-node problems

Hi Carl,

Unfortunately, we don't have iSCSI support in the Unity CSI driver yet.  That functionality is targetting for release in mid-April.

Thanks

Bryan

0 Kudos
nachoarrieta
2 Bronze

Re: CSI Driver for Unity VSA - unity-node problems

Hi, 

Seems iSCSI is supported now:

https://github.com/dell/csi-unity

Nacho

 

Anotación 2020-04-21 095814.png

0 Kudos
storage-dude
2 Bronze

Re: CSI Driver for Unity VSA - unity-node problems

I am having this same issue using version 1.5

We do not use FC, only iSCSI. How do I fix this problem?

Here is my output from the driver / registrar 

# kubectl -n unity logs unity-node-rkbdb -c driver
Endpoint unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock
ls: cannot access '/var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock': No such file or directory
csi-unity logger initiated. This should be called only once.
time="2021-04-01T13:47:14Z" level=info runid=start msg="Driver Mode:node"
time="2021-04-01T13:47:14Z" level=info runid=start msg="X_CSI_UNITY_NODENAME:taxmd-k8n01-v"
time="2021-04-01T13:47:14Z" level=info runid=config-0 msg="*************Synchronizing driver config**************"
time="2021-04-01T13:47:14Z" level=info msg="configured csi-unity.dellemc.com" ArrayId=apm00203025495 Insecure=true IsDefaultArray=true RestGateway="https://10.0.0.20" password="*******" username=emcapi
time="2021-04-01T13:47:14Z" level=info runid=node-0 msg="Starting goroutine to add Node information to storage array"
time="2021-04-01T13:47:14Z" level=info runid=config-0 msg="configured csi-unity.dellemc.com"
time="2021-04-01T13:47:14Z" level=info msg="identity service registered"
time="2021-04-01T13:47:14Z" level=info runid=config-1 msg="Dynamic config load goroutine invoked"
time="2021-04-01T13:47:14Z" level=info msg="node service registered"
time="2021-04-01T13:47:14Z" level=info runid=RegisterAdditionalServers msg="Registering additional GRPC servers"
time="2021-04-01T13:47:14Z" level=info msg=serving endpoint="unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock"
time="2021-04-01T13:47:15Z" level=error arrayid=apm00203025495 runid=node-2 msg="Cannot read directory: /sys/class/fc_host Error: open /sys/class/fc_host: no such file or directory"
time="2021-04-01T13:47:15Z" level=info arrayid=apm00203025495 runid=node-2 msg="'FC Initiators' cannot be retrieved."
time="2021-04-01T13:47:15Z" level=info arrayid=apm00203025495 runid=node-2 msg="URI/api/instances/host/name:taxmd-k8n01-v?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields"

# kubectl -n unity logs unity-node-rkbdb -c registrar
I0401 13:53:17.940489 1 main.go:113] Version: v2.1.0
I0401 13:53:17.940908 1 main.go:137] Attempting to open a gRPC connection with: "/csi/csi_sock"
I0401 13:53:17.940929 1 connection.go:153] Connecting to unix:///csi/csi_sock
I0401 13:53:17.941399 1 main.go:144] Calling CSI driver to discover driver name
I0401 13:53:17.941416 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0401 13:53:17.941420 1 connection.go:183] GRPC request: {}
I0401 13:53:17.943464 1 connection.go:185] GRPC response: {"manifest":{"commit":"823c1cb2fd696e996b7ad71b6dfc7e666a203ae8","formed":"Mon, 22 Mar 2021 10:42:01 UTC","semver":"1.5.0","url":"http://github.com/dell/csi-unity"},"name":"csi-unity.dellemc.com","vendor_version":"1.5.0"}
I0401 13:53:17.943521 1 connection.go:186] GRPC error: <nil>
I0401 13:53:17.943530 1 main.go:154] CSI driver name: "csi-unity.dellemc.com"
I0401 13:53:17.943577 1 node_register.go:52] Starting Registration Server at: /registration/csi-unity.dellemc.com-reg.sock
I0401 13:53:17.943666 1 node_register.go:61] Registration Server started at: /registration/csi-unity.dellemc.com-reg.sock
I0401 13:53:17.943720 1 node_register.go:83] Skipping healthz server because HTTP endpoint is set to: ""
I0401 13:53:19.071047 1 main.go:80] Received GetInfo call: &Info Request{}
I0401 13:53:22.075370 1 main.go:90] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = runid=7 The node [taxmd-k8n01-v] is not added to any of the arrays,}
E0401 13:53:22.075422 1 main.go:92] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = runid=7 The node [taxmd-k8n01-v] is not added to any of the arrays, restarting registration container.

0 Kudos
ankur.patel
DellEMC

Re: CSI Driver for Unity VSA - unity-node problems

Hi, I installed this using unity vsa and iscsi. I created a video
https://www.youtube.com/watch?v=Gw3dh7NGbCs&t=5s

Perhaps you didnt comment out FC?

storage-dude
2 Bronze

Re: CSI Driver for Unity VSA - unity-node problems

@ankur.patel Excellent video! A welcomed sanity check for myself. I was following all the same steps you show in your video. 
Unfortunately, my problem persists. 
I made a post that provides more details on my problem here

I did a watch on kubectl get pods while the deployment process was running. The controllers come up fine (I specified 2 as I have a 4 node cluster, but I also tried using 1).

The nodes will then deploy, come up as running, then error out causing a 1/2 node status.

No matter what I try the driver log show 
"ls: cannot access /var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock: No such file or directory"
and 
"Cannot read directory: /sys/class/fc_host Error: open /sys/class/fc_host: no such file or directory"

Its not clear where that directory resides. Is the pod looking at the the actual EMC array? I checked the EMC with the service account, that file does not exist. Also why is the pod even checking that? I disabled FC, I only want iSCSI.

For the registrar pod I see the following
"Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = runid=7 The node [taxmd-k8n03-v] is not added to any of the arrays,}"
"E0402 13:44:07.602419 1 main.go:92] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = runid=7 The node [taxmd-k8n03-v] is not added to any of the arrays, restarting registration container."

I cant tell which error is actually causing the pods to fail. 

Couple of notes on some things I had to do to actually get the installer to work.
- We are using an ubuntu env, the controller appears to want to use root when logging into nodes. That will not work on ubuntu. I had to adjust the sudoers file to allow a local user rights to do things, then edit the script to perform the cat command it uses with sshpass as sudo.
- The verify script would fail every time when checking the pods unless I had each pod's iscsi initiator logged into a target and a drive mapped, is this expected? 

Why is this happening? How can I trouble shoot? 

0 Kudos
storage-dude
2 Bronze

Re: CSI Driver for Unity VSA - unity-node problems

Digging deeper in the code, I can see the go code is trying to check for fiber hba.
The pods breaks because of the code behind them.
Why are the pods checking for hba's when I dont have any?
Not sure how anyone else was able to get this installed but there are no hba drivers on our nodes. This will fail every time. How do I disable this?

0 Kudos
storage-dude
2 Bronze

Re: CSI Driver for Unity VSA - unity-node problems

I got it working!
The driver pod was not failing at all (can ignore the fc_host errors), it was the registrar pod.
Problem was the Unity still had old hosts built in it from the failed attempts. Cleared those out and it finally worked. There were other issues as well, I will make a new post detailing tips and tricks for others.