13 Posts
0
5575
csi-unity does not work
Hi,
we were trying to do pure NFS share with Unity in Kubernetes 19 using csi-unity using 3 master and 3 worker nodes on bare metal.
We have installed Dell EMC UnityVSA for testing purposes before we wanted to do it in production.
We have watched videos and study this community closely but we were unable to solve the issue.
We believe there are 2 major bugs:
1) NFS is not supported without other driver
2) iSCSI driver cannot run on multiple worker nodes
Ad 1)
This person was hit with the same issue: https://www.dell.com/community/Containers/csi-unity-nfs-claim-pending/m-p/7813344 Known bugs: "For NFS volume and pod creation to succeed there must be minimum one worker node with iSCSI support and with a successful iSCSI login in to the array"
We would like to see no attempts with FC or iSCSI communication in the daemon set or the controller. We do not understand why the FC or iSCSI must be present in the configuration because it is technologically absolutely something different.
Ad 2)
The first pod in the daemon set creates Initiator in unity, but the other pod from other node tries to create initiator with the same name and fails. We have found no configuration options in any of the files used with the installation or in the running pod. Also it does not create any initiator path.
Is there any estimate when the issue #1 will be solved?
Does anybody have any hint how to solve issue #2?
Thanks
muthup7
2 Posts
1
April 19th, 2021 04:00
happy to help to setup Zoom call for this issue, we already opened engineering case as well to track engineering asking to setup discussion.
LudovitScholtz
13 Posts
0
April 28th, 2021 05:00
The NFS issue was solved by increasing required drive PVC drive size from 10M to 5G
Also, iSCSI initiators will not be created when there is no iSCSI package installed on the worker node (to be tested)
Thank you
Flo_csI
2 Intern
2 Intern
•
166 Posts
0
March 8th, 2021 01:00
Hi @LudovitScholtz,
Point 1) will be solved with next release.
On point 2) is it possible that your nodes have the same hostname ?
Can you share logs with the error ?
If you need direct assistance you can ping me directly with a private message.
Thx !
LudovitScholtz
13 Posts
0
March 8th, 2021 02:00
Hi,
hostname is different for each pod:
when I exec in each pod in deamon set, and run "cat /etc/hostname" i get different hostname.
Ad1) do you have any time estimate? Days, months, years?
Ludo
Logs from failed service is below. Log for one day of running has 1 MB, the end of the log repeats
Endpoint unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock
ls: cannot access '/var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock': No such file or directory
time="2021-03-06T19:19:49Z" level=debug msg="enabled context injector"
time="2021-03-06T19:19:49Z" level=debug msg="init req & rep validation" withSpec=false
time="2021-03-06T19:19:49Z" level=debug msg="init implicit req validation" withSpecReq=false
csi-unity logger initiated. This should be called only once.
time="2021-03-06T19:19:49Z" level=debug msg="init req validation" withSpecReq=true
time="2021-03-06T19:19:49Z" level=debug msg="enabled request ID injector"
time="2021-03-06T19:19:49Z" level=debug msg="enabled request logging"
time="2021-03-06T19:19:49Z" level=debug msg="enabled response logging"
time="2021-03-06T19:19:49Z" level=debug msg="enabled spec validator opt: request validation"
time="2021-03-06T19:19:49Z" level=debug msg="enabled serial volume access"
time="2021-03-06T19:19:49Z" level=info runid=start msg="Driver Mode:node" func="github.com/dell/csi-unity/service.(*service).BeforeServe()" file="/go/src/csi-unity/service/service.go:147"
time="2021-03-06T19:19:49Z" level=info runid=start msg="X_CSI_UNITY_NODENAME:czphahwk8s02" func="github.com/dell/csi-unity/service.(*service).BeforeServe()" file="/go/src/csi-unity/service/service.go:154"
time="2021-03-06T19:19:49Z" level=debug runid=start msg="SyncNodeInfoTimeInterval 15" func="github.com/dell/csi-unity/service.(*service).BeforeServe()" file="/go/src/csi-unity/service/service.go:163"
time="2021-03-06T19:19:49Z" level=info runid=config-0 msg="*************Synchronizing driver config**************" func="github.com/dell/csi-unity/service.(*service).syncDriverConfig()" file="/go/src/csi-unity/service/service.go:381"
time="2021-03-06T19:19:49Z" level=debug runid=config-0 msg="unity client init" func="github.com/dell/gounity.NewClientWithArgs()" file="dell/gounity@v1.4.0/unityclient.go:160"
time="2021-03-06T19:19:49Z" level=info msg="configured csi-unity.dellemc.com" ArrayId=virt2109yfhrla Insecure=true IsDefaultArray=false RestGateway="https://192.168.48.200" password="*******" username=testk8s
time="2021-03-06T19:19:49Z" level=info runid=node-0 msg="Starting goroutine to add Node information to storage array" func="github.com/dell/csi-unity/service.(*service).syncNodeInfoRoutine()" file="/go/src/csi-unity/service/node.go:1610"
time="2021-03-06T19:19:49Z" level=debug runid=node-0 msg="Config change identified. Adding node info" func="github.com/dell/csi-unity/service.(*service).syncNodeInfoRoutine()" file="/go/src/csi-unity/service/node.go:1614"
time="2021-03-06T19:19:49Z" level=info runid=config-0 msg="configured csi-unity.dellemc.com" func="github.com/dell/csi-unity/service.(*service).BeforeServe.func1()" file="/go/src/csi-unity/service/service.go:142"
time="2021-03-06T19:19:49Z" level=info msg="identity service registered"
time="2021-03-06T19:19:49Z" level=info msg="node service registered"
time="2021-03-06T19:19:49Z" level=info msg=serving endpoint="unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock"
time="2021-03-06T19:19:49Z" level=info runid=config-1 msg="Dynamic config load goroutine invoked" func="github.com/dell/csi-unity/service.(*service).loadDynamicConfig()" file="/go/src/csi-unity/service/service.go:315"
time="2021-03-06T19:19:49Z" level=debug runid=config-1 msg="Config folder:/unity-config" func="github.com/dell/csi-unity/service.(*service).loadDynamicConfig()" file="/go/src/csi-unity/service/service.go:320"
time="2021-03-06T19:19:49Z" level=debug runid=node-0 msg="Synchronizing Node Info" func="github.com/dell/csi-unity/service.(*service).syncNodeInfo()" file="/go/src/csi-unity/service/node.go:1649"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Probing controller service automatically" func="github.com/dell/csi-unity/service.(*service).requireProbe()" file="/go/src/csi-unity/service/service.go:617"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Inside Controller Probe" func="github.com/dell/csi-unity/service.(*service).probe()" file="/go/src/csi-unity/service/service.go:654"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Executing Authenticate REST client" func="github.com/dell/gounity.(*Client).Authenticate()" file="dell/gounity@v1.4.0/unityclient.go:50"
time="2021-03-06T19:19:49Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0001: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:19:49Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0001: Name=csi-unity.dellemc.com, VendorVersion=1.4.0, Manifest=map[commit:715f0326bbcd619b4672457e4c108438cd8667ae formed:Tue, 15 Dec 2020 10:25:07 UTC semver:1.4.0 url:http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/types/loginSessionInfo" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Authentication response code: 200" func="github.com/dell/gounity.(*Client).Authenticate()" file="dell/gounity@v1.4.0/unityclient.go:64"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Authentication successful" func="github.com/dell/gounity.(*Client).Authenticate()" file="dell/gounity@v1.4.0/unityclient.go:74"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Controller Probe Success" func="github.com/dell/csi-unity/service.singleArrayProbe()" file="/go/src/csi-unity/service/service.go:645"
time="2021-03-06T19:19:49Z" level=error arrayid=virt2109yfhrla runid=node-2 msg="Cannot read directory: /sys/class/fc_host Error: open /sys/class/fc_host: no such file or directory" func="github.com/dell/csi-unity/service/utils.GetFCInitiators()" file="/go/src/csi-unity/service/utils/emcutils.go:97"
time="2021-03-06T19:19:49Z" level=info arrayid=virt2109yfhrla runid=node-2 msg="'FC Initiators' cannot be retrieved." func="github.com/dell/csi-unity/service.(*service).addNodeInformationIntoArray()" file="/go/src/csi-unity/service/node.go:1397"
time="2021-03-06T19:19:49Z" level=info arrayid=virt2109yfhrla runid=node-2 msg="URI/api/instances/host/name:czphahwk8s02?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*host).FindHostByName()" file="dell/gounity@v1.4.0/host.go:41"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/instances/host/name:czphahwk8s02?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/instances/host/name:czphahwk8s02?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Execution successful on Method: GET, URI: /api/instances/host/name:czphahwk8s02?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Host czphahwk8s02 exists on the array" func="github.com/dell/csi-unity/service.(*service).addNodeInformationIntoArray()" file="/go/src/csi-unity/service/node.go:1434"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Adding iSCSI Initiator: Host_18 to host: iqn.1993-08.org.debian:01:af5bf2af245 " func="github.com/dell/csi-unity/service.(*service).addNodeInformationIntoArray()" file="/go/src/csi-unity/service/node.go:1471"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Finding Initiator: iqn.1993-08.org.debian:01:af5bf2af245" func="github.com/dell/gounity.(*host).CreateHostInitiator()" file="dell/gounity@v1.4.0/host.go:183"
time="2021-03-06T19:19:49Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/types/hostInitiator/instances?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-06T19:19:50Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/types/hostInitiator/instances?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-06T19:19:50Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Execution successful on Method: GET, URI: /api/types/hostInitiator/instances?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-06T19:19:50Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="FindHostInitiatorByName: &{ {HostInitiator_14 {10 [ALRT_INITIATOR_NO_LOGGED_IN_PATH] [The initiator does not have any logged in initiator paths. Check the connection between the initiator and the storage system.]} 2 iqn.1993-08.org.debian:01:af5bf2af245 false {Host_16 [] [] [] } []}} Error: " func="github.com/dell/gounity.(*host).CreateHostInitiator()" file="dell/gounity@v1.4.0/host.go:185"
time="2021-03-06T19:19:50Z" level=debug runid=node-2 msg="Adding node [virt2109yfhrla] failed, Error: [rpc error: code = Internal desc = runid= Adding iSCSI initiator error: Initiator found (iqn.1993-08.org.debian:01:af5bf2af245), and attached to someother host (Host_16) instead of host: Host_18]" func="github.com/dell/csi-unity/service.(*service).syncNodeInfo.func2.1()" file="/go/src/csi-unity/service/node.go:1661"
time="2021-03-06T19:19:50Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0002: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:19:50Z" level=debug runid=2 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778"
time="2021-03-06T19:19:53Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0002: rpc error: code = Unavailable desc = runid=2 The node [czphahwk8s02] is not added to any of the arrays"
time="2021-03-06T19:19:54Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0003: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:19:54Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0003: Name=csi-unity.dellemc.com, VendorVersion=1.4.0, Manifest=map[commit:715f0326bbcd619b4672457e4c108438cd8667ae formed:Tue, 15 Dec 2020 10:25:07 UTC semver:1.4.0 url: http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:19:55Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0004: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:19:55Z" level=debug runid=4 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778"
time="2021-03-06T19:19:58Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0004: rpc error: code = Unavailable desc = runid=4 The node [czphahwk8s02] is not added to any of the arrays"
time="2021-03-06T19:20:13Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0005: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:20:13Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0005: Name=csi-unity.dellemc.com, VendorVersion=1.4.0, Manifest=map[commit:715f0326bbcd619b4672457e4c108438cd8667ae formed:Tue, 15 Dec 2020 10:25:07 UTC semver:1.4.0 url: http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:20:14Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0006: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:20:14Z" level=debug runid=6 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778"
time="2021-03-06T19:20:17Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0006: rpc error: code = Unavailable desc = runid=6 The node [czphahwk8s02] is not added to any of the arrays"
time="2021-03-06T19:20:44Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0007: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-06T19:20:44Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0007: Name=csi-unity.dellemc.com, VendorVersion=1.4.0, Manifest=map[commit:715f0326bbcd619b4672457e4c108438cd8667ae formed:Tue, 15 Dec 2020 10:25:07 UTC semver:1.4.0 url: http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
Flo_csI
2 Intern
2 Intern
•
166 Posts
0
March 9th, 2021 00:00
1) every Dell CSI driver has a quarterly release pace. So it should be a matter of weeks before getting that fix.
2) can you compare the values of /etc/iscsi/initiatorname.iscsi from your two servers ? Are they the same ? If so you can generate a new name with iscsi-iname > /etc/iscsi/initiatorname.iscsi ; or simply uninstall / install the package iscsi-initiator-utils.
LudovitScholtz
13 Posts
0
March 11th, 2021 14:00
Hi, indeed the files you are refering are the same on all 3 worker nodes as well as 3 master nodes
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator. The InitiatorName must be unique
## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:af5bf2af245
We are on ubuntu 20.04 environment and i dont see package iscsi-initiator-utils
However we will try the "iscsi-iname > /etc/iscsi/initiatorname.iscsi" fix ..
There should be probably some notion in the documentation about requirements of the utils on the worker/master nodes perhaps?
LudovitScholtz
13 Posts
0
March 11th, 2021 15:00
Great news, we are one step closer to the solution. The initiators issue has been fixed.
The problem is that it is still not working. I performed uninstall and install of the csi driver, but after i map the drive it does not create it..
I have allowed both nfs and iscsi.
I think the error is on this line: time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="FindHostInitiatorByName: &{ {HostInitiator_14 {10 [ALRT_INITIATOR_NO_LOGGED_IN_PATH] [The initiator does not have any logged in initiator paths. Check the connection between the initiator and the storage system.]} 2 iqn.1993-08.org.debian:01:af5bf2af245 false {Host_16 [] [] [] } []}} Error: " func="github.com/dell/gounity.(*host).CreateHostInitiator()" file="dell/gounity@v1.4.0/host.go:185"
Log from the daemon set pod:
Endpoint unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock
ls: cannot access '/var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock': No such file or directory
time="2021-03-11T22:58:17Z" level=debug msg="enabled context injector"
time="2021-03-11T22:58:17Z" level=debug msg="init req & rep validation" withSpec=false
time="2021-03-11T22:58:17Z" level=debug msg="init implicit req validation" withSpecReq=false
time="2021-03-11T22:58:17Z" level=debug msg="init req validation" withSpecReq=true
time="2021-03-11T22:58:17Z" level=debug msg="enabled request ID injector"
time="2021-03-11T22:58:17Z" level=debug msg="enabled request logging"
time="2021-03-11T22:58:17Z" level=debug msg="enabled response logging"
time="2021-03-11T22:58:17Z" level=debug msg="enabled spec validator opt: request validation"
time="2021-03-11T22:58:17Z" level=debug msg="enabled serial volume access"
csi-unity logger initiated. This should be called only once.
time="2021-03-11T22:58:17Z" level=info runid=start msg="Driver Mode:node" func="github.com/dell/csi-unity/service.(*service).BeforeServe()" file="/go/src/csi-unity/service/service.go:147"
time="2021-03-11T22:58:17Z" level=info runid=start msg="X_CSI_UNITY_NODENAME:czphahwk8s01" func="github.com/dell/csi-unity/service.(*service).BeforeServe()" file="/go/src/csi-unity/service/service.go:154"
time="2021-03-11T22:58:17Z" level=debug runid=start msg="SyncNodeInfoTimeInterval 15" func="github.com/dell/csi-unity/service.(*service).BeforeServe()" file="/go/src/csi-unity/service/service.go:163"
time="2021-03-11T22:58:17Z" level=info runid=config-0 msg="*************Synchronizing driver config**************" func="github.com/dell/csi-unity/service.(*service).syncDriverConfig()" file="/go/src/csi-unity/service/service.go:381"
time="2021-03-11T22:58:17Z" level=debug runid=config-0 msg="unity client init" func="github.com/dell/gounity.NewClientWithArgs()" file="dell/gounity@v1.4.0/unityclient.go:160"
time="2021-03-11T22:58:17Z" level=info msg="configured csi-unity.dellemc.com" ArrayId=virt2109yfhrla Insecure=true IsDefaultArray=false RestGateway="https://192.168.48.200" password="*******" username=testk8s
time="2021-03-11T22:58:17Z" level=info runid=node-0 msg="Starting goroutine to add Node information to storage array" func="github.com/dell/csi-unity/service.(*service).syncNodeInfoRoutine()" file="/go/src/csi-unity/service/node.go:1610"
time="2021-03-11T22:58:17Z" level=info runid=config-1 msg="Dynamic config load goroutine invoked" func="github.com/dell/csi-unity/service.(*service).loadDynamicConfig()" file="/go/src/csi-unity/service/service.go:315"
time="2021-03-11T22:58:17Z" level=debug runid=node-0 msg="Config change identified. Adding node info" func="github.com/dell/csi-unity/service.(*service).syncNodeInfoRoutine()" file="/go/src/csi-unity/service/node.go:1614"
time="2021-03-11T22:58:17Z" level=debug runid=node-0 msg="Synchronizing Node Info" func="github.com/dell/csi-unity/service.(*service).syncNodeInfo()" file="/go/src/csi-unity/service/node.go:1649"
time="2021-03-11T22:58:17Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Probing controller service automatically" func="github.com/dell/csi-unity/service.(*service).requireProbe()" file="/go/src/csi-unity/service/service.go:617"
time="2021-03-11T22:58:17Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Inside Controller Probe" func="github.com/dell/csi-unity/service.(*service).probe()" file="/go/src/csi-unity/service/service.go:654"
time="2021-03-11T22:58:17Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Executing Authenticate REST client" func="github.com/dell/gounity.(*Client).Authenticate()" file="dell/gounity@v1.4.0/unityclient.go:50"
time="2021-03-11T22:58:17Z" level=info runid=config-0 msg="configured csi-unity.dellemc.com" func="github.com/dell/csi-unity/service.(*service).BeforeServe.func1()" file="/go/src/csi-unity/service/service.go:142"
time="2021-03-11T22:58:17Z" level=info msg="identity service registered"
time="2021-03-11T22:58:17Z" level=info msg="node service registered"
time="2021-03-11T22:58:17Z" level=info msg=serving endpoint="unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock"
time="2021-03-11T22:58:17Z" level=debug runid=config-1 msg="Config folder:/unity-config" func="github.com/dell/csi-unity/service.(*service).loadDynamicConfig()" file="/go/src/csi-unity/service/service.go:320"
time="2021-03-11T22:58:17Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0001: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-11T22:58:17Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0001: Name=csi-unity.dellemc.com, VendorVersion=1.4.0, Manifest=map[commit:715f0326bbcd619b4672457e4c108438cd8667ae formed:Tue, 15 Dec 2020 10:25:07 UTC semver:1.4.0 url:http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-11T22:58:18Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0002: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-03-11T22:58:18Z" level=debug runid=2 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/types/loginSessionInfo" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Authentication response code: 200" func="github.com/dell/gounity.(*Client).Authenticate()" file="dell/gounity@v1.4.0/unityclient.go:64"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Authentication successful" func="github.com/dell/gounity.(*Client).Authenticate()" file="dell/gounity@v1.4.0/unityclient.go:74"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Controller Probe Success" func="github.com/dell/csi-unity/service.singleArrayProbe()" file="/go/src/csi-unity/service/service.go:645"
time="2021-03-11T22:58:18Z" level=error arrayid=virt2109yfhrla runid=node-2 msg="Cannot read directory: /sys/class/fc_host Error: open /sys/class/fc_host: no such file or directory" func="github.com/dell/csi-unity/service/utils.GetFCInitiators()" file="/go/src/csi-unity/service/utils/emcutils.go:97"
time="2021-03-11T22:58:18Z" level=info arrayid=virt2109yfhrla runid=node-2 msg="'FC Initiators' cannot be retrieved." func="github.com/dell/csi-unity/service.(*service).addNodeInformationIntoArray()" file="/go/src/csi-unity/service/node.go:1397"
time="2021-03-11T22:58:18Z" level=info arrayid=virt2109yfhrla runid=node-2 msg="URI/api/instances/host/name:czphahwk8s01?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*host).FindHostByName()" file="dell/gounity@v1.4.0/host.go:41"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/instances/host/name:czphahwk8s01?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/instances/host/name:czphahwk8s01?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Execution successful on Method: GET, URI: /api/instances/host/name:czphahwk8s01?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Host czphahwk8s01 exists on the array" func="github.com/dell/csi-unity/service.(*service).addNodeInformationIntoArray()" file="/go/src/csi-unity/service/node.go:1434"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/instances/hostInitiator/HostInitiator_14?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/instances/hostInitiator/HostInitiator_14?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Execution successful on Method: GET, URI: /api/instances/hostInitiator/HostInitiator_14?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-11T22:58:18Z" level=info arrayid=virt2109yfhrla runid=node-2 msg="Node initiators are synchronized with the Host Wwns on the array" func="github.com/dell/csi-unity/service.(*service).checkHostIdempotency()" file="/go/src/csi-unity/service/node.go:1544"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Adding iSCSI Initiator: Host_16 to host: iqn.1993-08.org.debian:01:af5bf2af245 " func="github.com/dell/csi-unity/service.(*service).addNodeInformationIntoArray()" file="/go/src/csi-unity/service/node.go:1471"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Finding Initiator: iqn.1993-08.org.debian:01:af5bf2af245" func="github.com/dell/gounity.(*host).CreateHostInitiator()" file="dell/gounity@v1.4.0/host.go:183"
time="2021-03-11T22:58:18Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/types/hostInitiator/instances?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/types/hostInitiator/instances?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Execution successful on Method: GET, URI: /api/types/hostInitiator/instances?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="FindHostInitiatorByName: &{ {HostInitiator_14 {10 [ALRT_INITIATOR_NO_LOGGED_IN_PATH] [The initiator does not have any logged in initiator paths. Check the connection between the initiator and the storage system.]} 2 iqn.1993-08.org.debian:01:af5bf2af245 false {Host_16 [] [] [] } []}} Error: " func="github.com/dell/gounity.(*host).CreateHostInitiator()" file="dell/gounity@v1.4.0/host.go:185"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Initiator found and already added to existing host Initiator: iqn.1993-08.org.debian:01:af5bf2af245 to host: Host_16
" func="github.com/dell/gounity.(*host).CreateHostInitiator()" file="dell/gounity@v1.4.0/host.go:208"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/instances/hostIPPort/HostNetworkAddress_31?fields=id,address" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/instances/hostIPPort/HostNetworkAddress_31?fields=id,address" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Execution successful on Method: GET, URI: /api/instances/hostIPPort/HostNetworkAddress_31?fields=id,address" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/instances/hostIPPort/HostNetworkAddress_32?fields=id,address" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/instances/hostIPPort/HostNetworkAddress_32?fields=id,address" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Execution successful on Method: GET, URI: /api/instances/hostIPPort/HostNetworkAddress_32?fields=id,address" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="copied 41 bytes to /etc/multipath.conf" func="github.com/dell/csi-unity/service.(*service).copyMultipathConfigFile()" file="/go/src/csi-unity/service/node.go:1259"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="URI: /api/types/ipInterface/instances?fields=id,ipAddress,type" func="github.com/dell/gounity.(*ipinterface).ListIscsiIPInterfaces()" file="dell/gounity@v1.4.0/ipinterface.go:32"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/types/ipInterface/instances?fields=id,ipAddress,type" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Response code:200 for url: /api/types/ipInterface/instances?fields=id,ipAddress,type" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Execution successful on Method: GET, URI: /api/types/ipInterface/instances?fields=id,ipAddress,type" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Valid IPs: []" func="github.com/dell/csi-unity/service.(*service).iScsiDiscoverAndLogin()" file="/go/src/csi-unity/service/node.go:1156"
time="2021-03-11T22:58:19Z" level=debug arrayid=virt2109yfhrla runid=node-2 msg="Completed discovery and rescan of all IP Interfaces" func="github.com/dell/csi-unity/service.(*service).iScsiDiscoverAndLogin()" file="/go/src/csi-unity/service/node.go:1180"
time="2021-03-11T22:58:19Z" level=debug runid=node-2 msg="Node [virt2109yfhrla] Added successfully" func="github.com/dell/csi-unity/service.(*service).syncNodeInfo.func2.1()" file="/go/src/csi-unity/service/node.go:1659"
time="2021-03-11T22:58:21Z" level=info runid=2 msg="NodeGetInfo success" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:792"
time="2021-03-11T22:58:21Z" level=error runid=2 msg="Cannot read directory: /sys/class/fc_host Error: open /sys/class/fc_host: no such file or directory" func="github.com/dell/csi-unity/service/utils.GetFCInitiators()" file="/go/src/csi-unity/service/utils/emcutils.go:97"
time="2021-03-11T22:58:21Z" level=info runid=2 msg="iSCSI/FC package found in this node proceeding to further validations" func="github.com/dell/csi-unity/service.(*service).validateProtocols()" file="/go/src/csi-unity/service/node.go:1694"
time="2021-03-11T22:58:21Z" level=info arrayid=virt2109yfhrla runid=2 msg="URI/api/instances/host/name:czphahwk8s01?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*host).FindHostByName()" file="dell/gounity@v1.4.0/host.go:41"
time="2021-03-11T22:58:21Z" level=debug arrayid=virt2109yfhrla runid=2 msg="Invoking REST API server info Method: GET, URI: /api/instances/host/name:czphahwk8s01?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-11T22:58:21Z" level=debug arrayid=virt2109yfhrla runid=2 msg="Response code:200 for url: /api/instances/host/name:czphahwk8s01?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-11T22:58:21Z" level=debug arrayid=virt2109yfhrla runid=2 msg="Execution successful on Method: GET, URI: /api/instances/host/name:czphahwk8s01?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-11T22:58:21Z" level=info runid=2 msg="Got iSCSI Initiators, Checking health of initiators:[{HostInitiator_14}]" func="github.com/dell/csi-unity/service.(*service).validateProtocols()" file="/go/src/csi-unity/service/node.go:1728"
time="2021-03-11T22:58:21Z" level=debug arrayid=virt2109yfhrla runid=2 msg="Invoking REST API server info Method: GET, URI: /api/instances/hostInitiator/HostInitiator_14?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:105"
time="2021-03-11T22:58:21Z" level=debug arrayid=virt2109yfhrla runid=2 msg="Response code:200 for url: /api/instances/hostInitiator/HostInitiator_14?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity@v1.4.0/api/restclient.go:290"
time="2021-03-11T22:58:21Z" level=debug arrayid=virt2109yfhrla runid=2 msg="Execution successful on Method: GET, URI: /api/instances/hostInitiator/HostInitiator_14?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity@v1.4.0/unityclient.go:108"
time="2021-03-11T22:58:21Z" level=info runid=2 msg="iSCSI Health is bad for array:virt2109yfhrla, Health:ALRT_INITIATOR_NO_LOGGED_IN_PATH" func="github.com/dell/csi-unity/service.(*service).validateProtocols()" file="/go/src/csi-unity/service/node.go:1741"
time="2021-03-11T22:58:21Z" level=debug runid=2 msg="Topology Keys--->map[]" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:796"
time="2021-03-11T22:58:21Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0002: NodeId=czphahwk8s01,czphahwk8s01, MaxVolumesPerNode=0, XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
Flo_csI
2 Intern
2 Intern
•
166 Posts
0
March 12th, 2021 06:00
The package name on ubuntu is open-iscsi.
If I may recommend to :
1. uninstall the driver
2. remove all entries host and initiator entries in unity
3. uninstall open-iscsi
4. reinstall open-iscsi
5. reinstall the driver and try again
If the same error persist, try to check the logs from the iscsi daemon (journalctl -u iscsi or similar for ubuntu).
You can also run iscsiadm -m session ; you should see an established tcp connection.
Let me know how it works and if you need a Zoom session ; please contact me in PM
LudovitScholtz
13 Posts
0
April 9th, 2021 07:00
Hi, we have removed everything and installed it again. We are now using v 1.5.
result 1: NFS does not work at all. It does not even log anything
result 2: we do not want iscsi, but it is only thing that shows some activity between servers and unity, and it also does not work
Error logs: https://apps.scholtz.sk/error-logs-unity-csi-2021-04-09.zip
I would like to pinpoint probably the main error:
time="2021-04-08T11:22:31Z" level=info runid=3 msg="iSCSI Health is bad for array:virt2109yfhrla, Health:ALRT_INITIATOR_NO_LOGGED_IN_PATH"
3 hosts are connected, then 3 initiators are created, then it fails because initiator does not have any initiator path.
So the first question. How can we set it up with NFS? There is no tutorial for this nor the documentation is clear enough. The obvious configurations does not work.
Second question: What are we missing with the initiator path? Should we create initiator path manually or should the driver create it? The only difference with this guide i think is that we are using 3 master and 3 worker nodes (also we use now v1.5). https://www.youtube.com/watch?v=Gw3dh7NGbCs in the video is only single host
Thanks for the response
Thar_J
42 Posts
0
April 12th, 2021 03:00
Hi
With respect to Unity 1.5 driver only NFS protocol should work without any dependencies of iscsi.
Kindly try to create pvc , pod with NFS protocol and let us know the updates.
Kindly make the csiDebug variable true in myvalues.yaml to get the detailed debug logs and share the logs as well.
LudovitScholtz
13 Posts
0
April 14th, 2021 02:00
We are testing it with stateful set
This does not work.. Does not log anything as provided above
Note that NFS as direct storage works..
Flo_csI
2 Intern
2 Intern
•
166 Posts
0
April 14th, 2021 06:00
Hi @LudovitScholtz,
Can you contact me in PM and share your contact details ?
It will be easier to share logs and continue the debugging together.
Thanks !
LudovitScholtz
13 Posts
1
April 15th, 2021 05:00
I have share logs few messages above apps.scholtz.sk/error-logs-unity-csi-2021-04-09.zip
I believe that solving this issue in the public thread might be beneficial for the community..
Is there anything else I may provide to resolve the issue?
Thanks
Thar_J
42 Posts
0
April 15th, 2021 09:00
Hi @LudovitScholtz
Thanks for Sharing the Logs, Let us analyze the Logs and get back to you.
If there is anything more i need , will definitely ask you.
We will definitely analyze the Logs to provide a solution.
Regards
Thar_J
Thar_J
42 Posts
0
April 15th, 2021 23:00
Hi @LudovitScholtz
After Analyzing the logs We could see that there is a problem with respect to the API calling and looks like it is an environment issue.
failed to invoke Unity REST API server.
We feel like the ticket description says csi-unity does not work , it would be appropriate if you could kindly change the description to Unable to create NFS volume.
Thanks
Thar_J