csi-unity crashloop due to "fc_host: no such file or directory"

Hi @jgazeley,

This is a "normal" error, I requested to reclassify it as a warning.

Do you have more logs on the crashing container ?

Can you make sure you have the csiDebug set to true : https://github.com/dell/csi-unity/blob/master/helm/csi-unity/values.yaml#L16

Thanks.

Hi @Flo_csI 

Thanks for your reply. I set X_CSI_UNITY_DEBUG to true in OperatorHub and redeployed. This is the config I'm running now, which is 99% standard except I removed the storageClass for FibreChannel:

apiVersion: storage.dell.com/v1
kind: CSIUnity
metadata:
  annotations:
    storage.dell.com/snapshotter.Image: 'k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0'
    storage.dell.com/resizer.Image: 'quay.io/k8scsi/csi-resizer:v1.1.0'
    storage.dell.com/provisioner.Image: 'k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0'
    storage.dell.com/attacher.Image: 'k8s.gcr.io/sig-storage/csi-attacher:v3.1.0'
    storage.dell.com/snapshotter.Image.IsDefault: 'true'
    storage.dell.com/resizer.Image.IsDefault: 'true'
    storage.dell.com/provisioner.Image.IsDefault: 'true'
    storage.dell.com/attacher.Image.IsDefault: 'true'
    storage.dell.com/registrar.Image: 'k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0'
    storage.dell.com/registrar.Image.IsDefault: 'true'
    storage.dell.com/CSIDriverConfigVersion: v4
  selfLink: >-
    /apis/storage.dell.com/v1/namespaces/openshift-operators/csiunities/test-unity2
  resourceVersion: '76868924'
  name: test-unity2
  uid: 7acc02e6-9ef4-4755-982d-ae559d399fb9
  creationTimestamp: '2021-04-13T13:53:37Z'
  generation: 5
  managedFields:
    - apiVersion: storage.dell.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          .: {}
          'f:driver':
            .: {}
            'f:common':
              .: {}
              'f:envs': {}
              'f:image': {}
              'f:imagePullPolicy': {}
            'f:configVersion': {}
            'f:replicas': {}
            'f:snapshotClass': {}
      manager: Mozilla
      operation: Update
      time: '2021-04-13T13:53:37Z'
    - apiVersion: storage.dell.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:storage.dell.com/resizer.Image.IsDefault': {}
            'f:storage.dell.com/provisioner.Image.IsDefault': {}
            'f:storage.dell.com/registrar.Image': {}
            'f:storage.dell.com/CSIDriverConfigVersion': {}
            .: {}
            'f:storage.dell.com/registrar.Image.IsDefault': {}
            'f:storage.dell.com/attacher.Image': {}
            'f:storage.dell.com/snapshotter.Image': {}
            'f:storage.dell.com/resizer.Image': {}
            'f:storage.dell.com/provisioner.Image': {}
            'f:storage.dell.com/attacher.Image.IsDefault': {}
            'f:storage.dell.com/snapshotter.Image.IsDefault': {}
          'f:finalizers': {}
        'f:spec':
          'f:driver':
            'f:controller': {}
            'f:node': {}
            'f:sideCars': {}
            'f:storageClass': {}
        'f:status':
          .: {}
          'f:controllerStatus':
            .: {}
            'f:available': {}
          'f:driverHash': {}
          'f:lastUpdate':
            .: {}
            'f:condition': {}
            'f:time': {}
          'f:nodeStatus':
            .: {}
            'f:stopped': {}
          'f:state': {}
      manager: manager
      operation: Update
      time: '2021-04-27T09:59:38Z'
  namespace: openshift-operators
  finalizers:
    - finalizer.dell.emc.com
spec:
  driver:
    common:
      envs:
        - name: X_CSI_UNITY_DEBUG
          value: 'true'
        - name: X_CSI_UNITY_ALLOW_MULTI_POD_ACCESS
          value: 'false'
      image: 'dellemc/csi-unity:v1.5.0'
      imagePullPolicy: IfNotPresent
    configVersion: v4
    controller: {}
    node: {}
    replicas: 2
    sideCars:
      - args:
          - '--volume-name-prefix=csiunity'
        image: 'k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0'
        imagePullPolicy: IfNotPresent
        name: provisioner
      - args:
          - '--snapshot-name-prefix=csiunitysnap'
        image: 'k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0'
        imagePullPolicy: IfNotPresent
        name: snapshotter
      - image: 'k8s.gcr.io/sig-storage/csi-attacher:v3.1.0'
        imagePullPolicy: IfNotPresent
        name: attacher
      - image: 'k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0'
        imagePullPolicy: IfNotPresent
        name: registrar
      - image: 'quay.io/k8scsi/csi-resizer:v1.1.0'
        imagePullPolicy: IfNotPresent
        name: resizer
    snapshotClass:
      - name: test-snap
        parameters:
          retentionDuration: ''
    storageClass:
      - allowVolumeExpansion: true
        name: ckm00201700161-iscsi
        parameters:
          arrayId: CKM00201700161
          protocol: iSCSI
          storagePool: pool_1
        reclaimPolicy: Delete
        volumeBindingMode: Immediate
      - allowVolumeExpansion: true
        name: ckm00201700161-nfs
        parameters:
          arrayId: CKM00201700161
          hostIoSize: '8192'
          nasServer: nas_1
          protocol: NFS
          storagePool: pool_1
        reclaimPolicy: Delete
        volumeBindingMode: Immediate
      - allowVolumeExpansion: true
        allowedTopologies:
          - matchLabelExpressions:
              - key: csi-unity.dellemc.com/ckm00201700161-iscsi
                values:
                  - 'true'
        name: ckm00201700161-iscsi-topology
        parameters:
          arrayId: CKM00201700161
          protocol: iSCSI
          storagePool: pool_1
        reclaimPolicy: Delete
        volumeBindingMode: WaitForFirstConsumer
status:
  controllerStatus:
    available:
      - unity-controller-768d787658-f75wk
      - unity-controller-768d787658-l4x8j
  driverHash: 3990132907
  lastUpdate:
    condition: Succeeded
    time: '2021-04-27T09:58:44Z'
  nodeStatus:
    stopped:
      - unity-node
  state: Succeeded

So my node pods all got redeployed and immediately started crashlooping:

[jgazeley@hephaestos okd4_cluster_builder]$ oc get po
NAME                                                   READY   STATUS              RESTARTS   AGE
dell-csi-operator-controller-manager-945c46857-xrwzp   1/1     Running             7          14d
test-pvc-pod                                           0/1     ContainerCreating   0          6d
unity-controller-768d787658-f75wk                      5/5     Running             0          5m45s
unity-controller-768d787658-l4x8j                      5/5     Running             0          5m41s
unity-node-79mt4                                       1/2     CrashLoopBackOff    5          5m10s
unity-node-7t8wq                                       1/2     CrashLoopBackOff    5          5m10s
unity-node-qbmk8                                       1/2     CrashLoopBackOff    5          5m7s
unity-node-qcs9m                                       1/2     CrashLoopBackOff    5          5m5s
unity-node-rxcwr                                       1/2     CrashLoopBackOff    5          5m10s

Here are the logs from one of the pods:

[jgazeley@hephaestos okd4_cluster_builder]$ oc logs unity-node-79mt4 driver
Endpoint unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock
ls: cannot access '/var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock': No such file or directory
time="2021-04-27T09:58:47Z" level=debug msg="enabled context injector"
time="2021-04-27T09:58:47Z" level=debug msg="init req & rep validation" withSpec=false
time="2021-04-27T09:58:47Z" level=debug msg="init implicit req validation" withSpecReq=false
time="2021-04-27T09:58:47Z" level=debug msg="init req validation" withSpecReq=true
time="2021-04-27T09:58:47Z" level=debug msg="enabled request ID injector"
time="2021-04-27T09:58:47Z" level=debug msg="enabled request logging"
time="2021-04-27T09:58:47Z" level=debug msg="enabled response logging"
time="2021-04-27T09:58:47Z" level=debug msg="enabled spec validator opt: request validation"
time="2021-04-27T09:58:47Z" level=debug msg="enabled serial volume access"
csi-unity logger initiated. This should be called only once.
time="2021-04-27T09:58:47Z" level=info  runid=start msg="Driver Mode:node" func="github.com/dell/csi-unity/service.(*service).BeforeServe()" file="/go/src/csi-unity/service/service.go:152"
time="2021-04-27T09:58:47Z" level=info  runid=start msg="X_CSI_UNITY_NODENAME:okd-worker-7itgdxajnoy5zqqjccfxnh-3" func="github.com/dell/csi-unity/service.(*service).BeforeServe()" file="/go/src/csi-unity/service/service.go:159"
time="2021-04-27T09:58:47Z" level=debug  runid=start msg="SyncNodeInfoTimeInterval 15" func="github.com/dell/csi-unity/service.(*service).BeforeServe()" file="/go/src/csi-unity/service/service.go:168"
time="2021-04-27T09:58:47Z" level=info  runid=config-0 msg="*************Synchronizing driver config**************" func="github.com/dell/csi-unity/service.(*service).syncDriverConfig()" file="/go/src/csi-unity/service/service.go:398"
time="2021-04-27T09:58:47Z" level=debug  runid=config-0 msg="unity client init" func="github.com/dell/gounity.NewClientWithArgs()" file="dell/gounity/unityclient.go:160"
time="2021-04-27T09:58:47Z" level=info msg="configured csi-unity.dellemc.com" ArrayId=ckm00201700161 Insecure=true IsDefaultArray=true RestGateway="https://staas-bs01.server.lan" password="*******" username=unity
time="2021-04-27T09:58:47Z" level=info  runid=node-0 msg="Starting goroutine to add Node information to storage array" func="github.com/dell/csi-unity/service.(*service).syncNodeInfoRoutine()" file="/go/src/csi-unity/service/node.go:1610"
time="2021-04-27T09:58:47Z" level=info  runid=config-1 msg="Dynamic config load goroutine invoked" func="github.com/dell/csi-unity/service.(*service).loadDynamicConfig()" file="/go/src/csi-unity/service/service.go:332"
time="2021-04-27T09:58:47Z" level=debug  runid=config-1 msg="Config folder:/unity-config" func="github.com/dell/csi-unity/service.(*service).loadDynamicConfig()" file="/go/src/csi-unity/service/service.go:337"
time="2021-04-27T09:58:47Z" level=debug  runid=node-0 msg="Config change identified. Adding node info" func="github.com/dell/csi-unity/service.(*service).syncNodeInfoRoutine()" file="/go/src/csi-unity/service/node.go:1614"
time="2021-04-27T09:58:47Z" level=debug  runid=node-0 msg="Synchronizing Node Info" func="github.com/dell/csi-unity/service.(*service).syncNodeInfo()" file="/go/src/csi-unity/service/node.go:1649"
time="2021-04-27T09:58:47Z" level=info  runid=config-0 msg="configured csi-unity.dellemc.com" func="github.com/dell/csi-unity/service.(*service).BeforeServe.func1()" file="/go/src/csi-unity/service/service.go:147"
time="2021-04-27T09:58:47Z" level=info msg="identity service registered"
time="2021-04-27T09:58:47Z" level=info msg="node service registered"
time="2021-04-27T09:58:47Z" level=info  runid=RegisterAdditionalServers msg="Registering additional GRPC servers" func="github.com/dell/csi-unity/service.(*service).RegisterAdditionalServers()" file="/go/src/csi-unity/service/service.go:243"
time="2021-04-27T09:58:47Z" level=info msg=serving endpoint="unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sock"
time="2021-04-27T09:58:47Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Probing controller service automatically" func="github.com/dell/csi-unity/service.(*service).requireProbe()" file="/go/src/csi-unity/service/service.go:634"
time="2021-04-27T09:58:47Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Inside Controller Probe" func="github.com/dell/csi-unity/service.(*service).probe()" file="/go/src/csi-unity/service/service.go:671"
time="2021-04-27T09:58:47Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Executing Authenticate REST client" func="github.com/dell/gounity.(*Client).Authenticate()" file="dell/gounity/unityclient.go:50"
handleServer method: /csi.v1.Identity/GetPluginInfo
time="2021-04-27T09:58:47Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0001: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-04-27T09:58:47Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0001: Name=csi-unity.dellemc.com, VendorVersion=1.5.0, Manifest=map[commit:823c1cb2fd696e996b7ad71b6dfc7e666a203ae8 formed:Mon, 22 Mar 2021 10:42:01 UTC semver:1.5.0 url:http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Response code:200 for url: /api/types/loginSessionInfo" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity/api/restclient.go:290"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Authentication response code: 200" func="github.com/dell/gounity.(*Client).Authenticate()" file="dell/gounity/unityclient.go:64"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Authentication successful" func="github.com/dell/gounity.(*Client).Authenticate()" file="dell/gounity/unityclient.go:74"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Controller Probe Success" func="github.com/dell/csi-unity/service.singleArrayProbe()" file="/go/src/csi-unity/service/service.go:662"
time="2021-04-27T09:58:48Z" level=error arrayid=ckm00201700161 runid=node-2 msg="Cannot read directory: /sys/class/fc_host Error: open /sys/class/fc_host: no such file or directory" func="github.com/dell/csi-unity/service/utils.GetFCInitiators()" file="/go/src/csi-unity/service/utils/emcutils.go:90"
time="2021-04-27T09:58:48Z" level=info arrayid=ckm00201700161 runid=node-2 msg="'FC Initiators' cannot be retrieved." func="github.com/dell/csi-unity/service.(*service).addNodeInformationIntoArray()" file="/go/src/csi-unity/service/node.go:1397"
time="2021-04-27T09:58:48Z" level=info arrayid=ckm00201700161 runid=node-2 msg="URI/api/instances/host/name:okd-worker-7itgdxajnoy5zqqjccfxnh-3?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*host).FindHostByName()" file="dell/gounity/host.go:41"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/instances/host/name:okd-worker-7itgdxajnoy5zqqjccfxnh-3?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity/unityclient.go:105"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Response code:200 for url: /api/instances/host/name:okd-worker-7itgdxajnoy5zqqjccfxnh-3?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity/api/restclient.go:290"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Execution successful on Method: GET, URI: /api/instances/host/name:okd-worker-7itgdxajnoy5zqqjccfxnh-3?fields=id,name,description,fcHostInitiators,iscsiHostInitiators,hostIPPorts?fields" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity/unityclient.go:108"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Host okd-worker-7itgdxajnoy5zqqjccfxnh-3 exists on the array" func="github.com/dell/csi-unity/service.(*service).addNodeInformationIntoArray()" file="/go/src/csi-unity/service/node.go:1434"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Adding iSCSI Initiator: Host_12 to host: iqn.1994-05.com.redhat:bfbbd4d9d631 " func="github.com/dell/csi-unity/service.(*service).addNodeInformationIntoArray()" file="/go/src/csi-unity/service/node.go:1471"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Finding Initiator: iqn.1994-05.com.redhat:bfbbd4d9d631" func="github.com/dell/gounity.(*host).CreateHostInitiator()" file="dell/gounity/host.go:183"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Invoking REST API server info Method: GET, URI: /api/types/hostInitiator/instances?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity/unityclient.go:105"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Response code:200 for url: /api/types/hostInitiator/instances?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity/api.(*client).DoAndGetResponseBody()" file="dell/gounity/api/restclient.go:290"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="Execution successful on Method: GET, URI: /api/types/hostInitiator/instances?fields=id,health,type,initiatorId,isIgnored,parentHost,paths" func="github.com/dell/gounity.(*Client).executeWithRetryAuthenticate()" file="dell/gounity/unityclient.go:108"
time="2021-04-27T09:58:48Z" level=debug arrayid=ckm00201700161 runid=node-2 msg="FindHostInitiatorByName: &{
  {HostInitiator_5 {5 [ALRT_COMPONENT_OK] [The component is operating normally. No action is required.]} 2 iqn.1994-05.com.redhat:bfbbd4d9d631 false {Host_7   [] [] [] } [{HostInitiator_5_03:00:00:02_1}]}} Error: 
  
   " func="github.com/dell/gounity.(*host).CreateHostInitiator()" file="dell/gounity/host.go:185" time="2021-04-27T09:58:48Z" level=debug runid=node-2 msg="Adding node [ckm00201700161] failed, Error: [rpc error: code = Internal desc = runid= Adding iSCSI initiator error: Initiator found (iqn.1994-05.com.redhat:bfbbd4d9d631), and attached to someother host (Host_7) instead of host: Host_12]" func="github.com/dell/csi-unity/service.(*service).syncNodeInfo.func2.1()" file="/go/src/csi-unity/service/node.go:1661" handleServer method: /csi.v1.Node/NodeGetInfo time="2021-04-27T09:58:49Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0002: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T09:58:49Z" level=debug runid=2 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778" time="2021-04-27T09:58:52Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0002: rpc error: code = Unavailable desc = runid=2 The node [okd-worker-7itgdxajnoy5zqqjccfxnh-3] is not added to any of the arrays" handleServer method: /csi.v1.Identity/GetPluginInfo time="2021-04-27T09:58:52Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0003: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T09:58:52Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0003: Name=csi-unity.dellemc.com, VendorVersion=1.5.0, Manifest=map[commit:823c1cb2fd696e996b7ad71b6dfc7e666a203ae8 formed:Mon, 22 Mar 2021 10:42:01 UTC semver:1.5.0 url:http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" handleServer method: /csi.v1.Node/NodeGetInfo time="2021-04-27T09:58:54Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0004: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T09:58:54Z" level=debug runid=4 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778" time="2021-04-27T09:58:57Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0004: rpc error: code = Unavailable desc = runid=4 The node [okd-worker-7itgdxajnoy5zqqjccfxnh-3] is not added to any of the arrays" handleServer method: /csi.v1.Identity/GetPluginInfo time="2021-04-27T09:59:12Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0005: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T09:59:12Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0005: Name=csi-unity.dellemc.com, VendorVersion=1.5.0, Manifest=map[commit:823c1cb2fd696e996b7ad71b6dfc7e666a203ae8 formed:Mon, 22 Mar 2021 10:42:01 UTC semver:1.5.0 url:http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" handleServer method: /csi.v1.Node/NodeGetInfo time="2021-04-27T09:59:14Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0006: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T09:59:14Z" level=debug runid=6 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778" time="2021-04-27T09:59:17Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0006: rpc error: code = Unavailable desc = runid=6 The node [okd-worker-7itgdxajnoy5zqqjccfxnh-3] is not added to any of the arrays" handleServer method: /csi.v1.Identity/GetPluginInfo time="2021-04-27T09:59:44Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0007: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T09:59:44Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0007: Name=csi-unity.dellemc.com, VendorVersion=1.5.0, Manifest=map[commit:823c1cb2fd696e996b7ad71b6dfc7e666a203ae8 formed:Mon, 22 Mar 2021 10:42:01 UTC semver:1.5.0 url:http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" handleServer method: /csi.v1.Node/NodeGetInfo time="2021-04-27T09:59:45Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0008: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T09:59:45Z" level=debug runid=8 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778" time="2021-04-27T09:59:48Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0008: rpc error: code = Unavailable desc = runid=8 The node [okd-worker-7itgdxajnoy5zqqjccfxnh-3] is not added to any of the arrays" handleServer method: /csi.v1.Identity/GetPluginInfo time="2021-04-27T10:00:30Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0009: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T10:00:30Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0009: Name=csi-unity.dellemc.com, VendorVersion=1.5.0, Manifest=map[commit:823c1cb2fd696e996b7ad71b6dfc7e666a203ae8 formed:Mon, 22 Mar 2021 10:42:01 UTC semver:1.5.0 url:http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" handleServer method: /csi.v1.Node/NodeGetInfo time="2021-04-27T10:00:31Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0010: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T10:00:31Z" level=debug runid=10 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778" time="2021-04-27T10:00:34Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0010: rpc error: code = Unavailable desc = runid=10 The node [okd-worker-7itgdxajnoy5zqqjccfxnh-3] is not added to any of the arrays" handleServer method: /csi.v1.Identity/GetPluginInfo time="2021-04-27T10:02:03Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REQ 0011: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T10:02:03Z" level=debug msg="/csi.v1.Identity/GetPluginInfo: REP 0011: Name=csi-unity.dellemc.com, VendorVersion=1.5.0, Manifest=map[commit:823c1cb2fd696e996b7ad71b6dfc7e666a203ae8 formed:Mon, 22 Mar 2021 10:42:01 UTC semver:1.5.0 url:http://github.com/dell/csi-unity], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" handleServer method: /csi.v1.Node/NodeGetInfo time="2021-04-27T10:02:04Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REQ 0012: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0" time="2021-04-27T10:02:04Z" level=debug runid=12 msg="Executing NodeGetInfo with args: {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}" func="github.com/dell/csi-unity/service.(*service).NodeGetInfo()" file="/go/src/csi-unity/service/node.go:778" time="2021-04-27T10:02:07Z" level=debug msg="/csi.v1.Node/NodeGetInfo: REP 0012: rpc error: code = Unavailable desc = runid=12 The node [okd-worker-7itgdxajnoy5zqqjccfxnh-3] is not added to any of the arrays"
  

Thanks

Hi @jgazeley , yes, your post is now visible

One thing that I noticed in the logs was this line...

Error: [rpc error: code = Internal desc = runid= Adding iSCSI initiator error: Initiator found (iqn.1994-05.com.redhat:bfbbd4d9d631), and attached to someother host (Host_7) instead of host: Host_12]"

I hit a similar issue once when I cloned a Linux VM and had to reset the iSCSI IQN to a new value before the driver would be able to register it properly with the array.

Good luck
Bryan

Thanks Bryan, I'll try this. Does each node need a unique IQN, or is it one IQN per cluster?