Ulfj
2 Bronze

Installing csi driver

Hi.

I'm trying to set up the CSI driver and connect to Isilon.

Getting this in the logs on the pods which are in CrashLoop.......

time="2021-03-29T09:01:17Z" level=info msg="Configured 'csi-isilon.dellemc.com'" accesspoint=System autoprobe=true insecure=true mode=controller path=/ifs/se1isl001/proj quotaenabled=true
time="2021-03-29T09:01:17Z" level=info msg="************* Synchronizing Isilon Clusters' config **************"
time="2021-03-29T09:01:17Z" level=info msg="removed sock file" path=/var/run/csi/csi.sock
time="2021-03-29T09:01:17Z" level=fatal msg="grpc failed" error="file ('/isilon-configs/config') error: open /isilon-configs/config: no such file or directory"

Any ideas?

Regards,

-Ulf

0 Kudos
12 Replies
Flo_csI
3 Argentum

Re: Installing csi driver

Hi @Ulfj,

The latest version of the driver (v1.5) introduced the support for multi-array.

That is to say a single instance of the driver can connect to multiple PowerScale/Isilon.

 

To configure the connectivity (URL & credentials) you have to configure a secret like here : https://github.com/dell/csi-powerscale/blob/master/helm/secret.json

And then create it in Kubernetes with :

kubectl create secret generic isilon-creds -n isilon --from-file=config=secret.json -o yaml --dry-run=client | kubectl replace -f -

 

I agree that the instructions are unclear and we fix it ASAP.

 

Let me know if that works.

0 Kudos
Anonymous
Not applicable

Re: Installing csi driver

Hi @Ulfj,

Does the csi-powerscale installation went fine. 

Let us know to help you further. 

 

 

 

0 Kudos
Ulfj
2 Bronze

Re: Installing csi driver

The secret was the problem....figured it out 🙂

Now the controller pods start and get ready, but the node pods go in CrashLoop complaining on unable to do reverse dns lookup.

Must the reverse map exist for the node subnet?

 

Regards,

-Ulf

 

0 Kudos
Ulfj
2 Bronze

Re: Installing csi driver

Got some logs from the nodes.

level=error runid=392 msg="error getting FQDN: 'lookup 54.68.4.10.in-addr.arpa. on 10.4.68.54:53: no such host'" file="/go/src/common/utils/utils.go:266"

 

-Ulf

 

0 Kudos
Flo_csI
3 Argentum

Re: Installing csi driver

Can we make sure that we can access the DNS  on 10.4.68.54 from a Pod ?

 

A good way to debug it, is to use a Pod : https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#create-a-simple-pod-to...

 

Then you can do a : nslookup 10.4.68.54

0 Kudos
Nit_csi
2 Bronze

Re: Installing csi driver

Hi Ulfj ,

If you still see failure in driver node pod. Let me know which platform you are trying to install driver is it k8s or OpenShift ?

Also, share one of describe node pod logs.

0 Kudos
Ulfj
2 Bronze

Re: Installing csi driver

Hi.

It still won't work.

See my previous post.

The cluster is  OpenShift 4.7.3.

And here comes a oc describe of a isilon node pod.

Name:         isilon-node-n5t8t
Namespace:    openshift-operators
Priority:     0
Node:         k8s-test-xptlx-worker-4qxkl/10.4.68.54
Start Time:   Tue, 30 Mar 2021 15:44:32 +0200
Labels:       app=isilon-node
              controller-revision-hash=68c84cbd44
              pod-template-generation=1
Annotations:  openshift.io/scc: privileged
Status:       Running
IP:           10.4.68.54
IPs:
  IP:           10.4.68.54
Controlled By:  DaemonSet/isilon-node
Containers:
  driver:
    Container ID:   cri-o://87b8fb82c88c9d9689f1cae05253f360af180533a4461d4767a75782a523aea2
    Image:          dellemc/csi-isilon:v1.5.0
    Image ID:       docker.io/dellemc/csi-isilon@sha256:415da1b070d5d273a0243be84545b13578014ac17e807cf89968a6cabd98b359
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 30 Mar 2021 15:44:33 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      CSI_ENDPOINT:                    /var/lib/kubelet/plugins/csi-isilon/csi_sock
      X_CSI_MODE:                      node
      X_CSI_ISI_AUTOPROBE:             true
      X_CSI_DEBUG:                     true
      X_CSI_ISI_INSECURE:              true
      X_CSI_NODE_NAME:                  (v1:spec.nodeName)
      X_CSI_NODE_IP:                    (v1:status.hostIP)
      SSL_CERT_DIR:                    /certs
      X_CSI_PRIVATE_MOUNT_DIR:         /var/lib/kubelet/plugins/csi-isilon/disks
      X_CSI_ISI_PORT:                  8080
      X_CSI_CUSTOM_TOPOLOGY_ENABLED:   false
      X_CSI_ALLOWED_NETWORKS:
      X_CSI_VERBOSE:                   1
      X_CSI_ISI_ENDPOINT:              10.4.30.15
      X_CSI_ISI_PATH:                  /ifs/se1isl001/proj
      X_CSI_ISILON_NO_PROBE_ON_START:  false
      X_CSI_ISILON_NFS_V3:             false
    Mounts:
      /certs from certs (ro)
      /dev from dev (rw)
      /isilon-configs from isilon-configs (ro)
      /var/lib/kubelet/plugins/csi-isilon from driver-path (rw)
      /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices from volumedevices-path (rw)
      /var/lib/kubelet/pods from pods-path (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from isilon-node-token-9wjxl (ro)
  registrar:
    Container ID:  cri-o://8569cdcdfc4228451562210d0fc6ffecef703d86636f1c4b3676be1145f2265c
    Image:         k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0
    Image ID:      k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)      --kubelet-registration-path=/var/lib/kubelet/plugins/csi-isilon/csi_sock
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 31 Mar 2021 10:23:57 +0200
      Finished:     Wed, 31 Mar 2021 10:23:58 +0200
    Ready:          False
    Restart Count:  223
    Environment:
      ADDRESS:         /csi/csi_sock
      KUBE_NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /csi from driver-path (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from isilon-node-token-9wjxl (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
  driver-path:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/csi-isilon
    HostPathType:  DirectoryOrCreate
  volumedevices-path:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices
    HostPathType:  DirectoryOrCreate
  pods-path:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:  Directory
  dev:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
  certs:
    Type:  Projected (a volume that contains injected data from multiple sources)
  isilon-configs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  isilon-creds
    Optional:    true
  isilon-node-token-9wjxl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  isilon-node-token-9wjxl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason   Age                   From                                  Message
  ----     ------   ----                  ----                                  -------
  Normal   Pulled   166m (x192 over 18h)  kubelet, k8s-test-xptlx-worker-4qxkl  Container image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0" already present on machine
  Warning  BackOff  84s (x5133 over 18h)  kubelet, k8s-test-xptlx-worker-4qxkl  Back-off restarting failed container
                    

 

Regards,

-Ulf 

0 Kudos
Ulfj
2 Bronze

Re: Installing csi driver

Hi.

I tested to install the dns test container in the namespace I use for the CSI driver and it seems to work.

[a2001008@vm000496 dell_csi]$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server: 172.30.0.10
Address: 172.30.0.10#53

Name: kubernetes.default.svc.cluster.local
Address: 172.30.0.1

If I try to resolve the 10.4.68.54 address.

[a2001008@vm000496 dell_csi]$ kubectl exec -i -t dnsutils -- nslookup 10.4.68.54
54.68.4.10.in-addr.arpa name = 10-4-68-54.machine-config-daemon.openshift-machine-config-operator.svc.cluster.local.

But the Isilon node pod is trying to resolve against 10.4.68.54, which is the pod's IP, which is the same as the nodes, vm, the pod is running on, is that correct?

If I try to resolve against 10.4.68.54 I get the following.

[a2001008@vm000496 dell_csi]$ kubectl exec -i -t dnsutils -- nslookup 10.4.68.54 10.4.68.54
** server can't find 54.68.4.10.in-addr.arpa: NXDOMAIN

Which is the same result as the pod gets.

I also tried to resolve a known address 🙂

[a2001008@vm000496 dell_csi]$ kubectl exec -i -t dnsutils -- nslookup 8.8.8.8 10.4.68.54
8.8.8.8.in-addr.arpa name = dns.google.

Which works as expected.

So I'm still not able to run the driver 😞

Regards,

-Ulf

 

0 Kudos
Nit_csi
2 Bronze

Re: Installing csi driver

Hi 

Can you paste me node pod registrar container logs :-

oc logs isilon-node-n5t8t -n isilon -c registrar

0 Kudos