Hi @Ulfj,

The latest version of the driver (v1.5) introduced the support for multi-array.

That is to say a single instance of the driver can connect to multiple PowerScale/Isilon.

To configure the connectivity (URL & credentials) you have to configure a secret like here : https://github.com/dell/csi-powerscale/blob/master/helm/secret.json

And then create it in Kubernetes with :

kubectl create secret generic isilon-creds -n isilon --from-file=config=secret.json -o yaml --dry-run=client | kubectl replace -f -

I agree that the instructions are unclear and we fix it ASAP.

Let me know if that works.

The secret was the problem....figured it out

Now the controller pods start and get ready, but the node pods go in CrashLoop complaining on unable to do reverse dns lookup.

Must the reverse map exist for the node subnet?

Regards,

-Ulf

Got some logs from the nodes.

level=error runid=392 msg="error getting FQDN: 'lookup 54.68.4.10.in-addr.arpa. on 10.4.68.54:53: no such host'" file="/go/src/common/utils/utils.go:266"

-Ulf

Can we make sure that we can access the DNS  on 10.4.68.54 from a Pod ?

A good way to debug it, is to use a Pod : https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#create-a-simple-pod-to-use-as-a-test-environment

Then you can do a : nslookup 10.4.68.54

Hi Ulfj ,

If you still see failure in driver node pod. Let me know which platform you are trying to install driver is it k8s or OpenShift ?

Also, share one of describe node pod logs.

Hi.

I tested to install the dns test container in the namespace I use for the CSI driver and it seems to work.

[a2001008@vm000496 dell_csi]$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server: 172.30.0.10
Address: 172.30.0.10#53

Name: kubernetes.default.svc.cluster.local
Address: 172.30.0.1

If I try to resolve the 10.4.68.54 address.

[a2001008@vm000496 dell_csi]$ kubectl exec -i -t dnsutils -- nslookup 10.4.68.54
54.68.4.10.in-addr.arpa name = 10-4-68-54.machine-config-daemon.openshift-machine-config-operator.svc.cluster.local.

But the Isilon node pod is trying to resolve against 10.4.68.54, which is the pod's IP, which is the same as the nodes, vm, the pod is running on, is that correct?

If I try to resolve against 10.4.68.54 I get the following.

[a2001008@vm000496 dell_csi]$ kubectl exec -i -t dnsutils -- nslookup 10.4.68.54 10.4.68.54
** server can't find 54.68.4.10.in-addr.arpa: NXDOMAIN

Which is the same result as the pod gets.

I also tried to resolve a known address

[a2001008@vm000496 dell_csi]$ kubectl exec -i -t dnsutils -- nslookup 8.8.8.8 10.4.68.54
8.8.8.8.in-addr.arpa name = dns.google.

Which works as expected.

So I'm still not able to run the driver

Regards,

-Ulf

Hi.

It still won't work.

See my previous post.

The cluster is  OpenShift 4.7.3.

And here comes a oc describe of a isilon node pod.

Name:         isilon-node-n5t8t
Namespace:    openshift-operators
Priority:     0
Node:         k8s-test-xptlx-worker-4qxkl/10.4.68.54
Start Time:   Tue, 30 Mar 2021 15:44:32 +0200
Labels:       app=isilon-node
              controller-revision-hash=68c84cbd44
              pod-template-generation=1
Annotations:  openshift.io/scc: privileged
Status:       Running
IP:           10.4.68.54
IPs:
  IP:           10.4.68.54
Controlled By:  DaemonSet/isilon-node
Containers:
  driver:
    Container ID:   cri-o://87b8fb82c88c9d9689f1cae05253f360af180533a4461d4767a75782a523aea2
    Image:          dellemc/csi-isilon:v1.5.0
    Image ID:       docker.io/dellemc/csi-isilon@sha256:415da1b070d5d273a0243be84545b13578014ac17e807cf89968a6cabd98b359
    Port:           
    Host Port:      
    State:          Running
      Started:      Tue, 30 Mar 2021 15:44:33 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      CSI_ENDPOINT:                    /var/lib/kubelet/plugins/csi-isilon/csi_sock
      X_CSI_MODE:                      node
      X_CSI_ISI_AUTOPROBE:             true
      X_CSI_DEBUG:                     true
      X_CSI_ISI_INSECURE:              true
      X_CSI_NODE_NAME:                  (v1:spec.nodeName)
      X_CSI_NODE_IP:                    (v1:status.hostIP)
      SSL_CERT_DIR:                    /certs
      X_CSI_PRIVATE_MOUNT_DIR:         /var/lib/kubelet/plugins/csi-isilon/disks
      X_CSI_ISI_PORT:                  8080
      X_CSI_CUSTOM_TOPOLOGY_ENABLED:   false
      X_CSI_ALLOWED_NETWORKS:
      X_CSI_VERBOSE:                   1
      X_CSI_ISI_ENDPOINT:              10.4.30.15
      X_CSI_ISI_PATH:                  /ifs/se1isl001/proj
      X_CSI_ISILON_NO_PROBE_ON_START:  false
      X_CSI_ISILON_NFS_V3:             false
    Mounts:
      /certs from certs (ro)
      /dev from dev (rw)
      /isilon-configs from isilon-configs (ro)
      /var/lib/kubelet/plugins/csi-isilon from driver-path (rw)
      /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices from volumedevices-path (rw)
      /var/lib/kubelet/pods from pods-path (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from isilon-node-token-9wjxl (ro)
  registrar:
    Container ID:  cri-o://8569cdcdfc4228451562210d0fc6ffecef703d86636f1c4b3676be1145f2265c
    Image:         k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0
    Image ID:      k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f
    Port:          
    Host Port:     
    Args:
      --v=5
      --csi-address=$(ADDRESS)      --kubelet-registration-path=/var/lib/kubelet/plugins/csi-isilon/csi_sock
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 31 Mar 2021 10:23:57 +0200
      Finished:     Wed, 31 Mar 2021 10:23:58 +0200
    Ready:          False
    Restart Count:  223
    Environment:
      ADDRESS:         /csi/csi_sock
      KUBE_NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /csi from driver-path (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from isilon-node-token-9wjxl (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
  driver-path:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/csi-isilon
    HostPathType:  DirectoryOrCreate
  volumedevices-path:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices
    HostPathType:  DirectoryOrCreate
  pods-path:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:  Directory
  dev:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
  certs:
    Type:  Projected (a volume that contains injected data from multiple sources)
  isilon-configs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  isilon-creds
    Optional:    true
  isilon-node-token-9wjxl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  isilon-node-token-9wjxl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason   Age                   From                                  Message
  ----     ------   ----                  ----                                  -------
  Normal   Pulled   166m (x192 over 18h)  kubelet, k8s-test-xptlx-worker-4qxkl  Container image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0" already present on machine
  Warning  BackOff  84s (x5133 over 18h)  kubelet, k8s-test-xptlx-worker-4qxkl  Back-off restarting failed container
                    

Regards,

-Ulf 

Hi 

Can you paste me node pod registrar container logs :-

oc logs isilon-node-n5t8t -n isilon -c registrar

Hi.

Here is the log.

I0331 09:25:42.055547       1 main.go:113] Version: v2.1.0
I0331 09:25:42.055977       1 main.go:137] Attempting to open a gRPC connection with: "/csi/csi_sock"
I0331 09:25:42.055994       1 connection.go:153] Connecting to unix:///csi/csi_sock
I0331 09:25:42.056645       1 main.go:144] Calling CSI driver to discover driver name
I0331 09:25:42.056659       1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0331 09:25:42.056662       1 connection.go:183] GRPC request: {}
I0331 09:25:42.059571       1 connection.go:185] GRPC response: {"manifest":{"commit":"964a54c79653516cbbb79fcd9a24ec8c1b3abfc7","formed":"Mon, 22 Mar 2021 06:38:35 UTC","semver":"1.5.0","url":"http://github.com/dell/csi-isilon"},"name":"csi-isilon.dellemc.com","vendor_version":"1.5.0"}
I0331 09:25:42.059652       1 connection.go:186] GRPC error: 
I0331 09:25:42.059663       1 main.go:154] CSI driver name: "csi-isilon.dellemc.com"
I0331 09:25:42.059753       1 node_register.go:52] Starting Registration Server at: /registration/csi-isilon.dellemc.com-reg.sock
I0331 09:25:42.059988       1 node_register.go:61] Registration Server started at: /registration/csi-isilon.dellemc.com-reg.sock
I0331 09:25:42.060054       1 node_register.go:83] Skipping healthz server because HTTP endpoint is set to: ""
I0331 09:25:43.747770       1 main.go:80] Received GetInfo call: &nfoRequest{}
I0331 09:25:43.759189       1 main.go:90] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unknown desc = lookup 54.68.4.10.in-addr.arpa. on 10.4.68.54:53: no such host,}
E0331 09:25:43.759228       1 main.go:92] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unknown desc = lookup 54.68.4.10.in-addr.arpa. on 10.4.68.54:53: no such host, restarting registration container.

Regards,

-Ulf

No more info about this?

Regards,

-Ulf