CSM: CSI Driver for PowerMax fails volume creation with errors finding storage group and volume
Summary: Dell Container Storage Modules (CSM) Container Storage Interface (CSI) Driver for PowerMax fails volume creation with errors finding storage group (SG) and volume.
Symptoms
The following errors appear in the controller driver logs when attempting to create a volume. The Representational State Transfer (REST) command retrieves the details on the SG. The REST response is "200 ok" but then the logs show "Unable to find storage group".
time="2024-02-14T17:15:03Z" level=debug msg="\n -------------------------- POWERMAX HTTP REQUEST -------------------------\n GET /univmax/restapi/100/sloprovisioning/symmetrix/<powermax id>/storagegroup/<storage group> HTTP/1.1\n Host: 0.0.0.0:2222\n Accept: application/json;version=100\n Application-Type: CSI Driver for Dell EMC PowerMax/v2.9.1\n Content-Type: application/json;version=100\n Symid: <powermax id>\n \n"
time="2024-02-14T17:15:03Z" level=debug msg="\n -------------------------- POWERMAX HTTP RESPONSE -------------------------\n HTTP/1.1 200 OK\n Transfer-Encoding: chunked\n Content-Security-Policy: default-src 'self'\n Content-Type: application/json\n Date: Wed, 14 Feb 2024 17:15:03 GMT\n Referrer-Policy: no-referrer-when-downgrade\n Restapi-Active-Session-Count: 4\n Restapi-Processing-Time: 25\n Strict-Transport-Security: max-age=31536000; includeSubDomains\n X-Content-Security-Policy: default-src 'self'\n X-Content-Type-Options: nosniff\n X-Frame-Options: SAMEORIGIN\n X-Webkit-Csp: default-src 'self'\n X-Xss-Protection: 1; mode=block\n \n 176\n {\"storageGroupId\":\"<storage group>\",\"slo\":\"Gold\",\"service_level\":\"Gold\",\"base_slo_name\":\"Gold\",\"srp\":\"SRP_1\",\"slo_compliance\":\"STABLE\",\"num_of_vols\":0,\"num_of_child_sgs\":0,\"num_of_parent_sgs\":0,\"num_of_masking_views\":0,\"num_of_snapshots\":0,\"cap_gb\":0.0,\"type\":\"Standalone\",\"unprotected\":true,\"compression\":true,\"compression_ratio_to_one\":0.0,\"unreducible_data_gb\":0.0}\n 0\n \n"
time="2024-02-14T17:15:03Z" level=debug msg="Unable to find storage group: <storage group>"
Then continuing, the REST checks if the volume exists before trying to create it. The REST returns that the volume is not found with a 404 Not Found message, but the code expects an empty list instead.
time="2024-02-14T17:15:03Z" level=debug msg="\n -------------------------- POWERMAX HTTP REQUEST -------------------------\n GET /univmax/restapi/100/sloprovisioning/symmetrix/<powermax id>/volume?volume_identifier=<volume name> HTTP/1.1\n Host: 0.0.0.0:2222\n Accept: application/json;version=100\n Application-Type: CSI Driver for Dell EMC PowerMax/v2.9.1\n Content-Type: application/json;version=100\n Symid: <powermax id>\n \n"
time="2024-02-14T17:15:04Z" level=debug msg="\n -------------------------- POWERMAX HTTP RESPONSE -------------------------\n HTTP/1.1 404 Not Found\n Content-Length: 26\n Content-Type: text/plain; charset=utf-8\n Date: Wed, 14 Feb 2024 17:15:04 GMT\n \n Invalid Response from API\n"
time="2024-02-14T17:15:04Z" level=error msg="Error looking up volume for idempotence check: Not Found"
time="2024-02-14T17:15:04Z" level=info msg="/csi.v1.Controller/CreateVolume: REP 0032: rpc error: code = Internal desc = Error looking up volume for idempotence check: Not Found"
time="2024-02-14T17:15:04Z" level=info msg="/csi.v1.Controller/GetCapacity: REQ 0033: VolumeCapabilities=[mount:<> access_mode:<> ], Parameters=map[SRP:SRP_1 SYMID:<powermax id> ServiceLevel:Gold csi.storage.k8s.io/fstype:ext4], AccessibleTopology=segments:<key:\"csi-powermax.dellemc.com/<powermax id>\" value:\"csi-powermax.dellemc.com\" > segments:<key:\"csi-powermax.dellemc.com/<powermax id>.vsphere\" value:\"csi-powermax.dellemc.com\" > , XXX_NoUnkeyedLiteral={}, XXX_sizecache=0"
This behavior then repeats, and the volume is never created.
Cause
The user added a trailing "/" in the values file or powermax-reverseproxy-config and powermax-array-config configmaps when entering the Unisphere for PowerMax endpoint.
https://primary-1.unisphe.re:8443/
The provided values YAML file shows no trailing "/" in the sample.
https://primary-1.unisphe.re:8443
Resolution
Installation should be done using a values file that does not contain the additional "/".
Refer to the documentation and follow the deployment instructions applicable to the specific environment: https://dell.github.io/csm-docs/
Additional Information
Enable powerMaxDebug to see the POWERMAX HTTP REQUEST or RESPONSE in the logs.
Refer to the documentation and follow the deployment instructions applicable to the specific environment: https://dell.github.io/csm-docs/
logLevel - CSI driver log level. Allowed values: "error", "warn"/"warning", "info", "debug"
powerMaxDebug - Enables low level and http traffic logging between the CSI driver and Unisphere. Do not enable this unless the support team asks to do so.