UNSOLVED
PowerMax_2000 rest API ReadTimeout issue.
Hi Team,
We are using PowerMax_2000 with Unisphere version V9.2.3.15 and openstack volume driver VMaxRestProductDriver (4.4.0).
While creating multiple volumes requests with above configuration, we are seeing ReadTimeout issue intermittently during invocation of Powermax Rest API's as shown below. Do you have any know issue of read timeout with PowerMax.
|__Flow 'volume_create_manager': requests.exceptions.ReadTimeout: The GET to Unisphere server httpspvc://9.3.118.149:8443/univmax/restapi has experienced a ReadTimeout error. Please check your Unisphere server connection/availability. Exception message: None: None
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager Traceback (most recent call last):
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager result = task.execute(**arguments)
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/flows/manager/create_volume.py", line 1201, in execute
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager model_update = self._create_raw_volume(
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/flows/manager/create_volume.py", line 1165, in _create_raw_volume
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager self.message.create(
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager self.force_reraise()
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager raise self.value
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/flows/manager/create_volume.py", line 1162, in _create_raw_volume
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager ret = self.driver.create_volume(volume)
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/powervc_cinder/volume/drivers/emc/vmax_product_rest.py", line 419, in create_volume
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager model_update = self.common.create_volume(volume)
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/common.py", line 514, in create_volume
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager volume_dict, rep_update, rep_info_dict = self._create_volume(
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/common.py", line 2132, in _create_volume
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager array_model, next_gen = self.rest.get_array_model_info(array)
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/rest.py", line 855, in get_array_model_info
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager system_info = self.get_array_detail(array)
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/rest.py", line 724, in get_array_detail
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager array_details = self.get_request(target_uri, 'system')
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/rest.py", line 586, in get_request
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager sc, message = self.request(target_uri, GET, params=params)
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/powervc_cinder/volume/drivers/emc/vmax_product_rest.py", line 2852, in request
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager sc, msg = super(VMAXPVCRest, self).request(
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/rest.py", line 308, in request
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager raise exc_class(msg % {'method': method,
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager requests.exceptions.ReadTimeout: The GET to Unisphere server httpspvc://9.3.118.149:8443/univmax/restapi has experienced a ReadTimeout error. Please check your Unisphere server connection/availability. Exception message: None: None
2022-11-15 03:06:17.593 3386169 ERROR cinder.volume.manager
DELL-Sam L
Community Manager
•
8103 Posts
•
34571 Points
411
0
Posted November 16th, 2022 02:00
Hello prashkre,
It is best to open a support case for this issue.
John Toner
1 Rookie
•
62 Posts
•
3 Points
395
0
Posted November 17th, 2022 08:00
Do you have a Service Request open for this issue? If so, please provide the number so we can review the details.
I would recommend enabling INFO and RESTAPI logging on your Unisphere server and having support review these logs for any additional insight.
prashkre
3 Posts
419
0
Posted November 15th, 2022 09:00
One more trace of ReadTimeout issue we had seen in cinder powermax driver logs.
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager Traceback (most recent call last):
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 1956, in initialize_connection
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager conn_info = self.driver.initialize_connection(volume, connector)
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/powervc_cinder/volume/drivers/emc/vmax_product_rest.py", line 336, in initialize_connection
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager ret_data = self.emc_super.initialize_connection(volume, connector)
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/fc.py", line 286, in initialize_connection
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager device_info = self.common.initialize_connection(
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/common.py", line 952, in initialize_connection
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager self.find_host_lun_id(volume, connector.get('host'), extra_specs,
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/powervc_cinder/volume/drivers/emc/vmax_product_rest.py", line 1982, in find_host_lun_id
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager device_id = self._find_device_on_array(volume, extra_specs)
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/common.py", line 1627, in _find_device_on_array
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager founddevice_id = self.rest.check_volume_device_id(
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/powervc_cinder/volume/drivers/emc/vmax_product_rest.py", line 3010, in check_volume_device_id
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager vol_details = self.get_volume(array, device_id)
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/rest.py", line 1435, in get_volume
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager volume_dict = self.get_resource(
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/rest.py", line 637, in get_resource
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager return self.get_request(target_uri, resource_type, params)
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/rest.py", line 586, in get_request
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager sc, message = self.request(target_uri, GET, params=params)
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/powervc_cinder/volume/drivers/emc/vmax_product_rest.py", line 2852, in request
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager sc, msg = super(VMAXPVCRest, self).request(
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/powermax/rest.py", line 308, in request
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager raise exc_class(msg % {'method': method,
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager requests.exceptions.ReadTimeout: The GET to Unisphere server httpspvc://9.3.118.149:8443/univmax/restapi has experienced a ReadTimeout error. Please check your Unisphere server connection/availability. Exception message: None: None
2022-11-15 04:07:02.985 3386169 ERROR cinder.volume.manager