MDM timeout when attempting map a volume from OpenStack

Question

Hi all; I'm trying to build a ScaleIO with OpenStack environment for a proof of concept. Now, I'm facing a symptom that creating and deleting volumes via OpenStack cinder are ok, but attaching them to an instance is always failed by reason of timeout. By my problem detection, It's caused by ScaleIO MDM, but unfortunately, I haven't  enough information to solve it especially on ScaleIO. Could anyone help me? bellows are pieces of logs; -- from api_operation.log 2016-08-18 16:52:36,024 [http-nio-443-exec-1] INFO audit - 10.24.23.27:/api/login 2016-08-18 16:52:36,086 [http-nio-443-exec-2] INFO audit - 10.24.23.27:/api/types/Volume/instances/getByName::hrdlcwiITfO7FdWSG7FAwg%253D%253D 2016-08-18 16:52:36,133 [http-nio-443-exec-6] INFO audit - 10.24.23.27:/api/instances/Volume::0d728e2a00000000/action/addMappedSdc {guid=1FBF5158-7ECA-43A2-B9AF-5157ADEC23B1, allowMultipleMappings=TRUE} --- -- from scaleio.log 2016-08-18 16:52:36,024 [http-nio-443-exec-1] INFO c.e.s.s.s.ConnectionServiceImpl - Send request Login to MDM 2016-08-18 16:52:36,024 [http-nio-443-exec-1] INFO c.e.s.s.s.ConnectionServiceImpl - Got rc SUCCESS 2016-08-18 16:52:36,086 [http-nio-443-exec-2] INFO c.e.s.s.s.ConnectionServiceImpl - Send request QuerySingleObject to MDM 2016-08-18 16:52:36,086 [http-nio-443-exec-2] INFO c.e.s.s.s.ConnectionServiceImpl - Got rc SUCCESS 2016-08-18 16:52:36,149 [http-nio-443-exec-6] INFO c.e.s.s.s.ConnectionServiceImpl - Send request MapVol to MDM 2016-08-18 16:52:37,225 [http-nio-443-exec-6] INFO c.e.s.s.s.ConnectionServiceImpl - Got rc TIMEOUT 2016-08-18 16:52:37,225 [http-nio-443-exec-6] INFO c.e.s.s.s.ConnectionServiceImpl - Send request MapVol to MDM 2016-08-18 16:52:38,239 [http-nio-443-exec-6] INFO c.e.s.s.s.ConnectionServiceImpl - Got rc TIMEOUT 2016-08-18 16:52:38,239 [http-nio-443-exec-6] ERROR c.e.s.s.s.ConnectionServiceImpl - Got error from cluster: TIMEOUT 2016-08-18 16:52:38,239 [http-nio-443-exec-6] ERROR c.e.s.s.w.c.RestApiController - Got an exception in handleException com.emc.s3g.scaleio.services.ScaleIORemoteRuntimeException: A timeout occurred -- -- just for addition, form nova-compute.log 2016-08-18 16:52:35.878 17005 INFO os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] ScaleIO get volume id by name request: https://10.24.25.234:443/api/types/Volume/instances/getByName::hrdlcwiITfO7FdWSG7FAwg%253D%253D 2016-08-18 16:52:35.947 17005 INFO os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] Token is invalid, going to re-login to get a new one 2016-08-18 16:52:36.058 17005 INFO os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] ScaleIO volume id is 0d728e2a00000000. 2016-08-18 16:52:36.058 17005 INFO os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] map volume request: https://10.24.25.234:443/api/instances/Volume::0d728e2a00000000/action/addMappedSdc 2016-08-18 16:52:38.221 17005 ERROR os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] Error mapping volume hrdlcwiITfO7FdWSG7FAwg==: A timeout occurred ---

osaddict · Answer

Hello, Can you provide more details about your environment?  Specifically, what version of ScaleIO as well as which distribution and version of OpenStack. Thank you, Jason

motoyama_m · Answer

Hi Jason, Thank you for reply. Scale IO: R1_32.3455.5 OpenStack: Mitaka by using RDO (centos-release-openstack-mitaka-1-3.el7) Thank you, Mashio

Himanshu9813 · Answer

Hi were you able to solve this problem? what was the solution? I am facing almost the same issue. regards Himanshu

motoyama_m · Answer

Hi Himanshu, No, I wasn't. I have no clue so far. I'll post here if this was solved or some useful information arrived. Thank you, Mashio

Matas1 · Answer

Hi,

did you configured SDC on your nova compute nodes ? it must be configured and SDC must be added to scaleIO cluster before Openstack cinder drived can connect it.

Try to connect any volume manually to nova compute node to check if it works.

motoyama_m · Answer

Hi matas,

Thank you for reply.

Yes, SDC are configured on each nova compute nodes, and the volumes which are created by ScaleIO directly, are able to attach to the nodes as a device like /dev/scinia.

BTW, could anyone list the all of possible causes which end up with MDM timeout?

I want to carry forward the detection, but I cannot get enough information on ScaleIO, internal behavior, detailed meanings of log entry, and so on.

Any information are welcome.

Mashio

motoyama_m · Answer

Hi all,

My team put an idea of replacing ScaleIO to a confirmed version which works correctly with OpenStack. But here are some concerns before trying this.

I followed up another thread posted by Himanshu. In his environment, the version of ScaleIO is the latest of 2.x stream if I have no misreading. And he informed here that he is also facing almost same symptom. It's my understanding that both of stream have a possibility of MDM timeout when attaching volume(s) to an instance.

Do you know which version (and stream) of ScaleIO works correctly with OpenStack?

If the version(s) is there, how can I get it? Maybe they are not the latest.

motoyama_m · Answer

I stopped the PoC and start another that is not related to ScaleIO. I've no other information to be shared here from the last post. Thank you for your help.

PowerFlex

Was this post helpful?