Unsolved
This post is more than 5 years old
6 Posts
0
1758
MDM timeout when attempting map a volume from OpenStack
Hi all;
I'm trying to build a ScaleIO with OpenStack environment for a proof of concept.
Now, I'm facing a symptom that creating and deleting volumes via OpenStack cinder are ok, but attaching them to an instance is always failed by reason of timeout. By my problem detection, It's caused by ScaleIO MDM, but unfortunately, I haven't enough information to solve it especially on ScaleIO.
Could anyone help me?
bellows are pieces of logs;
-- from api_operation.log
2016-08-18 16:52:36,024 [http-nio-443-exec-1] INFO audit - 10.24.23.27:/api/login
2016-08-18 16:52:36,086 [http-nio-443-exec-2] INFO audit - 10.24.23.27:/api/types/Volume/instances/getByName::hrdlcwiITfO7FdWSG7FAwg%253D%253D
2016-08-18 16:52:36,133 [http-nio-443-exec-6] INFO audit - 10.24.23.27:/api/instances/Volume::0d728e2a00000000/action/addMappedSdc {guid=1FBF5158-7ECA-43A2-B9AF-5157ADEC23B1, allowMultipleMappings=TRUE}
---
-- from scaleio.log
2016-08-18 16:52:36,024 [http-nio-443-exec-1] INFO c.e.s.s.s.ConnectionServiceImpl - Send request Login to MDM
2016-08-18 16:52:36,024 [http-nio-443-exec-1] INFO c.e.s.s.s.ConnectionServiceImpl - Got rc SUCCESS
2016-08-18 16:52:36,086 [http-nio-443-exec-2] INFO c.e.s.s.s.ConnectionServiceImpl - Send request QuerySingleObject to MDM
2016-08-18 16:52:36,086 [http-nio-443-exec-2] INFO c.e.s.s.s.ConnectionServiceImpl - Got rc SUCCESS
2016-08-18 16:52:36,149 [http-nio-443-exec-6] INFO c.e.s.s.s.ConnectionServiceImpl - Send request MapVol to MDM
2016-08-18 16:52:37,225 [http-nio-443-exec-6] INFO c.e.s.s.s.ConnectionServiceImpl - Got rc TIMEOUT
2016-08-18 16:52:37,225 [http-nio-443-exec-6] INFO c.e.s.s.s.ConnectionServiceImpl - Send request MapVol to MDM
2016-08-18 16:52:38,239 [http-nio-443-exec-6] INFO c.e.s.s.s.ConnectionServiceImpl - Got rc TIMEOUT
2016-08-18 16:52:38,239 [http-nio-443-exec-6] ERROR c.e.s.s.s.ConnectionServiceImpl - Got error from cluster: TIMEOUT
2016-08-18 16:52:38,239 [http-nio-443-exec-6] ERROR c.e.s.s.w.c.RestApiController - Got an exception in handleException
com.emc.s3g.scaleio.services.ScaleIORemoteRuntimeException: A timeout occurred
--
-- just for addition, form nova-compute.log
2016-08-18 16:52:35.878 17005 INFO os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] ScaleIO get volume id by name request: https://10.24.25.234:443/api/types/Volume/instances/getByName::hrdlcwiITfO7FdWSG7FAwg%253D%253D
2016-08-18 16:52:35.947 17005 INFO os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] Token is invalid, going to re-login to get a new one
2016-08-18 16:52:36.058 17005 INFO os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] ScaleIO volume id is 0d728e2a00000000.
2016-08-18 16:52:36.058 17005 INFO os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] map volume request: https://10.24.25.234:443/api/instances/Volume::0d728e2a00000000/action/addMappedSdc
2016-08-18 16:52:38.221 17005 ERROR os_brick.initiator.connector [req-79a30b03-8e42-4aa6-997e-6b8f6b4b5166 1f6d6616fff24e6c86187b58f45162b9 cd47ff5a9e3d4eedaf9f3be503f1af38 - - -] Error mapping volume hrdlcwiITfO7FdWSG7FAwg==: A timeout occurred
---
osaddict
110 Posts
0
August 22nd, 2016 15:00
Hello,
Can you provide more details about your environment? Specifically, what version of ScaleIO as well as which distribution and version of OpenStack.
Thank you,
Jason
motoyama_m
6 Posts
0
August 22nd, 2016 18:00
Hi Jason,
Thank you for reply.
Scale IO: R1_32.3455.5
OpenStack: Mitaka by using RDO (centos-release-openstack-mitaka-1-3.el7)
Thank you,
Mashio
Himanshu9813
8 Posts
0
September 5th, 2016 06:00
Hi
were you able to solve this problem? what was the solution?
I am facing almost the same issue.
regards
Himanshu
motoyama_m
6 Posts
0
September 5th, 2016 18:00
Hi Himanshu,
No, I wasn't. I have no clue so far.
I'll post here if this was solved or some useful information arrived.
Thank you,
Mashio
Matas1
22 Posts
0
September 5th, 2016 23:00
Hi,
did you configured SDC on your nova compute nodes ? it must be configured and SDC must be added to scaleIO cluster before Openstack cinder drived can connect it.
Try to connect any volume manually to nova compute node to check if it works.
motoyama_m
6 Posts
0
September 6th, 2016 19:00
Hi matas,
Thank you for reply.
Yes, SDC are configured on each nova compute nodes, and the volumes which are created by ScaleIO directly, are able to attach to the nodes as a device like /dev/scinia.
BTW, could anyone list the all of possible causes which end up with MDM timeout?
I want to carry forward the detection, but I cannot get enough information on ScaleIO, internal behavior, detailed meanings of log entry, and so on.
Any information are welcome.
Mashio
motoyama_m
6 Posts
0
September 6th, 2016 21:00
Hi all,
My team put an idea of replacing ScaleIO to a confirmed version which works correctly with OpenStack. But here are some concerns before trying this.
I followed up another thread posted by Himanshu. In his environment, the version of ScaleIO is the latest of 2.x stream if I have no misreading. And he informed here that he is also facing almost same symptom. It's my understanding that both of stream have a possibility of MDM timeout when attaching volume(s) to an instance.
Do you know which version (and stream) of ScaleIO works correctly with OpenStack?
If the version(s) is there, how can I get it? Maybe they are not the latest.
motoyama_m
6 Posts
0
October 2nd, 2016 20:00
I stopped the PoC and start another that is not related to ScaleIO.
I've no other information to be shared here from the last post.
Thank you for your help.