Unsolved
This post is more than 5 years old
522 Posts
1
2649
XtremIO vRO Workflow Datastore Expand Issue
Hi,
Anyone out there using the vRO plugin from EMC for XtremIO and the "Datastore Expand" workflow? This workflow seems like it should expand a datastore on the VMware side AND in doing so, expand the underlying XtremIO LUN as well prior to expanding the datastore. This consistently will expand my XtremIO LUN, but fail on the datastore expand with an error that the full partition is in use. Running this workflow manually there is never an option to select "Extend" because the datastore is initially not "extendable" as one would expect. So it seems this workflow is not ordered correctly or doesn't do what it claims it can do, but wanted to see if anyone else has used it to confirm or deny those thoughts.
Thanks!
-K
rroscoe1
7 Posts
0
July 12th, 2017 11:00
Hi,
What version of the XtremIO vRO plugin are you using? Hopefully the 1.1 version if not please switch to that version and retry. Having said that the 1.1 version of the workflow will first detect whether or not the datastore can be expanded in place without the need for expanding the XtremIO volume. If the datastore can be expanded without expanding the XtremIO volume you will have the opportunity to do so. If the XtremIO volume must first be expanded you must enter in the new size of the XtremIO volume. When the workflow is run the volume will first be expanded to the new supplied size and then datastore will also be expanded. I believe only the 1.1.0 version of the workflow has this capability, the 1.0.0 version does not. if this does not solve your issue then I suggest you open up a case with customer support in order to get proper resolution for this issue.
thanks
R
rroscoe1
7 Posts
0
July 12th, 2017 12:00
Hi Keith,
Glad to read you are using the 1.1.0 workflows. This error is obviously the key to the issue as you have noted:
>>> "The datastore is not expandable -- the entire partition maybe in use (Workflow:Datastore Expand/Datastore Expand (item1)"
Here is something to try. It's possible we need a call to the "host refresh" workflow after the call to the volume expand workflow but prior to expanding the datastore. We can try this out two different ways. The first would be to modify the copy of the "Datastore Expand" workflow and add in a call to the host rescan workflow after the volume is created but before the datastore is expanded.
The other way would be to run the volume expand workflow first which is in the XtremIO Volume Management folder. After you run that then run the Host Rescan workflow for the host or hosts that can see this volume. Then run the Datastore Expand workflow. The Datastore Expand workflow should detect that the volume is larger than the datastore and allow you to expand the datastore in place without first expanding the volume.
Let me know how that goes.
Also what is the initial size of the volume and what is the new size of the volume? How many hosts is the volume being exposed to? Also how many additional volumes are also exposed to the same hosts?
thanks
rich
echolaughmk
522 Posts
0
July 12th, 2017 12:00
Hi Rich,
Thanks for the quick reply. I am using v 1.1.0 of the plugin. Your synopsis is exactly what I had expected as well but it is not working like that and when it detects that my datastore has no space to be expanded, it will blank out the Expand drop down and just allow me to expand the underlying volume and give it a new size. When I increase the underlying size of the LUN, it does go out and increase the XtremIO LUN as expected, but then bombs out on expanding the datastore with an error of "The datastore is not expandable -- the entire partition maybe in use (Workflow:Datastore Expand/Datastore Expand (item1)"
It's almost as if the ordering is incorrect or it is not refreshing the value or noticing that the datastore now has space to be expanded - as if it is stuck on the first query of the datastore when the workflow was initially run (when it really wasn't possible to expand).
I've tried banging around a couple of items and making a copy of the workflow, but just haven't gotten to far. I'll look at the support case route if you think that is the best route and post any answers back here. Not sure if this is a bug or something on my end.
Thanks!
-Keith
echolaughmk
522 Posts
0
July 12th, 2017 13:00
good Idea - I will test that out after a failed expansion run and let you know. I think I figured out an added the Host Rescan workflow so once I get that to save OK, I will run that other test. I suspect you are right since I can re-run the workflow after an initial failure and then I am presented with "Expand" which expands the datastore on the second run just fine - so my gut tells me you are on the right path. thanks!
-Keith
echolaughmk
522 Posts
0
July 12th, 2017 13:00
Hi Rich,
Thanks again for the quick reply. I agree on your thoughts. Let me try to get the first option in place and figure that out in vRO. I like this approach because I'd ideally like to have a single workflow to call that does it all. I am using this to catch an SNMP trigger and automatically expand the LUN and Datastore for a customer so in the larger code that is needed to call this, I think it works cleaner if I am just calling one workflow versus a couple. Let me try to work on this and get back to you asap.
For the volume size, I have tested this a bunch of times trying to fix this error so right now the LUN is 160GB in size. I increment it by 10GB each test. It is presented to an ESX cluster of 6 hosts right now which also see about 8 other LUNs from this same array.
Thanks for the help and I'll get back asap once I can get option #1 in effect to test it.
-Keith
rroscoe1
7 Posts
0
July 12th, 2017 13:00
Sounds good. Also if you manually run the host refresh workflow now, and then run the datastore expand workflow it should give you the opportunity to expand the datastore in place. That would tell us if we are on the right track pretty quickly. I'll wait to hear back from you.
/Rich
echolaughmk
522 Posts
0
July 12th, 2017 15:00
Hi Rich,
So after troubleshooting this for a bit, I am starting to think the Host Rescan workflow is not working because I can't run it as standalone with the following error always coming up:
[2017-07-12 17:58:45.156] Error when rescanning undefined. Reason: TypeError: Cannot read property "configManager" from null
[2017-07-12 17:58:45.209] Workflow execution stack:
***
item: 'Host Rescan/item2', state: 'failed', business state: 'null', exception: 'Error when rescanning undefined. Reason: TypeError: Cannot read property "configManager" from null (Workflow:Host Rescan / Host Rescan (item1)#60)'
workflow: 'Host Rescan' (d34ad379-37d6-498f-a11a-f42fb3f74222)
I can do a Host List workflow with no issues and the vCenter List workflow with no issues, but some of the Host workflows are failing (I haven't touched any of these, so not sure if they came like that?". I get errors like above running the Host Rescan and Host Show workflows. I did not add a vCenter with the XtremIO vCenter Add workflow, rather I did it with the built-in vRO vcenter add and these other workflows see it fine. The vCenter Add workflow actually pops up from the start with 2 validation errors. I don't think this is due to anything I did since I never touched these before running them tonight and I can't edit the imported workflows without making so not sure if you see this in your environments by default?
I opened a case, but they weren't sure of where to route it and couldn't help initially so I am waiting on that as well.
thanks!
-Keith
rroscoe1
7 Posts
0
July 12th, 2017 19:00
It's a little strange that the Host Rescan workflow is failing to run, we simply call into the VMware supplied workflow for this. It looks like the error is coming from the vmware supplied workflow that we call. What version of vRO are you running with?
echolaughmk
522 Posts
0
July 13th, 2017 06:00
Hi Rich, I am running the latest 7.3 vRO appliance. I thought I saw it supported 7.x in one of the guides - is 7.3 a potential issue?
echolaughmk
522 Posts
0
July 13th, 2017 09:00
Hi Rich,
So I tested this on another 7.2 deployment (so I have 2 vRO instances now) and the same thing happens with Host Rescan. The funny thing during the testing is that if I leave the host value blank in the workflow (by default and it is not required) - it fails with the error above.
However, if I manually give it a host in the host value to rescan, it works and runs fine - so it looks to be something maybe with the code of that workflow to discover "all" hosts in the vcenter?
The issue I have now is that I can't really hardcode a host on the rescan and need it to do all the hosts since that is how it would be wanted in the real world and because I also don't chose any host value (not required and defaults to all hosts) in the Datastore Expand workflow. This sounds like we are onto something with the Host Rescan not working for all hosts in a vcenter (the default option if nothing is supplied).
Do you know if that is a known issue or if that works maybe in one of your environments?
I can also attest to the fact that I think you were right with the host rescan inside the Datastore Expand workflow to begin with because if I run that to failure, the XtremIO LUN increases and if I then manually rescan a host in virtual center, it will allow me to extend it in vcenter - so it seems the rescan is critical.
Let me know what you think. Thanks!!
-Keith
rroscoe1
7 Posts
0
July 13th, 2017 13:00
Hi Keith,
sorry for the delay I was not well this morning, better now. So we definitely have some issues with vRO 7.3. the "host show" and "host conformance" workflows stopped working as a result of some architectural changes they made in vro 7.3 around the indexof operator. Just this week we finalized with vmware how we would resolve this issue. As it happens are coming out with a XtremIO vro 2.0.0 plugin that will allow these two workflows to work with vro 7.3. as well as previous 7.x vro releases. So for now please stick with the vRO 7.2 release until the 2.0.0 XtremIO plugin is available.
As far as host rescan goes yes we had a bug in that workflow and it's exactly as you describe. If you select an individual host it works, if you don't select a host it fails as you saw. let me work on getting you and updated Host Rescan workflow. Please mention via the case that you opened for them to contact me regarding the Host Rescan Issue. We should be able to get that to you very shortly. I'll also look into having Datastore Expand modified to do a host rescan.
Out of curiosity do you have one cluster in your vcenter server or multiple clusters?
thanks
Rich
rroscoe1
7 Posts
0
July 13th, 2017 14:00
Hi Keith,
we have a new 1.1 vRO package for you the version is 1.1.1-3. It fixes the Host Rescan issue and has a few other fixes as well. As soon as I get contacted by customer support we should be able to pass along the kit to you and you should be all set. We will try and get the "datastore expand" workflow updated to include a call to the host rescan workflow for the 2.0.0 release.
thanks
Rich
echolaughmk
522 Posts
0
July 13th, 2017 16:00
Hi Rich!
You rock...thanks. I will let the support case owner know.
That is great info and I will stick with the 7.2 deployment I did today instead of the 7.3 deployment. I'm not sure what the shift schedule the support owner is on, but I will reach out to them now...stay tuned.
Thanks again!
-Keith
echolaughmk
522 Posts
0
July 17th, 2017 06:00
Hi Rich,
The new packages were spot on! Thank you very much for the help. I am sticking with vRO 7.3 and the new 1.1.3 package - the host rescan works for all hosts in the vcenter. I am gonna test a little more this week and get to the scripting process of inserting the host rescan into the datastore expand function, but a manual run ran fine and I wanted to let you know and thank you for the help. I am traveling this week so when I get back at the testing, I will post updates here. Thanks again!!
-Keith
rroscoe1
7 Posts
0
July 17th, 2017 07:00
Hi Keith,
good news. With regards to using vRO 7.3.0, In addition to the Host Show and Host Conformance workflows not working (along with any workflows that call these workflows) the vCenter Add workflow also does not work. We are going to deprecate that workflow in the 2.0.0 release of our vRO package. To add a VMware vCenter Server instance to vRO one way it can be done is to use the VMware supplied workflow "Add a vCenter Server Instance". This workflow can be found in the vCenter\Configuration folder.
/Rich