XtremIO vRO Workflow Datastore Expand Issue

Hi Keith,

Glad to read you are using the 1.1.0 workflows. This error is obviously the key to the issue as you have noted:

>>> "The datastore is not expandable -- the entire partition maybe in use (Workflow:Datastore Expand/Datastore Expand (item1)"

Here is something to try. It's possible we need a call to the "host refresh" workflow after the call to the volume expand workflow but prior to expanding the datastore.  We can try this out two different ways.  The first would be to modify the copy of the "Datastore Expand" workflow and add in a call to the host rescan workflow after the volume is created but before the datastore is expanded.

The other way would be to run the volume expand workflow first which is in the XtremIO Volume Management folder.  After you run that then run the Host Rescan workflow for the host or hosts that can see this volume.  Then run the Datastore Expand workflow.  The Datastore Expand workflow should detect that the volume is larger than the datastore and allow you to expand the datastore in place without first expanding the volume.

Let me know how that goes.

Also what is the initial size of the volume and what is the new size of the volume?  How many hosts is the volume being exposed to? Also how many additional volumes are also exposed to the same hosts?

thanks

rich

Hi Rich,

Thanks for the quick reply. I am using v 1.1.0 of the plugin. Your synopsis is exactly what I had expected as well but it is not working like that and when it detects that my datastore has no space to be expanded, it will blank out the Expand drop down and just allow me to expand the underlying volume and give it a new size. When I increase the underlying size of the LUN, it does go out and increase the XtremIO LUN as expected, but then bombs out on expanding the datastore with an error of "The datastore is not expandable -- the entire partition maybe in use (Workflow:Datastore Expand/Datastore Expand (item1)"

It's almost as if the ordering is incorrect or it is not refreshing the value or noticing that the datastore now has space to be expanded - as if it is stuck on the first query of the datastore when the workflow was initially run (when it really wasn't possible to expand).

I've tried banging around a couple of items and making a copy of the workflow, but just haven't gotten to far. I'll look at the support case route if you think that is the best route and post any answers back here. Not sure if this is a bug or something on my end.

Thanks!

-Keith

good Idea - I will test that out after a failed expansion run and let you know. I think I figured out an added the Host Rescan workflow so once I get that to save OK, I will run that other test. I suspect you are right since I can re-run the workflow after an initial failure and then I am presented with "Expand" which expands the datastore on the second run just fine - so my gut tells me you are on the right path. thanks!

-Keith

Hi Rich,

Thanks again for the quick reply. I agree on your thoughts. Let me try to get the first option in place and figure that out in vRO. I like this approach because I'd ideally like to have a single workflow to call that does it all. I am using this to catch an SNMP trigger and automatically expand the LUN and Datastore for a customer so in the larger code that is needed to call this, I think it works cleaner if I am just calling one workflow versus a couple. Let me try to work on this and get back to you asap.

For the volume size, I have tested this a bunch of times trying to fix this error so right now the LUN is 160GB in size. I increment it by 10GB each test. It is presented to an ESX cluster of 6 hosts right now which also see about 8 other LUNs from this same array.

Thanks for the help and I'll get back asap once I can get option #1 in effect to test it.

-Keith

Sounds good.  Also if you manually run the host refresh workflow now, and then run the datastore expand workflow it should give you the opportunity to expand the datastore in place.  That would tell us if we are on the right track pretty quickly. I'll wait to hear back from you.

/Rich

Hi Rich,

So after troubleshooting this for a bit, I am starting to think the Host Rescan workflow is not working because I can't run it as standalone with the following error always coming up:

[2017-07-12 17:58:45.156] Error when rescanning undefined. Reason: TypeError: Cannot read property "configManager" from null

[2017-07-12 17:58:45.209] Workflow execution stack:

***

item: 'Host Rescan/item2', state: 'failed', business state: 'null', exception: 'Error when rescanning undefined. Reason: TypeError: Cannot read property "configManager" from null (Workflow:Host Rescan / Host Rescan (item1)#60)'

workflow: 'Host Rescan' (d34ad379-37d6-498f-a11a-f42fb3f74222)

I can do a Host List workflow with no issues and the vCenter List workflow with no issues, but some of the Host workflows are failing (I haven't touched any of these, so not sure if they came like that?". I get errors like above running the Host Rescan and Host Show workflows. I did not add a vCenter with the XtremIO vCenter Add workflow, rather I did it with the built-in vRO vcenter add and these other workflows see it fine. The vCenter Add workflow actually pops up from the start with 2 validation errors. I don't think this is due to anything I did since I never touched these before running them tonight and I can't edit the imported workflows without making so not sure if you see this in your environments by default?

I opened a case, but they weren't sure of where to route it and couldn't help initially so I am waiting on that as well.

thanks!

-Keith

It's a little strange that the Host Rescan workflow is failing to run, we simply call into the VMware supplied workflow for this. It looks like the error is coming from the vmware supplied workflow that we call. What version of vRO are you running with?

Hi Rich, I am running the latest 7.3 vRO appliance. I thought I saw it supported 7.x in one of the guides - is 7.3 a potential issue?

Hi Rich,

So I tested this on another 7.2 deployment (so I have 2 vRO instances now) and the same thing happens with Host Rescan. The funny thing during the testing is that if I leave the host value blank in the workflow (by default and it is not required) - it fails with the error above.

However, if I manually give it a host in the host value to rescan, it works and runs fine - so it looks to be something maybe with the code of that workflow to discover "all" hosts in the vcenter?

The issue I have now is that I can't really hardcode a host on the rescan and need it to do all the hosts since that is how it would be wanted in the real world and because I also don't chose any host value (not required and defaults to all hosts) in the Datastore Expand workflow. This sounds like we are onto something with the Host Rescan not working for all hosts in a vcenter (the default option if nothing is supplied).

Do you know if that is a known issue or if that works maybe in one of your environments?

I can also attest to the fact that I think you were right with the host rescan inside the Datastore Expand workflow to begin with because if I run that to failure, the XtremIO LUN increases and if I then manually rescan a host in virtual center, it will allow me to extend it in vcenter - so it seems the rescan is critical.

Let me know what you think. Thanks!!

-Keith

Hi Keith,

sorry for the delay I was not well this morning, better now.  So we definitely have some issues with vRO 7.3.  the "host show" and "host conformance" workflows stopped working as a result of some architectural changes they made in vro 7.3 around the indexof operator.  Just this week we finalized with vmware how we would resolve this issue. As it happens are coming out with a XtremIO vro 2.0.0 plugin that will allow these two workflows to work with vro 7.3. as well as previous 7.x vro releases.  So for now please stick with the vRO 7.2 release until the 2.0.0 XtremIO plugin is available.

As far as host rescan goes yes we had a bug in that workflow and it's exactly as  you describe. If you select an individual host it works, if you don't  select a host it fails as you saw.   let me work on getting you and updated Host Rescan workflow.  Please mention via the case that you opened for them to contact me regarding the Host Rescan Issue. We should be able to get that to you very shortly. I'll also look into having Datastore Expand modified to do a host rescan. 

Out of curiosity do you have one cluster in your vcenter server or multiple clusters?

thanks

Rich

Hi Keith,

we have a new 1.1 vRO package for you the version is 1.1.1-3. It fixes the Host Rescan issue and has a few other fixes as well. As soon as I get contacted by customer support we should be able to pass along the kit to you and you should be all set. We will try and get the "datastore expand" workflow updated to include a call to the host rescan workflow for the 2.0.0 release.

thanks

Rich

Hi Rich!

You rock...thanks. I will let the support case owner know.

That is great info and I will stick with the 7.2 deployment I did today instead of the 7.3 deployment. I'm not sure what the shift schedule the support owner is on, but I will reach out to them now...stay tuned.

Thanks again!

-Keith

Hi Rich,

The new packages were spot on! Thank you very much for the help. I am sticking with vRO 7.3 and the new 1.1.3 package - the host rescan works for all hosts in the vcenter. I am gonna test a little more this week and get to the scripting process of inserting the host rescan into the datastore expand function, but a manual run ran fine and I wanted to let you know and thank you for the help. I am traveling this week so when I get back at the testing, I will post updates here. Thanks again!!

-Keith

Hi Keith,

good news. With regards to using vRO 7.3.0, In addition to the Host Show and Host Conformance workflows not working (along with any workflows that call these workflows) the vCenter Add workflow also does not work. We are going to deprecate that workflow in the 2.0.0 release of our vRO package. To add a VMware vCenter Server instance to vRO one way it can be done is to use the VMware supplied workflow "Add a vCenter Server Instance".  This workflow can be found in the vCenter\Configuration folder.

/Rich