High latency and failed clone of virtual machine

I will look in to that - I need to find a doc that tells me how.  ;-)

One additional thing that I found was interesting.  My test LUNs are thin, as that's what my vendor recommended I do.  I provisioned a thick LUN and the problem is gone - clones are wicked fast, as are block-zeroing operations.

Should I be able to run full VAAI operations against a thin LUN?

Thanks

And some additional detail - cloning from thin LUN to thick LUN works, as does thick to thick and thick to thin.  The only time that the clone fails and I see the issues noted above are when cloning or migrating thin to thin.

Dave

Thanks Beagless - I'll mention this to support.  I don't know that it's related to my issue (mine fails before any data is copied and I'm running vSphere 4.1), but you never know...

I'll report back soon!

Further detail - after kicking off auto tiering and allowing it to complete, now nothing works - all clones and migrations fail, be it to a thick or thin LUN.

However if I turn VAAI off on the host, all works fine.  If I turn it on, the problem is back.  I have a case open with EMC and my account team and will post the solution when they find it.

hi ddavenport

have a read of this it may be related

http://virtualgeek.typepad.com/virtual_geek/2011/10/urgent-vaaithin-provision-reclaim-on-hold-workaround.html

search through that site the blog is done by and EMC guy and he has posted a good bit of stuff in realtion to vaai

it may help

it will be interesting to see what comes from the service request

I'm seeing an extremely similar issue. Did you get any resolution yet? We're running 4.1 Update1 on our hosts and a VNX 5700. Anytime we have VAAI turned on and try either a clone or sVMotion we see extremely poor performance. We aren't seeing any complete task failures, but it gets so bad hosts will intermittently disconnect from vCenter and response time go to 6000-8000ms on the source/target LUNs. All the other LUNs are also impacted with latencies of 600+ms and our view clients are frequently disconnected from their sessions during these events.

Yesterday we did a code update to latest on the VNX and saw some improvements w/out VAAI, but still the same awful mess w/it enabled.

Beagless, the UNMAP primitive isn't supported on ESXi 4.x, so we're not going to be directly impacted by that specific issue, but that definitely doesn't discount the idea that EMC may have altered VAAI processing elsewhere recently...

Hi KRNewell,

Well, I can confirm that you aren't crazy - I'm working with L2 support, the Clariion core team, and VMware support on this and they have acknowledge the issue and informed me that I'm the 4th customer to report this.  We've been going back and forth since the 27th (when I first posted) and have run every test under the sun to try and get to the bottom of this.

Support has acknowledged that the problem is indeed with VAAI (makes sense...when I turn it off, everything works perfectly fine).  I have a call with the support teams at 13:00 PST today, where they will be giving me their recommendation and letting me know if this will be in the next official block code update, if it will be an EMC patch, or if it will be a VMware patch / update.

As soon as I know I will update this post...

All VAAI discussions aside, I'm pushing these units in to production with VAAI disabled and will enable it after the fix is available and verified.

Dave

Can you email me with your case #? I'd like to give it to the engineers working on our EMC case so we can try to get them up to speed more quickly. They have yet to reference any other instances of this issue.

Thanks
Ken

Here is what EMC engineering recommended and it works. In the mean time, I was told they are working on a patch for this.

Root cause: RecoverPoing Splitter intefers with VAAI Copy mechanism.

Temporary fix: Customers who are actively using RecoverPoint (as opposed to simply having the enabler installed) and who experience the problem should disable HardwareAcceleratedMove until a patch is available. Customers who are not actively using RecoverPoint should uninstall the enabler, and leave HardwareAcceleratedMove enabled. Use USM in Engineer mode to uninstall RPSplitter driver.

Yes, they asked us to try this as well, however we aren't using the RecoverPoint Splitter, so it didn't apply.

From what we hear, engineering is still working on it...

Dave

Yeah, I'm actually seeing it happen both on the same LUN, and across-LUNs.

We did not use RecoverPoint either.

As long as the driver is installed there is high latency when source

cloning source and target are on the same LUN.

But I am not sure if you are cloning within the same LUN.

Regards

Mu-Yeh Bau

Americas RPE SA&E, Platform Engineering-Storage Engineering

CSC

3170 Fairview Park Dr., Falls Church, VA 22042

Managed Services Sector | Office: +1 214-578-0452 | mbau@csc.com |

www.csc.com

mbau@csc.com

This is a PRIVATE message. If you are not the intended recipient, please

delete without copying and kindly advise us by e-mail of the mistake in

delivery.

NOTE: Regardless of content, this e-mail shall not operate to bind CSC to

any order or other contract unless pursuant to explicit written agreement

or government initiative expressly permitting the use of e-mail for such

purpose.