Question on VAAI - VNX - STORAGE vMotion - SP LUN OWNERSHIP

what flare version ?

See emc287223

As a best practice, it is highly recommended to use LUNs of the same owning Storage Processor for such VMware operations (VAAI)

wow, how impractical ...i want to distribute my workload between SPs not jumble them together on one SP or everytime i need to do svmotion check lun ownership.

VNX 7500

Block Software Version : 05.31.000.5.509

File software version: 7.0.40-1 

VNX 5700

Block Software Version : 05.31.000.5.509

File software version: 7.0.40-1  

VMware ESXi 5.0.0 build-469512 

EMC powermt for PowerPath (c) client Version 5.7 (build 173)

EMC PowerPath (c) Version 5.7 (build 173)

http://virtualgeek.typepad.com/virtual_geek/2011/09/vnx-and-vnxe-updates-and-vaai-hotfixes.html

The difference is incredibly huge. I have checked White Paper "VAAI for array integration with VNX for SAN" and haven't found requirement to put replication LUNs on different SP, etc. I suspect it's a code bug issue. VAAI fix mentioned by itzikr is recommended to install to see if it can be fixed.

Please refer to emc305253.

Not the first time that VNX engineering (and/or other storage engineering), PowerPath engineering not talking to each other, or even talking to each withing a very large scale of engineering team. Some times they the talk to VMware NMP team ...

It's time for PPVE engineering & support team to follow up and update emc272442 emc287223...

===

Here is the NEW on for VMware NMP:

"Unexplained trespass with ESX using NMP and Unified OE Block ALUA"

ID:          emc305253

Usage:          2

Date Created:          10/03/2012

Last Modified:          11/06/2012

STATUS:          Approved

Audience:          Customer

Knowledgebase Solution

Question:          Why are unexplained trespasses occurring on an ESX host using VmWare Native Multipathing Plugin (NMP) and Unified OE Block with ALUA?

Environment:          Product: VNX Unified

Environment:          EMC SW: VNX Operating Environment (OE) for Block 05.31

Environment:          OS: VMware ESX 4.1 and later

Environment:          OS: VMware ESXi 4.1 and later

Environment:          Application SW: VMware vStorage API for Array Integration (VAAI)

Environment: This statement does not apply: OS: VMware ESX  5.1

Problem:          Many unknown path trespasses occurred without any path failures reported.

Change:          VMware operation (cloning, deploying VM from template, etc.) from one LUN to another LUN owned by a different storage processor (SP).

Root Cause:          The VAAI feature enables the ESX host to offload specific virtual machine (VM) and storage management operations to the array. In cases where the operation involves two LUNs owned by different SPs, redirection of I/O occurs. When the number of redirections reaches a certain threshold, the redirector will perform an internal trespass to optimize the I/O load. In the event where there are both high application I/Os and VAAI Xcopy I/Os, it is possible that the LUNs will trespass back and forth.

During implicit trespass such as these, Native Multipathing Plugin (NMP) will receive a unit attention informing it of what happened. At this point the host will issue an explicit trespass, moving the LUN to the other SP. When the operation is completed, if the failover policy is set to Round Robin, the LUN will not be auto-restored. If failover policy is set to Fixed, the LUN should auto-restore to the correct SP.

Fix: This issue with the implicit/explicit code returned by the array has been fixed in VNX OE 05.31.000.5.726. It does not fix the trespass issue, that is still by design and attention should be taken to return trespassed LUNs to their Default owner once the operating is complete. In ESX 5.1, auto-restore will be added for Round Robin, so if the policy is set to either Fixed or Round Robin, LUNs should be restored back to the Default owner.

PS: What if we're running Inyo code... 05.32

Hi Baif,

It's still a problem with INYO... it's impacted us :/

We have disabled VAAI pending a support case from EMC.

You may also refer to the one I mentioned which was reported against PPVE. And OK... it hasn't been fixed in INYO.

"Unexplained trespass with PowerPath/VE 5.4 SP2 and CLARiiON ALUA."

ID:  emc287223
Usage:  4
Date Created:  02/06/2012
Last Modified:  02/08/2012
STATUS:  Approved
Audience:  Customer

Knowledgebase Solution

Question:  Unexplained trespass with PowerPath/VE 5.4 SP2 and CLARiiON ALUA.
Environment:  EM SW: PowerPath/VE 5.4 SP2
Environment:  EMC Firmware: CLARiiON Failover mode 4 (ALUA)
Environment:  System: VMware ESX/ESXi 4.1 and later
Environment:  Application SW: VMware vStorage API for Array Integration (VAAI)
Problem:  PowerPath: EmcpEsxLogEvent: Info:Mpx:Assigned volume 6006016027A02900044C64BFA6DAE011 to SPA
Problem:  Many unknown path trespasses occurred without any path failures reported
Change:  VMware operation (cloning, deploy VM from template, etc.) from one LUN to another LUN owned by a different Storage Processor
Root Cause:  With VAAI feature, it enables the ESX host to offload specific VM and storage management operations to the array. In cases where the operation involves two LUNs of different Storage Processors, redirection of I/O occurs. When the number of redirections hits a certain threshold, the redirector will perform an internal trespass to optimize the I/O load. In the event where there are both high application I/O and VAAI XCOPY I/Os, it is possible that the LUNs will trespass back and forth.
During implicit trespass such as these, PowerPath will receive a unit attention informing it of what happened. Because PowerPath has no way to tell if the load has changed, PowerPath will not restore the previous LUN ownership.

Fix:  This behavior is working as designed. As a best practice, it is highly recommended to use LUNs of the same owning Storage Processor for such VMware operations.
Notes:  When an implicit trespass is triggered, PowerPath should be able to sense this and update its internal records by the "followed to SPx" messages. Instead, PowerPath is sending an additional explicit trespass command. That is where we see the "Assigned volume to SPx" messages. However, it is sending to current owning Storage Processor. Hence, there is no impact. On the other hand, the logging of these messages is a PowerPath bug and is being looked at by engineering.

Hi Baif,

unexpected LUN trespasses caused by VAAI XCOPY is already addressed to our engineering.

To the best of my knowledge, the condition exists in VNX Block version R31 and R32 and will be corrected in R32 MR1.

But I don't have any information when we could expect this revision.

Hth,

Ralf

Ralf!!! Thanks!!!!

According to:

https://community.emc.com/thread/132762

https://community.emc.com/message/704402

https://community.emc.com/people/GearoidG/blog/2013/02/08/extended-copy-xcopy-vaai-primitive-not-behaving-as-expected-with-vnx-oe-release-32--emc313487

And NOW R32 MR1 is out!!! Please kindly help to verify if issues we mentioned are supposed to be fix by R32 MR1:

emc287223 "Unexplained trespass with PowerPath/VE 5.4 SP2 and CLARiiON ALUA."

emc305253 "Unexplained trespass with ESX using NMP and Unified OE Block ALUA"  

http://virtualgeek.typepad.com/virtual_geek/2013/02/vnx-vaai-fixes-smb-30-inyo-mr1-now-released.html

February 22, 2013

VNX VAAI fixes + SMB 3.0 + ODX – Inyo MR1 now released

In EMC-land, we have maintainance/service releases for major products – and often these don’t get the glory of the big launches, but contain a TON of important stuff.

Inyo is the code name for the currently shipping VNX Operating Environment that covers NAS and Block functionality (VNX OE 32/7), and Maintenance Release 1 (MR1) shipped to manufacturing on the 19th and is available for general download today (feb 22nd).

This contains a ton (read the release notes here – you can find all sorts of stuff on the VNX page on support.emc.com which is here), but there are 3 things I want to call out:

1. Critical VAAI block XCopy fixes (Primus emc313487 – go to http://support.emc.com and just search for the Primus case)