Hi Christopher,

Thanks for your reply.I have asked thecustomer to confirm what patch version they are on, the arrays are both on Ver32 but I am unsure what patch level. The last update was done toward sthe end of last year so chnaces are they are still behind on the leatest level. They tries to VMotion a 50GB VM last night and it took 2hrs!

Thanks

Ken

Customer is on 05.32.000.5.008 so an Sr has been raised to have this upgradaded.

Thanks for your help

So in the meantime, you could/should consider temporarily disabling the XCOPY primitive on the ESX/ESXi hosts  (Datamover.HardwareAcceleratedMove = 0) as mentioned in the KB article of course assuming the environment supports VAAI.  This also give you a chance now to verify that with it disabled that an svMotion then takes less than 2 hours which suggests this is the culprit instead.

On the other hand, you could also hold off from performing anything that triggers the VAAI XCOPY primitive (cloning, svMotion, deploying from template, etc) and then after the upgrade try again and see if you get the expected results.

The customer has logged a service request for theupgrade to be carried out. I will rpeort back on the results once the customer has fedback to myself.

Thanks

I have a single VNX 5300 and am experiencing the same issue with svMotions while DataMover.HardwareAcceleratedMove = 1.  If you check your vmkernel.logs, are you getting similar errors to these while the svMotions are running?

2013-03-26T13:56:00.024Z cpu8:3124900)ScsiDeviceIO: 2303: Cmd(0x412400173940) 0x2a, CmdSN 0x2c from world 8204 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.

2013-03-26T13:56:04.450Z cpu6:3124900)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" state in doubt; requested fast path state update...

2013-03-26T13:56:04.450Z cpu6:3124900)ScsiDeviceIO: 2303: Cmd(0x412408513540) 0x2a, CmdSN 0x2f from world 8204 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0.

2013-03-26T13:56:04.521Z cpu3:8215)ScsiDeviceIO: 2285: Cmd(0x412405882a40) 0x28, CmdSN 0x2e from world 8267 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x8 D:0x0 P:0x0

The issue described in emc313487 sounded like the problem we are experiencing also but even after upgrading to 05.32.000.5.201, which marks the issue as fixed, we are still experiencing the same problem.  The only way I have been able to workaround the problem is by setting DataMover.HardwareAcceleratedMove = 0 on all hosts connected to the VNX.  With it set to 1 (on), a 50GB VM will take 1+ hours to svMotion with these errors occurring every few seconds during the move.  With it set to 0, no errors and svMotions of the same VM take literally less than 2-3 minutes.  If I start an svMotion while DataMover.HardwareAcceleratedMove = 1 and then let it move slowly along for a while and then set DataMover.HardwareAcceleratedMove = 0 while the svMotion is in progress, the errors in vmkernel.log cease immediately and the svMotion completes successfully in less than a minute.

I would be interested to know if 05.32.000.5.201 fixes the problem for your customer because it has not fixed it for us.  Prior to upgrading to P201, we were on 05.32.000.5.011.

ceestep,

Quick question, I believe you left out the associated "physical path" from the error.  By chance is the LUN ID (L:) of that physical path = 0?  Also, can you confirm that LUNZ is masked out?

Regardless, there are many hits reported by users experiencing this across a wide range of arrays, so not certain that it is  specifically a VNX issue (not to say it isn't contributing to it though).  I'm sure you've seen the same list for the SCSI error codes:

http://kb.vmware.com/kb/1029039

I wonder if maybe we are looking at (host) queue issues and may need to look at the related settings.  Do you mind providing more information about the environment?

1) ESX/ESXi version

2) SATP (related to Unisphere host failover mode)

3) PSP (Round-Robin or Fixed)

4) Protocol: FC, FCoE, iSCSI (generic NIC, dependent/independent iSCSI adapter)

Just curious that's all.

No it's not L0 showing up in the vmkernel.log  The physical paths are not present in the errors I pasted above but they are in other excerpts of the log.

2013-03-26T13:54:57.001Z cpu10:3124900)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x2a (0x412407474880, 8204) to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" on path "vmhba38:C0:T1:L1" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0. Act:EVAL

2013-03-26T13:54:45.932Z cpu6:8326)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:647: Path "vmhba38:C0:T1:L12" (UP) command 0xa3 failed with status Timeout. H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

All my ESXi hosts are 5.1.  SATP as seen in the one error above is VMW_SATP_ALUA.  The LUNs are presented to ESXi as failover mode 4.  PSP is currently Round Robin although I have tried Fixed and the same issue occurs,  We are using iSCSI 10GB with the iSCSI Software Adapter (vmhba38 above).  Behind the scenes, the vmks associated with the software adapter  are configured on two ports of an Intel x520 (82599EB) NIC.

I'm curious as to how closely that matches to Ken's system, specifically if he is using iSCSI also, and if so, is he seeing the same types of errors in the vmkernel.logs.

EDIT:  Oh also, there aren't any MASK_PATH claimrules for L0.  Should there be?

Hi,

The customers arrays have been upgraded and this made no difference to the issue. If they try to use storage vmotion to move a powered on VM from one datastore on a Recoverpoint protected LUN to another datastore, also on a recoverpoint protected LUN it slows to an unacceptable speed and there is disk latency of 5 - 6 seconds which obviously has an impact on other VM's on these LUNs. Recoverpoint then also throws errors about possible splitter failures (Splitter is array based) and Recoverpoint also enter a high load state and then pauses transfers. If they use storage vmotion to move a VM on a non Recoverpoint protected LUN to another non Recoverpoint protected LUN it works fine. The only way we can get it to work as it should is by having DataMover.HardwareAcceleratedMove = 0 so disabling VAII. Once this is done the storage vmotions work as they should with Recoverpoint protected LUNs

Hope this helps someone

Ken

Yes I do have an open case with EMC at the moment.  I spoke with an EMC tech yesterday that had me gather trace logs and SPCollects again "for the engineers."