VNX 5500 with iSCSI Setup Very Slow Storage vMotion

Thank for quick response Itzik, I shall follow your recommendation tomorrow. Just a quick question "unispere failover wizard which needs to be set to failover mode=4" I assming Unisphere is the management for storage and failover wizard is accessable through SAN management interface. Therefore, I do not need to install any additional software on vCenter server.

Will post back the result tomorrow.

Thanks Again

Shaikh

Yes, as you point out, Unisphere is the Interface used to manage the VNX and the "Failover wizard" is available from within Unisphere via the "Hosts" menu.

I also wanted to make an additional comment about VAAI and the hardware offload.  The datastores participating in the Storage vMotion must also have the same block size or else it will failback to the "non-accelerated" method.  However, "30 to 40 mins moving 5 GB file" even for a traditional Storage vMotion is slow.

Finally, also have a look at the following post:

"Info - CX4/VNX Delayed ACK (ISCSI) Performance Issues"

https://community.emc.com/thread/124266?tstart=30

Hi Shaik

Glad we coud help! There's nothing that make me feel better than seeing issues getting resolved, well, except for seeing my daughter but anyway:

1. The best way is to create one vSwitch with two different vmKernel port groups when one nic is the active and one is not used and vice versa.

2. Ip hash is often used for the vmnetwork and not the vmkernel network, in any case it requires an lacp enable and configured switch

3. RR is sending 1000 iops by default and yes you can change it to =1 or even install powerpath/VE that will take care of te load balancing in am even better way (not to mention failback of a failed port, better management etc'..

Itzik Reich

vSpecialist Technical, EMEA South

ב-Sep 12, 2011, בשעה 11:09 PM, "Shaikh" > כתב/ה:

EMC Community Network

Re: VNX 5500 with iSCSI Setup Very Slow Storage vMotion

created by Shaikh in Everything VMware At EMC - View the full discussion

Hi,

Firstly thank you both. Massive improvement after making the necessary changes as you suggested. I am now getting following results:

ESXi Host 1

2 x 10GigE Network Adapter, EtherChannel on Cisco N5K stackable switch and IP Hash on vSwitch as load balance policy. VMNIC assignment

• VMNIC0 -->vSwitch0 (VMNIC1 is configured for not used)
• VMNIC1-->vSwitch0 (VMNIC0 is configured for not used)

I have done all the necessary binding for iSCSI side on ESXi hosts and using RR policy

Virtual Machine (40GB) - 37GB used and 3 GB free. Actual Storage vMotion finished under 6 minutes or sometimes under 5 minutes.

ESXi Host 2

2 x 10GigE Network Adapter, edge trunk on Cisco N5K stackable switch and route based on vSwitch as load balance policy. VMNIC assignment

• VMNIC0 -->vSwitch0 (VMNIC1 is configured for not used)
• VMNIC1-->vSwitch0 (VMNIC0 is configured for not used)

I have done all the necessary binding for iSCSI side on ESXi hosts and using RR policy

Virtual Machine (40GB) - 37GB used and 3 GB free. Actual Storage vMotion finished in 2 minutes and 30 seconds.

Questions

1. Should I be configuring single port group with 2 x VMNICs assigned as active?
2. Should I be using IP Hash or Route based?
3. Is there any benefit me configuring “esxcli nmp roundrobin setconfig –useANO 1”? by default it is 0

Look forward to hearing from you soon.

Cheers

Shaikh

Excellent thank you.

What are the differences in powerpath basic mode and licensed mode?  In addition do you know are the minimum plug-in I need to install on vCenter to offload cloning, zeroing etc function to Storage Array?

I was reading VMware KB http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1004048 recommend not too have that kind of setup with IP Hash policy but fully acceptable in route based policy.

Note: The only load balancing option for vSwitch or vDistributed Switch that can be used with EtherChannel is IP HASH.

• Do not use beacon probing with IP HASH load balancing. 
• Do not configure standby or unused uplinks with IP HASH load balancing.

Going to conduct future test tomorrow with active/active single port group with IP Hash policy to see any further performance again.

Cheers

Shaikh

Hi

There isn't really a basic mode for powerpath/VE as it has to be licensed, technically speaking, basic mode works just like you don't have powerpath installed.

NO vcenter plugin need to be installed in order to support VAAI, the confusion comes from the often used term "plugin" but really as long as you have a VNX it already comes with VAAI enabled (remember to enable failover mode=4 so the esx host will support it)

Itzik Reich

vSpecialist Technical, EMEA South

ב-Sep 12, 2011, בשעה 11:38 PM, "Shaikh" > כתב/ה:

EMC Community Network

Re: VNX 5500 with iSCSI Setup Very Slow Storage vMotion

created by Shaikh in Everything VMware At EMC - View the full discussion

Firstly, I just wanted to acknowledge the awesome feedback that was exchanged.  Thanks Shaikh for sharing your research and experiences, and I always look forward to what our vSpecialists like itzikr have to share and is definitely a group I have a lot of vspect for.

I simply wanted to comment on one thing, Shaikh you asked about configuring -useANO=1.  The "useANO" and "IOPS" values are not the same thing.   You should not enable -useANO=1 (Use Active-Non-Optimized) as noted in the following post:

http://www.yellow-bricks.com/2011/03/29/alua-and-the-useano-setting/

By setting that, you are telling your hosts to include the non-optimal paths even in a healthy environment where all configured paths from the host to the VNX are active/alive.  A non-optimal path would be a path (as nicely diagramed in the post above):

1. from the host to the owning SP's peer
2. then through the CMI (CLARiiON Messaging Interface)
3. then to the owning SP

By leaving it at 0 (default), then unoptimized paths reported by the array won't be included in the RR path selection until optimized paths become unavailable.

To borrow some of the verbiage from the post above:

• "Disclaimer: Now, lets make this very clear. Don’t touch “useANO” unless you are specifically instructed to do so, this article is just for educational purposes."

• "As you can understand having traffic flowing through a non-optimized path is normally something you would want to avoid as this will cause latency to go up. (more hops)"

Also, not mentioned is in the post is that the CMI is where the write cache is mirrored between SPA and SPB so adding traffic across that link could have an impact across your environment as writes by your hosts are only ack'd by the VNX when the write (to cache) is mirrored.

Hi

Performing virtual machines clone x3 to different datastore same time with 60 GB disk. It is taking 1 hour and 2 minutes to complete. Not sure if clone process should take this long?

Block size is 8 MB on all our datastores.

Are you sure I do not need any additional VNX software on vCenter to offload this process to storage?

Kind regards,
Sultan

Apology about my previous reply should have carried out further tests before posting it. Slow clone appeared to be VMDK issue. After creating a new template virtual machine clone process completed in less than 3 minutes.

We had to drop IP Hash teaming with Ethechannel for iSCSI as there are limitations with VMWare iSCSI software initiator can it cannot bind to two physical network adapters.  We have gone with “Route based on the originating virtual port ID “.

KB: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1004048 and http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001251

Above KBs suggest IP has can be used but when you have both cards Active / Active settings than go and bind VMK port to VMHBA its say’s cannot bind. Only way to bind is to drop one VMNICx on unused. So bit confused over this.

All seem to be working ok as expected but pre-production testing beings tomorrow.

I have final few questions if you are able to answer them would be great help to me.

Understand Windows 2008 now align the disk automatically to 1 MB. However, we are looking to do P2V lots of Windows 2003 servers which likely to have mis-aligned disk. All our VMFS datastore is set to 8 MB. Backend storage is VNX 5500.

1. What size should I align Windows 2008 virtual disk over 4 GB and less than 4 GB?
2. What size should I align Windows 2003 virtual disk over 4 GB and less than 4 GB?

I am aware of DISKPART tool within Windows OS to create new aligned disk but not sure of any tool which will re-align disk which has been already been created and contains data.

Can you recommend any tool/utility which will do the trick?

Kind regards,
Shaikh 

No additional software is required than what you already have, the fundamental requirements are as follows:

1) VAAI capable array which the VNX is (and has been since its initial release)

2) ESX(i) 4.1 or later servers (NOTE: not relevant for this conversation, but support for NAS storage devices was introduced in ESXi 5.0 and requires the latest VNX OE for block code)

3) Hosts registered with ALUA failover mode of 4

Once the foundation above is set, then for the rest of the requirements to look for and a list of things that would disqualify it from the hardware offload (such as the "Advanced Settings", the "Hardware Acceleration Support Status" for each storage device and datastore, etc.) have a look at the following VMware KB article (1021976) for a complete and thorough list (has since been updated to include ESX 5, btw):

vStorage APIs for Array Integration FAQ

For example, as noted in the KB article, there are several other items (one of which was mentioned above about similiarity in block size) that would disqualify it:

[...]

VAAI hardware offload cannot be used when:

• • The source and destination VMFS volumes have different block sizes
  • The source file type is RDM and the destination file type is non-RDM (regular file) 
  • The source VMDK type is eagerzeroedthick and the destination VMDK type is thin
  • The source or destination VMDK is any sort of sparse or hosted format
  • Cloning a Virtual Machine that has snapshots, since this process involves consolidating the snapshots into the virtual disks of the target Virtual Machine. 
  • The logical address and/or transfer length in the requested operation are not aligned to the minimum alignment required by the storage device (all datastores created with the vSphere Client are aligned automatically)
  • The VMFS datastore has multiple LUNs/extents spread across different arrays

[...]

Then, once you confirm your environment is compliant (or simply as a test to confirm), the best way to verify that the hardware offload is improving things is actually performing tests with and without it enabled.  Again, as itzikr mentioned above it is important to note that tasks via VAAI improves efficiency (and scalability) but efficiency often does equate to faster times as reported by most. 

There are many articles and videos showing other users' comparisons and how to make these.  I'd recommend the following video from Chad Sakac where he is demonstrating a benchmarking process and review via esxtop (CMDS/s) and the vSphere "Performance Charts" that work is, in fact, being offloaded from the host.  Also, he looks at it from the array's perspective via Unisphere Analyzer (licensed feature). 

vSphere 4.1 and EMC - VAAI - Hardware Accelerated Full Copy

If after seeing VAAI kicking in and you still feel performance doesn't match your calculated expectations, then this helps as you know to look elsewhere. 

Sorry about that Shaikh.  I had posted my last comment minutes after I realized you replied with your updated observations.  Ignore my last post now that you are past VAAI validation and troubleshooting.

Thank you all for your prompt responses. Excellent responses helped me alot.

Hi Christopher - Hardware Offloading is working fine with newly created template under 3 minutes which is good.

Did some digging around why the old template VMDK disk had issue. It appeared my colleague originally created it as thin disk than try to inflate it. However, half way through process it terminated than he tried again which completed successfully. I can only assume there must be some corruptions due to termination of inflate process.

Where as with Hardware Acceleration disabled takes around 10-15 minutes. VM template which has 60 GB disk  with base Windows 2008 R2 installed.

I have noticed few lun trespassing which I will looking into tomorrow.

Thank again.

Shaikh

Hey, sorry to revive an old thread.

Has anyone had any luck getting this too work on VMware ESXi 5.0?  Simply put I can't.  I've got a VNX 5500 (latest code base), and esxi 5.0 systems and I can only get the hosts to mount the LUN's if I buld the initator as two seperate vswitches, with a single nic in them, then put them in a Port Group together under the network configuration tab under iSCSI initiator properties.  everything works fine or seems too, I get 8 paths with 4 active and 4 failover.  Any suggestions as I am afraid this is not the best practice on setting this up and is definately not the normal way.

Thanks

Blake