Poor VM performance on iSCSI Datastore

Question

Hey All, AFAIK this is my first post on the community forums. I generally spend lots of time on the VMware IRC channel. As I'm not aware of a EMC based one. Anyway I'll get my infrastructure information out of the way so we have a baseline. I'm running a VNXe3150 with the latest firmware. I have been pretty displeased with this box since we got it, but I won't get to the many points that brought me to that conclusion I'll just get what I've observed. After much research I decided to move from NFS to iSCSI, mainly for MPIO so I could have increased svMotions when I need to make infrastructure changes. I did lots of networking redesign to free a bunch of ports on my Cisco 3750 (stacked) core layer 3 switches. I created 2 new vlans (22, and 24). I used the VNXe built in nics Eth 2 and 3 (on each SP respectively)  giving them each an IP as required (SPA iSCSI A 22.1 and 24.1) (SPB iSCSI B 22.2 and 24.4). I then created 2 new vSwitches on each of my ESXi 5.5 hosts, each with their own VMKport bound to one NIC each. (Host 1 22.200, 24.200) (Host 2 22.201, 24.201). I then verified each hosts software iSCSI initiators where set. Now in Unisphere, I have my hosts added to it, so Unisphere can configure my new datastore on my host when I create a VMware based datastore. There are a couple thing however that I noticed. It does add the static targets to the initiators and re scans and mounts the datastores; however. 1) I have to convert them to VMFS 5 from on of the hosts Storage section settings 2) I have to set the load balance algo for the managed paths from Fixed to Round Robin. Once I did this all, I was able to verify that MPIO was working and the load was being distributed equally. I wasn't too impressed with the increase performance since instead of hitting a 1 gbps limit as with NFS, I was only able to hit 140-150 MB/s, this is roughly only 55% of both gbps NICs. Even with this I expected a increase in VM performance for IOPs. To my dismay it was not what I had expected. See Test 1, VM on Hosts local Datastore, consisting of 6 - 300 Gig 10K SAS drives in - Pastebin.com I kept testing and trying things, then people on IRC told me about the Round Robin setting to change from iops=1000 to iops=1, which basically tells the host to load based on each iop instead of every 1000 IOP. After I made this change my mind was Blown. However my VM performance on the iSCSI datastore remained the same. What am I missing? (I already read the posts about NFS cache, when I check the settings under advances settings, cache is DISABLED)

Zewwy · Answer

Sorry I should have mentioned that my NFS connections to my hosts are based on dedicated BONDED nics (LACP) to my 3750's. I originally had my ESXi connections to the 3750's for the NFS subnet was bonded (etherchannel group # mode on) with Load Algo of (Route based on IP hash). I made the change on those connections to be not bonded, and changed the VMware load algo to (Route based on originating Virtual ID) the default. for NFS performance it didn't change.

The iSCSI network are still as specified separate subnets, each on a different switch (seperated via vlan tagging), No jumbo frames enabled (still on my todo list)

Also, I noticed I forgot to mentioned Why I was blown away by making the IOPs change from 1000 to 1. My svMotion speed over iSCSI went from 140 MB/s to over 230 MB/s. Th eincrease in performance for that was mind blowing. Which is why I was sad to see VM performance of I/O still being as poor as it was. See attached pastebin for test results.

Zewwy · Answer

I'd appreciate some input from others. After more testing, and googling. I found my results are very similar to this persons results. The only difference is I'm running all my Test on a Windows Server 2012, with no A/V installed. I even tried skipping the hypervisor layer by adding the VNXe iSCSI disc directly to the Windows VM (applying proper MPIO) and I get the same poor performance on Random I/O.

https://www.reddit.com/r/sysadmin/comments/2zwqpy/slow_iscsi_randomwrites_on_esx_55_to_netapp/

Zewwy · Answer

Bump Bump Bump... Anyone... someone has to be a storage expert at EMC.....

VNX

Was this post helpful?