This post is more than 5 years old
7 Posts
0
22990
MEM is slow with vSphere 5?
Three Poweredge r710s running vSphere 5, software ISCSI on Intel NICs, running through two HP switches (lower end switches) physically separated from all other network traffic and lagged together. Note that the hp switches have an issues running jumbo frames and flow control so I am using flow control without jumbo frames.
The SAN is a PS4000 with 600 GB SAS (10 or 15k I forget) in two raid 5 LUNs with dual controllers.
After installing MEM the throughput plummets to about 1/3rd of single ISCSI connection as measured on my backup servers speed/network utilization. I have been using MEM for quite sometime with ESX 4i so it is not a settings issue (not to mention the script runs you through the install) and have verified multiple paths are "active".
As soon as I change the setup back to a single active path the backup process and network utilization triples in speed.
Anyone having any luck with the new MEM and vSphere 5?
Thanks,
Bob
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
1
December 12th, 2011 17:00
So if RR made it worse, then the issue isn't MEM. It really sounds like the switch can't handle the additional traffic. The 28xx series is one I've worked with in the past and it tends to fail at higher loads. If it is, if you get the latest FW for it, there's a setting you can enable called "qos-passthrough-mode one-queue'" That will provide you the max buffer / port. By default it splits the available mem (which is small to start) into four QoS queues. So iSCSI only has access to 1/4 of the buffers.
With HP switches you are looking for switches that have the "Provision" chipset. That allows both Jumbo Frames and Flowcontrol. Also they tend to have more buffers as well.
Regards,
Osm3ums
7 Posts
0
December 11th, 2011 21:00
Note, I think the two ports being used are on a single quad port Intel card, could that be an issue? The only other ports I have available are broadcoms, but prefer to not mix manufactures when doing ISCSI.
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
1
December 11th, 2011 21:00
I've not seen any performance issues reported against MEM 1.1. What happens if you change it to VMware Round Robin.
The HP switch, would that happen to be a 28xx series switch?
Regards,
Osm3ums
7 Posts
0
December 12th, 2011 17:00
RR was slower than MEM which is slower than a single active connection. Very strange as MEM on ESXi 4 made a huge difference.
Not sure on the model of the switch but will check. I can see a number of paused frames as the ISCSI is over running the buffers on some of the ports. I wonder if the combination of over running buffers while trying to increase the throughput is actually making it worse?
Any thought on trying to use two ISCSI port on a single 4 port Intel NIC causing speed issues? (Other than the fact it is a single point of failure!)
Thanks for your time,
Bob
Osm3ums
7 Posts
0
December 22nd, 2011 16:00
The switches are HP 1810s. Best I have available for now.
However, I changed my ISCSI NICs to the embedded Broadcom from the single four port Intel NIC. The throughput has tripled (or more) in speed. So it appears to have been something with the Intel NIC (which is not embedded).
Sorry to say the above as I realy prefer Intel NICs.
Thanks for your time,
Bob
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
December 22nd, 2011 16:00
You definitely need to replace that switch though. That is not a good iSCSI switch. That's designed for GbE to desktop systems. Once you get going or worse if you added another array that switch will fall over. For small environments, the Dell PC5424 are very good iSCSi switches and not expensive. After that it's 6224's which are stackable and more scalable. HP29xx series would be the minimum starting point in the HP line.
Make sure you have updated ESXi v5 to the latest (aka Patch2) build. The build number will be 514841 when you're done. That has a number of fixes including NIC drivers and allows the login timeout value to be correctly set.
Setting iSCSI Login Timeout Values
When using VMware vSphere 5 software iSCSI initiator to attach to a Dell EqualLogic PS Series group, the iSCSI option: Login Timeout value defaults to 5 seconds and cannot be modified.
Dell EqualLogic has observed that in large configurations and under certain workloads this 5 second timeout is not long enough to process the large number of iSCSI login requests that occur after a network failure, switch reboot, or controller failover.
VMware has released a patch that enables editing of the iSCSI option: Login Timeout value on the ESXi 5.0 host. Dell recommends that customers apply this patch and increase this value to 60 seconds. Doing so ensures that there is sufficient time for all iSCSI logins to occur, regardless of the size of the environment.
You can set this value in two ways:
• Using the Advanced Properties of the iSCSI Software Adapter in the vSphere Client GUI.
• Entering the following vSphere CLI command: esxcli iscsi adapter param set -A -k LoginTimeout -v 6
***REMINDER: Requires ESXi v5 patch: ESXi500-201112001 in order to allow the modifcation of this field. Otherwise it's greyed out in the ESX GUI ***
The minimum build number that includes this patch is: 514841
Checking the vmware patches, I see a new Build for ESXi 5 see below:
ESXi500-201112001
Download here: hostupdate.vmware.com/.../ESXi500-201112001.zip
Product: : ESXi (Embedded and Installable) 5.0.0
md5sum:107ec1cf6ee1d5d5cb8ea5c05b05cc10
sha1sum:aff63c8a170508c8c0f21a60d1ea75ef1922096d
Download Size: 297.7 MB
Build Number: 515841
See VMware KB 2007680 kb.vmware.com/.../2007680
Release Date: : 12/15/2011
System Impact : VM Shutdown & Host Reboot
gwomick
1 Message
0
January 4th, 2012 14:00
Is there a link to the Equallogic/Dell iSCSI recommendations in ESXI 5?
When using VMware vSphere 5 software iSCSI initiator to attach to a Dell EqualLogic PS Series group, the iSCSI option: Login Timeout value defaults to 5 seconds and cannot be modified.
Dell EqualLogic has observed that in large configurations and under certain workloads this 5 second timeout is not long enough to process the large number of iSCSI login requests that occur after a network failure, switch reboot, or controller failover.
VMware has released a patch that enables editing of the iSCSI option: Login Timeout value on the ESXi 5.0 host. Dell recommends that customers apply this patch and increase this value to 60 seconds. Doing so ensures that there is sufficient time for all iSCSI logins to occur, regardless of the size of the environment.
DELL-Kenny K
685 Posts
0
January 4th, 2012 14:00
For all of the Equallogic recommendations with ESX vSphere 5.0 can be found in the installation and user guide for the Multipath Extension module. If you go to the www.support.equallogic.com , you will need to log in with your equallogic credentials, and go to the downloads section and then VMWare Integration then look for Version 1.1 Early Production Access (EPA) under the Multipathing Extension Module for VMware - EPA. When you go into that you will see a section for documentation and in there will be the installation and user guide. Let me know if that has everything you are looking for.
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
1
January 4th, 2012 16:00
FYI: The documentation Kenny discusses does NOT have info on the text you've quoted above. That issue came to light after the documentation was created. However, once you have patched ESXi v5 to the current revision, 515841 or better. Then you can change the timeout value as the VMware KB describes. Configuration->Storage Adapters->SW iSCSI->Properties->Advanced->Login timeout. Need to reboot afterwards.