This post is more than 5 years old
2 Posts
0
161003
MD3200i / ESXi 4.1 Multipathing problem
I'm having a problem with a MD3200i and our new T710 virtual server - it works, but performance is absolutely abysmal. Here's the details....
Server is a T710, 4 onboard nics + a 4 port Intel Server NIC. Its running the latest update of ESXi 4.1 - right now it has 6 hosts on it, going to be 10 total (most are lightly loaded special purpose servers). Server is connected to the SAN by 4 cables direct from the 4 port Intel Server NIC to the 4 ports on the management controller
We have 4 drives (2TB SAS) in the SAN - not my first choice - but what the customer wanted. Array is set up as a RAID-10. We have 3 logical volumes - 2 TB, 1 TB, and 750 GB. Each is formatted as a VMFS datastore.
I followed the instructions in the Dell whitepaper for setting up the MD3200 with ESX 4.1 and it doesn't appear we have multpathing working correctly. The VCenter Client is only showing 6 paths usable - should be 12 total (3x volumes * 4 network cards).
I've gone through the docs a handful of times, can't find anything that strikes me as an obvious problem. Below is the output of some of the commands in the documentation - if anything else would be helpful let me know. I feel like i'm running in to a brick wall here, any advice would be great!
Thanks in advance....
~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:01:00.00 bnx2 Up 1000Mbps Full 00:25:64:fe:51:c1 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T
vmnic1 0000:01:00.01 bnx2 Down 0Mbps Half 00:25:64:fe:51:c3 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T
vmnic2 0000:02:00.00 bnx2 Down 0Mbps Half 00:25:64:fe:51:c5 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T
vmnic3 0000:02:00.01 bnx2 Down 0Mbps Half 00:25:64:fe:51:c7 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T
vmnic4 0000:83:00.00 igb Up 1000Mbps Full 00:1b:21:77:3b:08 9000 Intel Corporation 82576 Gigabit Network Connection
vmnic5 0000:83:00.01 igb Up 1000Mbps Full 00:1b:21:77:3b:09 9000 Intel Corporation 82576 Gigabit Network Connection
vmnic6 0000:84:00.00 igb Up 1000Mbps Full 00:1b:21:77:3b:0c 9000 Intel Corporation 82576 Gigabit Network Connection
vmnic7 0000:84:00.01 igb Up 1000Mbps Full 00:1b:21:77:3b:0d 9000 Intel Corporation 82576 Gigabit Network Connection
~ # esxcfg-swiscsi -q
Software iSCSI is enabled
~ # esxcfg-scsidevs -a
vmhba0 ata_piix link-n/a sata.vmhba0 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
vmhba1 mptsas link-n/a sas.5842b2b04cd6af00 (0:3:0.0) LSI Logic / Symbios Logic Dell SAS 6/iR Integrated
vmhba32 ata_piix link-n/a sata.vmhba32 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
vmhba33 iscsi_vmk online iqn.1998-01.com.vmware:spag-261d8287 iSCSI Software Adapter
~ # esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 Management Network IPv4 10.100.101.251 255.255.255.0 10.100.101.255 00:25:64:fe:51:c1 1500 65535 true STATIC
vmk1 iSCSI Network IPv4 10.100.102.20 255.255.255.0 10.100.102.255 00:50:56:74:38:cc 9000 65535 true STATIC
vmk2 iSCSI Network 2 IPv4 10.100.102.21 255.255.255.0 10.100.102.255 00:50:56:7b:ce:6a 9000 65535 true STATIC
vmk3 iSCSI Network 3 IPv4 10.100.102.22 255.255.255.0 10.100.102.255 00:50:56:75:19:70 9000 65535 true STATIC
vmk4 iSCSI Network 4 IPv4 10.100.102.23 255.255.255.0 10.100.102.255 00:50:56:78:e8:66 9000 65535 true STATIC
~ # esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 128 13 128 1500 vmnic0,vmnic1,vmnic2,vmnic3
PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 7 vmnic0,vmnic1,vmnic2,vmnic3
Management Network 0 1 vmnic0,vmnic3
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch2 128 9 128 9000 vmnic4,vmnic5,vmnic6,vmnic7
PortGroup Name VLAN ID Used Ports Uplinks
iSCSI Network 4 0 1 vmnic4
iSCSI Network 3 0 1 vmnic5
iSCSI Network 2 0 1 vmnic6
iSCSI Network 0 1 vmnic7
Dev Mgr
4 Operator
4 Operator
•
9.3K Posts
1
October 28th, 2010 11:00
This is a very nice opening post as it provides the info that's needed to begin to figure this out. One thing that's missing is the "esxcli swiscsi nic list -d vmhba 33". This output confirms if the vmkernels were assigned to the iSCSI initiator.
Your problem is that you are using a single subnet for the whole iSCSI setup.
Each NIC (that you want to use for iSCSI) needs it's own subnet. If you cannot get (from your network group) more than the 10.10.102.0-255 range, just subnet it with a 27 bit subnet mask. Gateways are irrelevant as I doubt your server is going to try to connect to a SAN in a completely different network, so you don't have to worry about needing to make/create additional gateways on the switch(es).
Here is an example setup based on your IP ranges and modifying them with a 27 bit (255.255.255.224) subnet mask (to prevent needing to get more IP ranges):
SAN iSCSI ports:
Controller 0:
iSCSI port 0: 10.100.102.1
iSCSI port 1: 10.100.102.33
iSCSI port 2: 10.100.102.65
iSCSI port 3: 10.100.102.97
Controller 1:
iSCSI port 0: 10.100.102.2
iSCSI port 1: 10.100.102.34
iSCSI port 2: 10.100.102.66
iSCSI port 3: 10.100.102.98
Server iSCSI info (in a context that's hopefully understandable):
- vmnic7 -> iSCSI Network -> vmk1 -> 10.100.102.3
- vmnic6 -> iSCSI Network 2 -> vmk2 -> 10.100.102.35
- vmnic5 -> iSCSI Network 3 -> vmk3 -> 10.100.102.67
- vmnic4 -> iSCSI Network 4 -> vmk4 -> 10.100.102.99
Does this info make sense?
Dev Mgr
4 Operator
4 Operator
•
9.3K Posts
0
October 29th, 2010 11:00
This isn't so much an MD3200i recommendation, but just a general storage recommendation for ESX/ESXi; limit your virtual disks (their sizes) to a size that can hold no more than 4 or 5 VMs. Optimal is a single virtual disk per VM even.
The reason for this is that when a VM wants to write to it's disk, the datastore where this VM is located gets locked (from being written to) to allow this one VM to write it's data. Therefor, if you have too many VMs on a single datastore, they'll end up having to wait for eachother to finish writing before they can write their own data.
This doesn't affect the general raid configuration concerns of course, as these recommendations are due to the VMFS filesystem limitations. So, you can definitely make a single raid set and carve it into multiple virtual disks if you think the drives and raid type can handle the total IO.
kwarnimo
2 Posts
0
October 29th, 2010 11:00
Yep, that was it, worked perfectly. As soon as I got everything set up on separate subnets, all the paths work fine. Odd thing is in the Dell documentation it shows it as being all in the same subnet - good thing to note for future reference.
Any other tips for performance tuning with ESXi 4.1 & a MD3200i?
mrokkam
154 Posts
0
October 29th, 2010 14:00
The MD3200/MD3200i Array tuning guide can help you tune your array for best performance
The current location of this guide is at:
http://www.dell.com/downloads/global/products/pvaul/en/powervault-md3200-array-tuning.pdf
-Mohan
jadus01
21 Posts
0
November 24th, 2010 11:00
This thread has a lot of excellent information, but I'm still left with a related question:
What if one is just direct-connecting two servers straight to the MD3220i over iSCSI? We're planning on cabling like this:
Each individual colour is a unique subnet. So if I take the above example, ports 0,1,2,3 on each MD3220i controller is in a unique subnet, but that means the Hyper-V host will only be able to access that subnet on a single controller.
Would it be recommended to us only 2 subnets in this configuration?
jlauro
10 Posts
0
November 27th, 2010 08:00
Actually, you can place a lot more vms on a datastore. I have some with 20+. I never spread volumes over multiple LUNs, so the 2TB limit per LUN keeps from having more.
IO from VMs does not require locks. It's IO that creates or removes stuff not from existing VMs that do... (Well, if you use sparse files then I/O from VMs could be an issue as they may need to grow, so my advice assumes storage is dedicated per VM).
Snapshots can cause changes, logs, creating and deleting vms, etc... So unless you add/remove VMs a lot, or have more than a couple of VMs with snapshots enabled, don't worry how many VMs you put in each.
Also, the locking is more an issue for the number of host servers, not the number of VMs. Vmware only locks out other hosts when changing the size of files. The VMs inside are already locked to the vm, and so don't slow down. It's just operations that add/remove/grow files outside the vm that are impacted (or inside if you use snapshots or don't preallocate the storage as that can cause growth).
If you do it right you can have a dozen VM hosts all with access to a single VMFS that has 30 vms on it on a single 2TB volume.
mrokkam
154 Posts
0
December 1st, 2010 08:00
It doesn't matter that each host can connect to only 1 controller on a single subnet. As long as you setup all your iSCSI sessions correctly, you will be able to access your data redundantly. It might be easiest to set up your iSCSI sessions if you use 4 separate subnets. However, if you find it easier, 2 subnets will also work.
One thing to remember if you are using newer Linux releases like RHEL 5.5 and SLES 11.1... by default, all IO on a subnet will go out of only 1 port even if you have multiple NICs on the same subnet. So, for those scenarios, using 4 subnets would be required.
-Mohan
mgibson85
2 Posts
0
January 17th, 2011 03:00
Hi,
Apologies for re-opening this call.
My setup is a Dell R610 connecting to an MD3200i.
I followed the same setup guide so assumed that both the iSCSI host ports and VMKernel should be on the same subnet.
However, I've followed your example and split them into their own subnets.
Here's my current setup:
RAID Controller Module 0
iSCSI host port 0 : 10.20.0.1/29
iSCSI host port 1 : 10.20.0.8/29
iSCSI host port 2 : 10.20.0.15/29
iSCSI host port 3 : 10.20.0.22/29
RAID Controller Module 1
iSCSI host port 0 : 10.20.0.2/29
iSCSI host port 1 : 10.20.0.9/29
iSCSI host port 2 : 10.20.0.16/29
iSCSI host port 3 : 10.20.0.23/29
vmk0 : 10.20.0.3
vmk1 : 10.20.0.10
vmk2 : 10.20.0.17
vmk3 : 10.20.0.24
# esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:01:00.00 bnx2 Up 1000Mbps Full f0:4d:a2:03:10:56 1500 Broadcom Corporation PowerEdge R610 BCM5709 Gigabit Ethernet
vmnic1 0000:01:00.01 bnx2 Up 1000Mbps Full f0:4d:a2:03:10:58 1500 Broadcom Corporation PowerEdge R610 BCM5709 Gigabit Ethernet
vmnic2 0000:02:00.00 bnx2 Up 1000Mbps Full f0:4d:a2:03:10:5a 1500 Broadcom Corporation PowerEdge R610 BCM5709 Gigabit Ethernet
vmnic3 0000:02:00.01 bnx2 Up 1000Mbps Full f0:4d:a2:03:10:5c 1500 Broadcom Corporation PowerEdge R610 BCM5709 Gigabit Ethernet
vmnic4 0000:06:00.00 igb Up 1000Mbps Full 00:1b:21:86:31:60 1500 Intel Corporation 82576 Gigabit Network Connection
vmnic5 0000:06:00.01 igb Up 1000Mbps Full 00:1b:21:86:31:61 1500 Intel Corporation 82576 Gigabit Network Connection
vmnic6 0000:07:00.00 igb Up 1000Mbps Full 00:1b:21:86:31:64 1500 Intel Corporation 82576 Gigabit Network Connection
vmnic7 0000:07:00.01 igb Up 1000Mbps Full 00:1b:21:86:31:65 1500 Intel Corporation 82576 Gigabit Network Connection
vmnic8 0000:0a:00.00 igb Up 1000Mbps Full 00:1b:21:86:37:a0 1500 Intel Corporation 82576 Gigabit Network Connection
vmnic9 0000:0a:00.01 igb Up 1000Mbps Full 00:1b:21:86:37:a1 1500 Intel Corporation 82576 Gigabit Network Connection
vmnic10 0000:0b:00.00 igb Up 1000Mbps Full 00:1b:21:86:37:a4 1500 Intel Corporation 82576 Gigabit Network Connection
vmnic11 0000:0b:00.01 igb Up 1000Mbps Full 00:1b:21:86:37:a5 1500 Intel Corporation 82576 Gigabit Network Connection
# esxcfg-swiscsi -q
Software iSCSI is enabled
# esxcfg-scsidevs -a
vmhba0 megaraid_sas link-n/a unknown.vmhba0 (0:3:0.0) LSI Logic / Symbios Logic Dell PERC H700 Integrated
vmhba1 ata_piix link-n/a sata.vmhba1 (0:0:31.2) Intel Corporation PowerEdge R610 SATA IDE Controller
vmhba34 bnx2i unbound iscsi.vmhba34 Broadcom iSCSI Adapter
vmhba35 bnx2i unbound iscsi.vmhba35 Broadcom iSCSI Adapter
vmhba36 bnx2i unbound iscsi.vmhba36 Broadcom iSCSI Adapter
vmhba37 bnx2i unbound iscsi.vmhba37 Broadcom iSCSI Adapter
vmhba38 ata_piix link-n/a sata.vmhba38 (0:0:31.2) Intel Corporation PowerEdge R610 SATA IDE Controller
vmhba39 iscsi_vmk online iscsi.vmhba39 iSCSI Software Adapter
# esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 iSCSI1 IPv4 10.20.0.3 255.255.255.248 10.20.0.7 00:50:56:70:88:53 1500 65535 true STATIC
vmk1 iSCSI2 IPv4 10.20.0.10 255.255.255.248 10.20.0.15 00:50:56:7a:b6:c7 1500 65535 true STATIC
vmk2 iSCSI3 IPv4 10.20.0.17 255.255.255.248 10.20.0.23 00:50:56:72:14:2f 1500 65535 true STATIC
vmk3 iSCSI4 IPv4 10.20.0.24 255.255.255.248 10.20.0.31 00:50:56:75:46:15 1500 65535 true STATIC
# esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 128 3 128 1500 vmnic0
PortGroup Name VLAN ID Used Ports Uplinks
Management 0 0 vmnic0
Service Console 0 1 vmnic0
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 3 128 1500 vmnic1,vmnic2
PortGroup Name VLAN ID Used Ports Uplinks
LAN 0 0 vmnic1,vmnic2
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch2 128 2 128 1500 vmnic3
PortGroup Name VLAN ID Used Ports Uplinks
DMZ 0 0 vmnic3
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch3 128 9 128 1500 vmnic4,vmnic5,vmnic6,vmnic7
PortGroup Name VLAN ID Used Ports Uplinks
iSCSI4 0 1 vmnic7
iSCSI3 0 1 vmnic6
iSCSI2 0 1 vmnic5
iSCSI1 0 1 vmnic4
# esxcli swiscsi nic list -d vmhba39
vmk0
pNic name: vmnic4
ipv4 address: 10.20.0.3
ipv4 net mask: 255.255.255.248
ipv6 addresses:
mac address: 00:1b:21:86:31:60
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
vlanId: 0
ports reserved: 63488~65536
link connected: true
ethernet speed: 1000
packets received: 137117
packets sent: 1209
NIC driver: igb
driver version: 1.3.19.12.2-1vmw
firmware version: 1.5-1
vmk1
pNic name: vmnic5
ipv4 address: 10.20.0.10
ipv4 net mask: 255.255.255.248
ipv6 addresses:
mac address: 00:1b:21:86:31:61
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
vlanId: 0
ports reserved: 63488~65536
link connected: true
ethernet speed: 1000
packets received: 137002
packets sent: 580
NIC driver: igb
driver version: 1.3.19.12.2-1vmw
firmware version: 1.5-1
vmk2
pNic name: vmnic6
ipv4 address: 10.20.0.17
ipv4 net mask: 255.255.255.248
ipv6 addresses:
mac address: 00:1b:21:86:31:64
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
vlanId: 0
ports reserved: 63488~65536
link connected: true
ethernet speed: 1000
packets received: 137107
packets sent: 459
NIC driver: igb
driver version: 1.3.19.12.2-1vmw
firmware version: 1.5-1
vmk3
pNic name: vmnic7
ipv4 address: 10.20.0.24
ipv4 net mask: 255.255.255.248
ipv6 addresses:
mac address: 00:1b:21:86:31:65
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
vlanId: 0
ports reserved: 63488~65536
link connected: true
ethernet speed: 1000
packets received: 137163
packets sent: 264
NIC driver: igb
driver version: 1.3.19.12.2-1vmw
firmware version: 1.5-1
I can vmkping 10.20.0.1 from one of the vmware servers I'm setting up.
When I add 10.20.0.1 into the dynamic discovery in vmware, it doesn't find any disks.
I can't see anything obvious and going round in circles.
Thanks in advance, I appreciate any help given.
Kind Regards,
Mark
mgibson85
2 Posts
0
January 17th, 2011 06:00
Hi,
Thanks for the quick reply. So my vmware setup looks ok?
On the MDSM, as a test I setup 2x 500gbs in RAID0.
I've assigned these to the host group rather than the server as we will be adding an additional vmware server in.
Here's the host setup - http://i52.tinypic.com/2j1tms5.jpg
Here's the host port identifier - http://i55.tinypic.com/2cnjgk6.jpg
iqn.1998-01.com.vmware:bm-vm-06-4aebf8f7 is the name of the iSCSI initiator on my vmware server. Is this correct?
Kind Regards,
Mark
Dev Mgr
4 Operator
4 Operator
•
9.3K Posts
0
January 17th, 2011 06:00
After you do the dynamic discovery, you do still need to register the server in the MDSM application, then create the diskgroup(s) and virtual disk(s), and assign them to the server.
NOTE: Keep in mind that virtual disks cannot be larger than "2TB minus 512 bytes" (see VMware's configuration maximums guide (direct link to the pdf, so I'm not sure how long this link will be good once a newer version than 4.1 comes out)).
dlreid
3 Posts
0
January 20th, 2011 12:00
Hi kwarnimo,
You mention a Dell white paper for setting up the Md3200 with ESXi 4.1 I have looked for it but can't find it. Could you put the link to it here so I could get a copy?
Thanks.
mrokkam
154 Posts
0
January 21st, 2011 13:00
www.dell.com/md3200i <--look on the right hand side
direct link: http://www.dell.com/downloads/global/products/pvaul/en/powervault-md32x0i-vmware-solution.pdf
Hope this helps,
-Mohan
Solema
5 Posts
0
January 24th, 2011 09:00
Does the separate subnet requirement apply also to the SAN controller ports themselves, or just to the hosts? What about for Windows 2008 R2 hosts? I've spent over a week with Dell support trying to fix our awful SAN read speeds (6MB/s!) and still no solution.
I have two R710's with two iSCSI ports each, connected to an MD3200i with two 4-port controllers, all connected to two 5424 switches. My config is as follows (Subnet Masks are all 255.255.255.0):
Switch 1:
All 192.168.130.x iSCSI traffic
Switch 2:
All 192.168.131.x iSCSI traffic
SAN
Controller 0:
Port 0: 192.168.130.101 (Switch 1)
Port 1: 192.168.131.101 (Switch 2)
Port 2: 192.168.130.103 (Switch 1)
Port 3: 192.168.131.103 (Switch 2)
Controller 1:
Port 0: 192.168.130.102 (Switch 1)
Port 1: 192.168.131.102 (Switch 2)
Port 2: 192.168.130.104 (Switch 1)
Port 3: 192.168.131.104 (Switch 2)
SERVERS
Server1:
Port 0: 192.168.130.51 (Switch 1)
Port 1: 192.168.131.51 (Switch 2)
Server2:
Port 0: 192.168.130.52 (Switch 1)
Port 1: 192.168.131.52 (Switch 2)
Now what I have found out is that if I DISCONNECT ports 2&3 on each controller, the SAN read speed jumps from 6MB/s to 230MB/s. Can anyone shed some light on why this is and what I can do to permanently fix the issue? Does it have anything to do with the fact that ports 2&3 on each controller share the same subnet as 0&1, respectively?
Thanks,
Nick
Dev Mgr
4 Operator
4 Operator
•
9.3K Posts
0
January 24th, 2011 13:00
You don't want to do this type of setup.
Change it to:
Switch 1:
All 192.168.130.x and 192.168.132.x iSCSI traffic
Switch 2:
All 192.168.131.x and 192.168.133.x iSCSI traffic
SAN
Controller 0:
Port 0: 192.168.130.101 (Switch 1)
Port 1: 192.168.131.101 (Switch 2)
Port 2: 192.168.132.101 (Switch 1)
Port 3: 192.168.133.101 (Switch 2)
Controller 1:
Port 0: 192.168.130.102 (Switch 1)
Port 1: 192.168.131.102 (Switch 2)
Port 2: 192.168.132.102 (Switch 1)
Port 3: 192.168.133.102 (Switch 2)
SERVERS
Server1:
Port 0: 192.168.130.51 (Switch 1)
Port 1: 192.168.131.51 (Switch 2)
Server2:
Port 0: 192.168.132.52 (Switch 1)
Port 1: 192.168.133.52 (Switch 2)
This way each server connects with each NIC only to one port on each controller (e.g. server 1 port 0 connects to 192.168.130.101 and 192.168.130.102, but not 192.168.130.103 and 192.168.130.104).
This also gives each server full redundancy and it's own iSCSI ports on the SAN to communicate with.
tcastantine
2 Posts
0
February 16th, 2013 17:00
Curiously, how does this look if you are adding a 3rd host to the mix? You are kind of losing that 1-to-1 connection exclusivity.