MD3200i / ESXi 4.1 Multipathing problem

This isn't so much an MD3200i recommendation, but just a general storage recommendation for ESX/ESXi; limit your virtual disks (their sizes) to a size that can hold no more than 4 or 5 VMs. Optimal is a single virtual disk per VM even.

The reason for this is that when a VM wants to write to it's disk, the datastore where this VM is located gets locked (from being written to) to allow this one VM to write it's data. Therefor, if you have too many VMs on a single datastore, they'll end up having to wait for eachother to finish writing before they can write their own data.

This doesn't affect the general raid configuration concerns of course, as these recommendations are due to the VMFS filesystem limitations. So, you can definitely make a single raid set and carve it into multiple virtual disks if you think the drives and raid type can handle the total IO.

Yep, that was it, worked perfectly.  As soon as I got everything set up on separate subnets, all the paths work fine.  Odd thing is in the Dell documentation it shows it as being all in the same subnet - good thing to note for future reference. 

Any other tips for performance tuning with ESXi 4.1 & a MD3200i?

The MD3200/MD3200i Array tuning guide can help you tune your array for best performance

The current location of this guide is at:

http://www.dell.com/downloads/global/products/pvaul/en/powervault-md3200-array-tuning.pdf

-Mohan

This thread has a lot of excellent information, but I'm still left with a related question:

What if one is just direct-connecting two servers straight to the MD3220i over iSCSI? We're planning on cabling like this:

Each individual colour is a unique subnet. So if I take the above example, ports 0,1,2,3 on each MD3220i controller is in a unique subnet, but that means the Hyper-V host will only be able to access that subnet on a single controller.

Would it be recommended to us only 2 subnets in this configuration?

Actually, you can place a lot more vms on a datastore.  I have some with 20+.  I never spread volumes over multiple LUNs, so the 2TB limit per LUN keeps from having more.

IO from VMs does not require locks.  It's IO that creates or removes stuff not from existing VMs that do...  (Well, if you use sparse files then I/O from VMs could be an issue as they may need to grow, so my advice assumes storage is dedicated per VM).

Snapshots can cause changes, logs, creating and deleting vms, etc...  So unless you add/remove VMs a lot, or have more than a couple of VMs with snapshots enabled, don't worry how many VMs you put in each.

Also, the locking is more an issue for the number of host servers, not the number of VMs.  Vmware only locks out other hosts when changing the size of files.  The VMs inside are already locked to the vm, and so don't slow down.  It's just operations that add/remove/grow files outside the vm that are impacted (or inside if you use snapshots or don't preallocate the storage as that can cause growth).

If you do it right you can have a dozen VM hosts all with access to a single VMFS that has 30 vms on it on a single 2TB volume.

It doesn't matter that each host can connect to only 1 controller on a single subnet. As long as you setup all your iSCSI sessions correctly, you will be able to access your data redundantly. It might be easiest to set up your iSCSI sessions if you use 4 separate subnets. However, if you find it easier, 2 subnets will also work.

One thing to remember if you are using newer Linux releases like RHEL 5.5 and SLES 11.1... by default, all IO on a subnet will go out of only 1 port even if you have multiple NICs on the same subnet. So, for those scenarios, using 4 subnets would be required.

 -Mohan

Hi,

Apologies for re-opening this call.
My setup is a Dell R610 connecting to an MD3200i.

I followed the same setup guide so assumed that both the iSCSI host ports and VMKernel should be on the same subnet.
However, I've followed your example and split them into their own subnets.

Here's my current setup:

RAID Controller Module 0

iSCSI host port 0 : 10.20.0.1/29
iSCSI host port 1 : 10.20.0.8/29
iSCSI host port 2 : 10.20.0.15/29
iSCSI host port 3 : 10.20.0.22/29

RAID Controller Module 1

iSCSI host port 0 : 10.20.0.2/29
iSCSI host port 1 : 10.20.0.9/29
iSCSI host port 2 : 10.20.0.16/29
iSCSI host port 3 : 10.20.0.23/29

vmk0 : 10.20.0.3
vmk1 : 10.20.0.10
vmk2 : 10.20.0.17
vmk3 : 10.20.0.24

# esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:01:00.00 bnx2        Up   1000Mbps  Full   f0:4d:a2:03:10:56 1500   Broadcom Corporation PowerEdge R610 BCM5709 Gigabit Ethernet
vmnic1  0000:01:00.01 bnx2        Up   1000Mbps  Full   f0:4d:a2:03:10:58 1500   Broadcom Corporation PowerEdge R610 BCM5709 Gigabit Ethernet
vmnic2  0000:02:00.00 bnx2        Up   1000Mbps  Full   f0:4d:a2:03:10:5a 1500   Broadcom Corporation PowerEdge R610 BCM5709 Gigabit Ethernet
vmnic3  0000:02:00.01 bnx2        Up   1000Mbps  Full   f0:4d:a2:03:10:5c 1500   Broadcom Corporation PowerEdge R610 BCM5709 Gigabit Ethernet
vmnic4  0000:06:00.00 igb         Up   1000Mbps  Full   00:1b:21:86:31:60 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic5  0000:06:00.01 igb         Up   1000Mbps  Full   00:1b:21:86:31:61 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic6  0000:07:00.00 igb         Up   1000Mbps  Full   00:1b:21:86:31:64 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic7  0000:07:00.01 igb         Up   1000Mbps  Full   00:1b:21:86:31:65 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic8  0000:0a:00.00 igb         Up   1000Mbps  Full   00:1b:21:86:37:a0 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic9  0000:0a:00.01 igb         Up   1000Mbps  Full   00:1b:21:86:37:a1 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic10 0000:0b:00.00 igb         Up   1000Mbps  Full   00:1b:21:86:37:a4 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic11 0000:0b:00.01 igb         Up   1000Mbps  Full   00:1b:21:86:37:a5 1500   Intel Corporation 82576 Gigabit Network Connection

# esxcfg-swiscsi -q
Software iSCSI is enabled

# esxcfg-scsidevs -a
vmhba0  megaraid_sas      link-n/a  unknown.vmhba0                          (0:3:0.0) LSI Logic / Symbios Logic Dell PERC H700 Integrated
vmhba1  ata_piix                   link-n/a  sata.vmhba1                                   (0:0:31.2) Intel Corporation PowerEdge R610 SATA IDE Controller
vmhba34 bnx2i                      unbound   iscsi.vmhba34                            Broadcom iSCSI Adapter
vmhba35 bnx2i                      unbound   iscsi.vmhba35                           Broadcom iSCSI Adapter
vmhba36 bnx2i                      unbound   iscsi.vmhba36                           Broadcom iSCSI Adapter
vmhba37 bnx2i                      unbound   iscsi.vmhba37                           Broadcom iSCSI Adapter
vmhba38 ata_piix                  link-n/a  sata.vmhba38                            (0:0:31.2) Intel Corporation PowerEdge R610 SATA IDE Controller
vmhba39 iscsi_vmk             online    iscsi.vmhba39                           iSCSI Software Adapter

# esxcfg-vmknic -l
Interface  Port Group/DVPort   IP Family IP Address                         Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type
vmk0       iSCSI1              IPv4      10.20.0.3                                     255.255.255.248 10.20.0.7       00:50:56:70:88:53 1500    65535     true    STATIC
vmk1       iSCSI2              IPv4      10.20.0.10                                  255.255.255.248 10.20.0.15      00:50:56:7a:b6:c7 1500    65535     true    STATIC
vmk2       iSCSI3              IPv4      10.20.0.17                                  255.255.255.248 10.20.0.23      00:50:56:72:14:2f 1500    65535     true    STATIC
vmk3       iSCSI4              IPv4      10.20.0.24                                  255.255.255.248 10.20.0.31      00:50:56:75:46:15 1500    65535     true    STATIC

# esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         128         3           128               1500    vmnic0

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  Management            0        0           vmnic0
  Service Console       0        1           vmnic0

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch1         128         3           128               1500    vmnic1,vmnic2

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  LAN                   0        0           vmnic1,vmnic2

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch2         128         2           128               1500    vmnic3

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  DMZ                   0        0           vmnic3

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch3         128         9           128               1500    vmnic4,vmnic5,vmnic6,vmnic7

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  iSCSI4                0        1           vmnic7
  iSCSI3                0        1           vmnic6
  iSCSI2                0        1           vmnic5
  iSCSI1                0        1           vmnic4

# esxcli swiscsi nic list -d vmhba39
vmk0
    pNic name: vmnic4
    ipv4 address: 10.20.0.3
    ipv4 net mask: 255.255.255.248
    ipv6 addresses:
    mac address: 00:1b:21:86:31:60
    mtu: 1500
    toe: false
    tso: true
    tcp checksum: false
    vlan: true
    vlanId: 0
    ports reserved: 63488~65536
    link connected: true
    ethernet speed: 1000
    packets received: 137117
    packets sent: 1209
    NIC driver: igb
    driver version: 1.3.19.12.2-1vmw
    firmware version: 1.5-1

vmk1
    pNic name: vmnic5
    ipv4 address: 10.20.0.10
    ipv4 net mask: 255.255.255.248
    ipv6 addresses:
    mac address: 00:1b:21:86:31:61
    mtu: 1500
    toe: false
    tso: true
    tcp checksum: false
    vlan: true
    vlanId: 0
    ports reserved: 63488~65536
    link connected: true
    ethernet speed: 1000
    packets received: 137002
    packets sent: 580
    NIC driver: igb
    driver version: 1.3.19.12.2-1vmw
    firmware version: 1.5-1

vmk2
    pNic name: vmnic6
    ipv4 address: 10.20.0.17
    ipv4 net mask: 255.255.255.248
    ipv6 addresses:
    mac address: 00:1b:21:86:31:64
    mtu: 1500
    toe: false
    tso: true
    tcp checksum: false
    vlan: true
    vlanId: 0
    ports reserved: 63488~65536
    link connected: true
    ethernet speed: 1000
    packets received: 137107
    packets sent: 459
    NIC driver: igb
    driver version: 1.3.19.12.2-1vmw
    firmware version: 1.5-1

vmk3
    pNic name: vmnic7
    ipv4 address: 10.20.0.24
    ipv4 net mask: 255.255.255.248
    ipv6 addresses:
    mac address: 00:1b:21:86:31:65
    mtu: 1500
    toe: false
    tso: true
    tcp checksum: false
    vlan: true
    vlanId: 0
    ports reserved: 63488~65536
    link connected: true
    ethernet speed: 1000
    packets received: 137163
    packets sent: 264
    NIC driver: igb
    driver version: 1.3.19.12.2-1vmw
    firmware version: 1.5-1

I can vmkping 10.20.0.1 from one of the vmware servers I'm setting up.

When I add 10.20.0.1 into the dynamic discovery in vmware, it doesn't find any disks.
I can't see anything obvious and going round in circles.

Thanks in advance, I appreciate any help given.

Kind Regards,

Mark

Hi,

Thanks for the quick reply. So my vmware setup looks ok?

On the MDSM, as a test I setup 2x 500gbs in RAID0.
I've assigned these to the host group rather than the server as we will be adding an additional vmware server in.

Here's the host setup - http://i52.tinypic.com/2j1tms5.jpg

Here's the host port identifier - http://i55.tinypic.com/2cnjgk6.jpg

iqn.1998-01.com.vmware:bm-vm-06-4aebf8f7 is the name of the iSCSI initiator on my vmware server. Is this correct?

Kind Regards,

Mark

After you do the dynamic discovery, you do still need to register the server in the MDSM application, then create the diskgroup(s) and virtual disk(s), and assign them to the server.

NOTE: Keep in mind that virtual disks cannot be larger than "2TB minus 512 bytes" (see VMware's configuration maximums guide (direct link to the pdf, so I'm not sure how long this link will be good once a newer version than 4.1 comes out)).

Hi kwarnimo,

You mention a Dell white paper for setting up the Md3200 with ESXi 4.1    I have looked for it but can't find it.  Could you put the link to it here so I could get a copy?

Thanks.

www.dell.com/md3200i <--look on the right hand side

direct link: http://www.dell.com/downloads/global/products/pvaul/en/powervault-md32x0i-vmware-solution.pdf

Hope this helps,

-Mohan

Does the separate subnet requirement apply also to the SAN controller ports themselves, or just to the hosts?  What about for Windows 2008 R2 hosts?  I've spent over a week with Dell support trying to fix our awful SAN read speeds (6MB/s!) and still no solution.

I have two R710's with two iSCSI ports each, connected to an MD3200i with two 4-port controllers, all connected to two 5424 switches.  My config is as follows (Subnet Masks are all 255.255.255.0):

Switch 1:
All 192.168.130.x iSCSI traffic

Switch 2:
All 192.168.131.x iSCSI traffic

SAN
Controller 0:
Port 0: 192.168.130.101 (Switch 1)
Port 1: 192.168.131.101 (Switch 2)
Port 2: 192.168.130.103 (Switch 1)
Port 3: 192.168.131.103 (Switch 2)

Controller 1:
Port 0: 192.168.130.102 (Switch 1)
Port 1: 192.168.131.102 (Switch 2)
Port 2: 192.168.130.104 (Switch 1)
Port 3: 192.168.131.104 (Switch 2)

SERVERS
Server1:
Port 0: 192.168.130.51 (Switch 1)
Port 1: 192.168.131.51 (Switch 2)

Server2:
Port 0: 192.168.130.52 (Switch 1)
Port 1: 192.168.131.52 (Switch 2)

Now what I have found out is that if I DISCONNECT ports 2&3 on each controller, the SAN read speed jumps from 6MB/s to 230MB/s.  Can anyone shed some light on why this is and what I can do to permanently fix the issue?  Does it have anything to do with the fact that ports 2&3 on each controller share the same subnet as 0&1, respectively?

Thanks,
Nick

You don't want to do this type of setup.

Change it to:

Switch 1:
All 192.168.130.x and 192.168.132.x iSCSI traffic

Switch 2:
All 192.168.131.x and 192.168.133.x iSCSI traffic

SAN
Controller 0:
Port 0: 192.168.130.101 (Switch 1)
Port 1: 192.168.131.101 (Switch 2)
Port 2: 192.168.132.101 (Switch 1)
Port 3: 192.168.133.101 (Switch 2)

Controller 1:
Port 0: 192.168.130.102 (Switch 1)
Port 1: 192.168.131.102 (Switch 2)
Port 2: 192.168.132.102 (Switch 1)
Port 3: 192.168.133.102 (Switch 2)

SERVERS
Server1:
Port 0: 192.168.130.51 (Switch 1)
Port 1: 192.168.131.51 (Switch 2)

Server2:
Port 0: 192.168.132.52 (Switch 1)
Port 1: 192.168.133.52 (Switch 2)

This way each server connects with each NIC only to one port on each controller (e.g. server 1 port 0 connects to 192.168.130.101 and 192.168.130.102, but not 192.168.130.103 and 192.168.130.104).

This also gives each server full redundancy and it's own iSCSI ports on the SAN to communicate with.

Curiously, how does this look if you are adding a 3rd host to the mix?  You are kind of losing that 1-to-1 connection exclusivity.