MD3xxxi - disk groups, luns and VMware - tips to separate out LUN path

Background -

We had a bit of drama with our VMware 4.1 deployment on MD3220i and MD3000i arrays using iSCSI because although I had configured many LUNs there were basically only two disk groups in each array.

The key problem was that the majority of our physical servers (and all guests on those servers) were using the same path regardless of VMware pathing option, Round Robin, MRU, Fixed, Alternate Fixed A/P.

Dell & VMware separately logged in and validated the configuration. The VMware guys were not that familiar with the Md3220i and the Dell guys were not that familiar with VMware. It took 4 days to figure out the root cause.

It was only after several multi-hour con-calls with deep techs at VMware and Powervault storage that VMware figured out that pathing is at the disk group level rather than the LUN level for the MD3xxxi. In other words number of LUNs is irrelevant to the pathing, the MD instructs pathing at the disk group level. VMware is simply doing the bidding of the MD. MPIO on VMware was at the mercy of the paths the MD controller was providing. We had to create more disk groups to maximize use of the 8 iSCSI ports on the array and remove the bottleneck. Basically we broke our wide disk groups with many LUNS down into narrow disk groups each with a single LUN, a 1-to-1 relationship of disk group to LUN. Problem solved.

After a hasty maint window and all-weekend reconfig we have resolved the problem but now I have a seriously sub-optimal array config - there are too many disk groups with too few spindles. The most optimal solution would be what I had originally, a single disk group assigned to each controller with multiple layered LUNs within the disk group.

We were initially thrilled to deploy the MD3220i arrays in production (the MD3000i is online backup and Veeam repository). Now we regret using them in our vitualization plan. Would like to fix that.

Unique to the 3220i? Our multi-pathing seems to be working fine to our dual controller MD3000i's by lun. Are your VD's slow using the limited spindle counts?

We use the GUI to set up our arrays. ESX 4.1 with a recent conversion. The two paths say active/active, two other paths state standby. We use the round robin. I mean I can see the traffic going over both nic paths, one subnet to each iSCSI port on the controllers. You only get two with the MD3000i as each controller owns the particular luns.

Initially we were seeing issues sort of like yours when we tried to use the same subnet for all 4 controller ports, we went to two different subnets and then things seem to work properly.

Each lun is transmitting data over the two ports of each the controller. It sounds like the addition of the other ports in the 3220i has mucked up the MPIO some for sure.

Hi JohnADCO

Yes, with the original we had single disk group across 14 disk and another across 10 disk, then sliced into LUNs we had good disk performance but crushed the controller.

To fully maximize all four ports in each controller we had to split into smaller 3-spindle disk groups, too few spindles for concurrency. It is simple math, 24 disks in the array, 8 ports to distribute the disks. Move from RAID-10 to RAID-5 and make small disk groups.

I cannot tell in the MD3000i because we have four disk groups (big, slow, deep drives) and four ports. It did not occur to me the MD3000i would be better at LUN management than the MD32.

How did you setup the array - SMCLI or GUI?
What version of vCenter ESX and what sort of pathing options have you chosen?


And to be sure - VMware will indicate a path, but it will not use the path because the controller says not optimal. So just checking you are actually seeing concurrent active paths to all four ports in your MD3000i. If so then that's a good start to diferentiating a solution.

Answered -
On our fifth con call with Dell we got our answer and the solution to the problem.

You must setup a different subnet for every port on the controller or you will not get MPIO. If you do setup each port on a different subnet then the original configuration is valid, one disk group per controller, each disk group with 4 LUNS (one LUN per port). This is the best way to maximize spindles per LUN in such a small array. We tested and in fact get packets across all four active ports for a single large I/O task.

Ditto for the MD3000 as well, only it will be one disk group with two LUNS per controller due to the more limited controllers.

If Dell had been able to diagnose this on the first or second teck call we would not have reconfigured our arrays. At least as it stands now we know the architecture is solid and the configuration change will work.

Thank you for your input.

One last tidbit -
Although VMware recommends MRU for the path selection method to get MPIO the Dell tech recommended Round Robin. With the aforementioned change in subnets he was correct - the Round Robin produced active I/O paths on all ports of the array.

We just got our MD3xxxi boxes in and just about ready to configure the LUNs. With our old MD3000i box we used a flat (1 subnet) network with Round Robin and that seemed to work fine. Do we need to reconsider now if have the MD32xxxi box with 4 iscsi links per controller? Or should we continue using 1 subnet? I hate to add complexity to our storage network, like to keep it simple, and easier for troubleshooting....

Also, with your configuration you had to split up Drive groups to 1-to-1 LUNs. Is this still the case with the RR and vLANs?

I'm thinking about making a 22-disk RAID 10 of 15k drives, then carving it into small LUNs for VMware. Is this not recommended? We bought the High Performance Key thing too, whatever that does???? Thanks for posting this useful info!

Lance

Lance -

How many servers, how many pNics and the I/O access characteristics will affect how you configure the MD3220.
In our case 98% of I/O are reads in a mix of web/app/Oracle db servers. We have multiple ESXi servers.

Our lesson learned is that you will only have one active port on each controller of the array if you do not create additional subnets. Regardless of the pathing method in VMware. That will cause a bottle-neck at the controller.

The 1:1 disk-group-to-lun ratio was because we only had one subnet, that was a temp fix until we could get the other three subnets configured in our switches and ESXi servers. It is the only way to get pseudo MPIO without subnets. I cannot recommend because it creates concurrency issues with such small disk groups on the md3xxxi.

The solution is multiple subnets, plain and simple.

Our performance, since reconfiguring for multiple subnets has jumped dramatically, the latency has disappeared and the throughput is an order of magnitude higher.

If you have two controllers in your MD3220 you should not create a single disk group on the array. Dell MD3xxxi are Active/Active controllers, HOWEVER a disk group is "owned" by a single controller, that means you must have at least two disk groups in order to spread I/O across both controllers. Furthermore each disk group should have at least 4 LUNs to spread I/O within a controller across the four subnets/ports on the controller.

The two configs we've tried:
a) 1 RAID5 11-disk group & 1 RAID10 12-disk group (single spare globally assigned)
b) 2 RAID5 11-disk groups (2 spares globally assigned)

We ultimately settled on b) since our profile is 98% reads.
Note: in both cases we had at least 4 LUNs per disk group.

We also reconfigured the MD3000i in a similar fashion, multiple subnets (two) one for each port and three luns per disk group. We saw a little improvement over previous config but not as much as we've seen on the MD3220. The controllers on the MD3000i just aren't that capable.

Our topology is:
Four subnets, four vLANs
All of our Cisco switch iSCSI ports have port trunking enabled, all four vLANs allowed (makes cabling easier)
Each port on the MD3220i controllers have one of four vLANs assigned
Each port on the MD3000i controllers have one of two vLANs assigned
VMware single iSCSI switch on each ESXi server consolidating all ports
Each VMkernel port is set to a separate vLAN
Each iSCSI pNIC (port) is assigned a vLAN
There is no failover assigned in the vmkernel, the failover is in the multi-pathing across vLANs.

Rick

BTW - it is stated in this forum and we have learned first hand that Round Robin on a MD3000i will cause the array to go in and out of optimal mode as the pathing switches from preferred owner to other controller (assumes you have two controllers).

Once we separated out all of the ports onto different vLANs all of our arrays went Optimal and have remained ever since. Even though all of our devices are set to Round Robin.

Hi Rick -

We have just taken delivery of a MD3220i (dual controllers) and 2 x R710's to create a new VMware 4.1 environment. The iSCSI part is new technology to the team here so the information you've posted is invaluable.

We were going to follow the recommendations from Dell that they published in a deployment guide specifically for the MD3220i and VMware 4.1. Can you confirm that the information in it is flawed as they definitely only use a single VLAN/subnet for iSCSI communication?

http://www.dell.com/downloads/global/products/pvaul/en/powervault-md32xxi-deployment-guide-for-vmware-esx4.1.pdf

Many thanks,
Howard

Hi Howard -

Not sure about flawed, perhaps "incomplete".

What I can confirm is this - we deployed our cluster of R610's with ESXi 4.1 embed and MD3220i and Md3000i and I followed the Dell Deployment Guide for VMware 4.1 top to bottom. (I mean, you can't get much more on point with a title like that, can you?)

We tested the individual apps and servers and I can tell you that we were doing great until production at which time with all servers firing the concurrency to the controllers caused our disk latency to jump to 300ms continuous with spikes as high as 1500ms and throughput slogged down to 15MB/sec.

I can confirm that we found very few people who could offer insights on the Dell ESXi embed server with iSCSI and MD3220i. Dell knew the array, VMware knew the hypervisor, everyone had opinions but none could pinpoint the root cause.

I can confirm that it was a Dell storage tech that said, offhand like it was patently obvious, "well you have to have a different vlan for each port for true MPIO" and then the light clicked on for us. Actually it was more like a group groan than a light - because we knew we'd have to reconfigure both arrays while in production.

After our reconfig we no longer have latency issues (avgs around 1.4 - 5 ms) and throughput reaches 425MB/sec when pressed by Veeam or a migration.
We are using the iSCSI offload engines (ISOE) and hardware based initiators on our Broadcom pNICs on the Dells and so far love that as well.

I have read over this guide too, and it does not recommend vLans. It does however, point out having multiple VMKernel's assigned on a host to the iSCSI Network. I've always used one VMKernel for iSCSI traffic and two pNics uplinked (one in active mode and the other in Passive). What does multiple VMKernel's on a host do?

BTW, thanks again @VirtualWatts for providing detailed information about your environment. Helping out a lot of folks!

Lance

Lance,

Per Dell recommendation we have always had multiple vmk ports, each bound 1:1 with a pNIC and all are active. They are all in the same iSCSI vswitch as we couldn't find any supporting evidence that additional switches would help. It is the only way I know to have multiple subnets. I don't think I skilled enough to do trunking inside a vswitch. So for us it gives us MPIO per the configs discussed above.

After talking with my network admin, from what we have talked about so far, can you get the same results if you just use separate subnets and NO vLans? This is an isolated private network for storage. This would allow us to use more of the iSCSI ports on the MD3200I? Thanks!