MD3000i and vmware : redundant iscsi setup

sorry for the long post ... here is the continuation

conclusion :
- the one-subnet setup doesn't work at all ... if the switches are not connected
- ... and only works 'sometimes' with a crossover link between the two switches

2) Two separate subnets
Although discouraged by every support person I spoke with, I tried this setup :
- in VM : two separate virtual switches (each with its own service console and vmkernel) dedicated for iscsi. One physical nic attached to each virtual switch
With this setup ... :
- I was able to use 2 different data-paths over both of my physical NICs (creating a kind of manual load balancing)
- I could not make this to fail, as long as one of the 4 iscsi-paths was available. Failover sometimes took 'a while', but at least I never lost my data connection.

So "what's the question than" .... you would probably ask :-)
- First of all : I wanted to share this info with others, since not much info about this seems to be available and almost all people I spoke with (as well Dell as VMWare support) were unable to support me with this setup.
- Next, although the problem seems to be solved, I got a couple of concerns about it :
1) 'in theory' the "one-subnet" seems to provide more redundancy (think at nic-teaming, ...), but 'in practice' however ...
2) definitely, the one-subnet setup is easier to setup and maintain
3) VMWare support explicitly told me that the two-subnet setup is officialy not supported and only provided as a 'best effort' solution.

So for that reason, I'm still looking for a way to get this 'one-subnet' setup to work in my configuration ? Or (if applicable) an official recommandation that the 'two-subnet' setup is definitely the best solution.

Thanks in advance !

Koen.

Koen,

How do you have your virtual switches setup for iSCSI in the one subnet solution? I assume that you have a single virtual switch with both physical NICs. Also, how did you test the failures? Did you simply remove network cables? I want to test this same scenario in my lab so I just wanted a few more details of what you did.

All of the best practices that I have seen about this type of setup have recommended a single subnet, with two switches and an innerswitch link. I believe that it is possible to have multiple ISLs for redundancy and increased bandwidth between the switches.

Thanks,
Todd

(sorry, posted to soon)

... and yes : you 're trough.
Everyone advices to use one subnet with ESX
I myself see some extra advantages in it :
- setup is easier
- less resources used for additional vswitch, vmkernel, service console, ...

so if my setup could be fixed to run with one subnet ... that would be great !

thanks,
Koen

Hi Todd,
thanks for "picking this up"
To answer your questions :
1) indeed: one vswitch, two nics, nic teaming with default settings
2) testing, by disconnecting utp cables, or by powering down switches
If you want : I've got a couple of nice drawings (pdf) describing my setup and the test scenario's I went trough. So I would be happy to share them with others ... problem is, that I don't know how to upload them to this site :-(
Maybe someone can advice me on this

thanks in advance for the assistance,
Koen.

I tested the MD3000i with dual controllers, attached to two Power Connect switches with ESX host attached with two physical NICs. Each of the MD3000i controllers has two ethernet ports, so each was controller was attached to both switches. There was an interconnect ISL between the two Power Connect 5448s. The ESX host was attached to both powerconnect switches - One NIC to each. These two NICs were then connected to the same virtual switch.

I tested failures by removing cables for either NIC on the server, powering off one switch at a time, and removing both cables from a storage controller on the MD3000i.

I did not get a failure at any time. I also did not have to manually rescan. So send me your PDF and I will take a look at them. (Check the message that I sent you)

Thanks,
Todd

Koen,

Thanks for sending over the detailed pictures with the specifc failure scenarios you are testing. There are two issues that I found:

1. To test more than one failed component at a time is not really fair. The probability of having multiple failed components at the same time is less than having an entire site failure. Usually testing is done to ensure no single point of failure.

2. The switches that you are using are entry level non-managed switches - the PowerConnet 2716. I would recommend that you use a higher class managed ethernet switch. (Of course it does appear from your testing that the switches do work in normal use).

Thanks,
Todd

Dell's configuration guide http://www.dell.com/downloads/global/solutions/esx_storage_deployment_guide_v1.pdf (Great Document, by the way !) gives 2 nice drawings on how to do the cabling for both possible setups. But refers to vmware's http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_iscsi_san_cfg.pdf for software configuration of it. I reread the chapter about 'setting up software iscsi initiator' (p. 50-55) in this document a hundred times, but could not find a description on how to add a second nic and/or redundant data paths
- should this second nic be added to the same vswitch or should I create an additional vswitch ?
- If using nic teaming, what parameters should be set ?
- or if I add a second vswitch : should I give it an extra vmkernel ? and an extra service console ?
- ...

Thats why I posted my questions in this group, hoping to find some extra advice. (which I 'm happy to find here at this moment :-)

2. Regarding the gigabit switch : I asked my account manager on before what kind of benefits a PC5424 switch would have, but he couldn't find any (for our particular setup). That, on top of the fact that the PC2716 is officialy supported with the MD3000i (http://support.dell.com/support/edocs/systems/md3000i/en/SUPPORTMATRIX/3000isupportmatrix.pdf), made us decide to buy two PC2716 switches.
But knowing what troubles we have with the setup now, I 'm thinking : Would my problems be solved by just changing the switches to PC5424 models ?

Once again : thanks a lot for helping me out with this.

Thanks,
Koen.

Hi Todd,

1. Sorry that I created an un-fair test-scenario. I was not aware of the fact that there were specific rules about this. I was only thinking on 'what can go wrong'. And I agree : the failover scenario I tested is not likely to happen every day but came in my mind at the moment I found out that 'that simple connection' between both switches was responsible for keeping alive 50% of my iscsi-paths (ie: 2 out of 4).
As I described it in my original post :
<
in case of a failure of the interswitch-link (which can't be monitored) a subsequent failure of nic1 or switch1 would cause a failure of the data-connection (and a manual rescan to fail-over to nic2)
>
Especialy the fact that a failure of the inter-switch-link can't be automaticly noticed, makes this (in my opinion) a risky situation.
That, on top of the two other issues I mentioned :
<
- all connections were initiated by the first nic (no load balancing over the 2 nics)
- more often than not, the most NON-optimal 'data paths' were used. I mean : [nic1-switch1-switch2-san] while it could have been [nic1-switch1-san]
>
... gave me (and still gives me) an un-comfortable feeling with this setup.

That's why I came to the 2-subnet-setup (a bit inspired by the FC-setup I 'm used to work with).
At least this setup seems to survive my worst-case-scenario's :-)
And I seem to be able to guide the network traffic over the most optimal data paths, and even trough both nics at the same time (ie: on a per-lun-basis)

Unfortunatly I couldn't find any documentation/whitepaper/recommandation/... that confirms (or objects) and arguments my experiences.
In fact : I couldn't find any document that described how to do the network-part of the iscsi-setup.

sorry (my post is to long ... again) ...

Koen,

I believe that the way to confiure the iSCSI network on the VMware side for this scenario is to create a single virtual switch and attach both physical NICs.

The PC2716 will work with the MD3000i, as you have seen in your testing. However the features that you have in terms of managment and configuration are less than what you would get with a 5424. This additional management and configuaration would most likely help you with the ISL configuration and monitoring. Trying to get into my own 5448 to give some more details about what can be done.

Todd

My understanding is that the PC2716 does not support Multi/Rapid Spanning Tree, Port fast, this would explain why you have to rescan after a link failure. Where the PC5424 has much better Spanning Tree features so can much better detect link changes.
The other advantage of a PC5424 is that you could monitor the trunk between the two switch via snmp, syslog, telnet.

According to the information I find here, I asume that the problems I've got with my 1-subnet-setup are related to the fact that I'm using the wrong switches.

Suppose I buy new switches (5424 or 6xxx ?), I think that simply replacing the two 2716's I've got, would not solve my problems, would it ?
In other words : what can these switches do more ? And especialy : what special configuration should they get ?

As an example: roadfix already mentioned SPT and portfast, and yes, i know that on an SPT-enabled switch with incorrect (ie: fast link=disabled) setting, it can take a while (45 sec ?) before a new connection 'comes up'. But since my switches have no SPT-config at all, that 'effect' can't play in my config, can it ?

What surprises me so much is (although pretty important, as I see it now) that none of the "configuration guides" (including the ones I refered to in the previous posts) mentions any switch-feature or -setting that should be set/checked in order to get a good-working iscsi setup.

Does anyone know of a good whitepaper/configurationguide for setting up the networking-part of iscsi ???

Specific for my setup : the iscsi-network is physicaly separated from the 'normal network', so i suppose that makes it easier than in a mixed environment.

Thanks in advance,
Koen.

Hi,
regarding the previous remarks about my non-manageble switches :
I might be able to "get my hands on" two PC5324 switches (ssstt: don't tell anyone ... they might be missing them :-)
Does it make sence to use these onces in stead of the PC2716 I'm using now ?
What settings on these manageble switches should I play with to make my setup 100% redundant ?
TIA,
Koen.

Dear All,

During my setup I was getting confused about the different network diagrams that are given on Dell's website regarding the correct iscsi network cabling. With this posting I hope to help others in getting there setup right from the first time.

Please compare the following 3 references :
1) Dell™ PowerVault™ Modular Disk 3000i - Systems Installation Guide
http://support.dell.com/support/edocs/systems/md3000i/en/IG/PDF/IGbk0HR.pdf
figure 2-5 on page : 15
2) Video : MD3000i Installation and Configuration
http://delltechcenter.com/page/Storage+Demos
diagram shown from 00:19 'till 00:44
3) DellTM PowerVaultTM Configuration Guide for VMware ESX/ESXi 3.5
http://www.dell.com/downloads/global/solutions/esx_storage_deployment_guide_v1.pdf
figure 1a on page 5

I first set up my MD3000i following the figure in the system installation guide [ref 1) ].
This is the same setup as in the installation video [ref 2) ]
In both setups, the iscsi cables from switch 1 to the MD3000i are connected to port 0-0 and 0-1. ie: the primary ports of each Storage processor are connected to the same switch

Having problems during reduncy tests, I was adviced by VMWare support to change the setup as described in [ref 3) ]. Doing this, the primary iscsi port of the first storage processor (0-0) is connected to switch1. And the primary iscsi port of the second storage processor (1-0) is connected to switch2.
Connection this way, the redundant setup survived a failure of switch1
=> The iscsi setup of [ref 3) ] is the only correct one, I asume !!

Please correct me if I'm wrong ... Or correct /complete the documentation in case I'm right :-)

additionaly : in case of a 1-subnet setup (with vmware iscsi initiator ?) : the ISL (network connection between switch1 and switch2) is also essential

Kind regards,
Koen.

Koen:
On your second post you mentioned you tried a dual-subnet setup although it was discouraged by VMware support. Would you post a few more details on this setup? For example, I'm curious about:

- Does it still work as a single Software iSCSI initiator, or you got 2 with this setup (differents iQNs) on the ESX server?

- Does it shows as 2 "iSCSI Storage Adapters" under "Storage Adapters" tab on ESX Configuration?

- Do you have to "Rescan..." only once to see the LUNs from the ESX, or have to do it twice?

- How does ESX handle multipathing in such setup?. In fact, on the single-subnet setup, "multipathing" using NIC teaming is handle by the network layer, and strictly speaking it's not a Storage Multipath feature.

- Does this setup have a performance advantage over the single-subnet setup in terms of load balancing?

Now talking about the switches (PC 2716), as you probably already known, you can create Link Aggregation Groups containing up to 4 ports and then connect this LAGs groups between the two switches to implement the ISL's. As you mentioned, this is mandatory for the single-subnet setup.

Generally speaking, and this is my personal opinion, VMware iSCSI support have a lot of dark spots, which are more evident when you compare it with MS iSCSI initiator features, strong multipathing support, and configuration flexibility.

Best Regards,
Tony

... continued ... :

5) I'm not sure about the performance impact. In theory the maximum troughput should be better, since both nics can be used simultanious, where as in the one-subnet-setup one-and-only-one nic is used. However during my load-tests, the maximum troughput I saw on my setup was only about 160Mbit/sec. So that was much smaller than the (theoretical) maximum of one (1) Gigabit NIC. Of course there are other parameters that influent this (number of disks, raid config, test program, host-performance, os, ...)
If you like discussing about the performance of our SAN : this afternoon/evening (15:00 CST = 22:00 CET) there is a techchat session about MD3000i Performance with the DELL guys of the delltechcenter. You can participate on this link : http://www.delltechcenter.com/page/12-02-2008+MD3000i+Performance
If I'm home in time, I might meet you there :-)

6) thanks for your tip about the LAG's. I didn't knew what they were used for before. All I did to create the ISL, was connecting a crosscable between both switches. Do you maybe have a good link to learn more about this subject ? It sounds like this might solve the problems I'm facing with the one-subnet setup !

7) I took a short look at the MS iscsi initiator, but I choose for the VMWare initiator since I only had to do the iscsi-setup once for all my VM-guests. This is particular interesting because my VM Guests contain different OS's (from WinNT to Win2003 and a couple different linux flavours)

Kindest regards,
Koen.