The reason for the multiple network subnets is to allow for redundant networks. This would be a best practice to ensure connectivity to the storage. If you don't actually have redundant network switches you can use VLANs for now and be ready to add the switches later. The redundant netowrks is very simliar to how most fibre channel SANs are setup with redundant fabrics.

If you are not worried about losing a switch or worried about losing connectivity to the storage for a period of time, then you could just run it all across one network. This would make total sense for a test environment for example.

Thanks - Todd

Thanks Todd,
That all sounds good, and that is how I've got my SAN configured at the moment, but the theory behind it is still bugging me...
Assuming everything was on the same subnet, and I had 2 switches (which I do). If a switch died I still wouldn't lose connectivity to the storage array. If a storage processor went, failover should work there too. On paper it actually offers more redundant paths than having multiple network subnets. So I don't see how I would lose connectivity to the storage array.
I'm used to working with fibre SANs which is the reason I separated the subnets in the first place, however the more I think about it, it seems that the same subnet still ensures connectivity and would be easier to configure.
Am I missing something? What scenarios can you see where I might lose connectivity to the storage, if everything is on the same subnet?

You bring up a good point which I hadn't really thought through before. So I asked a couple of iSCSI experts here at Dell and the info below is what I got back.:

"It does not matter if you have separate subnets or the same subnet it's really user preference. Some folks like to run separate subnets to isolate I/O traffic from each other. You can do the same with VLANs.

What it boils down to is the switch configuration and type of switches.
Some switches have full port duplex meaning they have full port speed no matter how many connections to the switch. Others utilize a "port cluster" architecture where a cluster or group of ports share resources and with heavy I/O can be saturated by a single port in the cluster.
Other switches share resources across all the ports in the switch (this is really the worst case). Saturation can occur by a select few ports running really heavy I/O. If one app is killing the performance of the SAN because and hogging switch resources then it's a problem.

From a storage perspective he can use either configuration described but if he uses a single subnet with redundant switches he'll want to ISL (inter switch links) the switches so that if one link goes down somewhere in the mix, he doesn't lose performance."

Thanks,
Todd

There are a lot of differnet theories as to what is the best practices.. iSCSI is very versital and can be setup and supported to what works best for each environment.
There are several reasons I like seperate subnets for Windows hosts... I'm still working to come to a conclusion for ESX.
If you are using the ESX iSCSI initiator, you may want to have all IP addresses on the same subnet and team the NICs, but it seems ether way will work. There have been some posts on the Dell Community Forum regarding this. If you are using the Windows iSCSI initiator, I would use seperate subnets.
First, Windows typically complains when 2 NICs are on the same subnet unless they are teamed. Keep in mind, teaming is not supported by Microsoft for iSCSI traffic. Having redundant isolated networks increases reliability by limiting any broadcast storms to a single subnet. Another reason to place the NICs on separate subnets is for troubleshooting purposes. If you have 2 NICs on the same subnet, PING will only work on the first NIC in the binding order.

For ESX with two switches, I think you need to use seperate subnets, unless you connect the switches. We ran into problems with this in our failover testing. We found that ESX wouldn't fail over to a secondary card unless that card could see the same ISCSI controller addresses in the storage that the first card could see. We solved this by connecting our two switches, but I believe you can get this to work without interconnected switches, if you have two subnets.

We are having a chat today at 3 Central to discuss VMware and iSCSI - http://www.delltechcenter.com/page/TechCenter+Chat

If you miss the chat the transcript will be posted the following day.

Todd

I would like to use this configuration on MD3000i for use with Microsoft iSCSI initiator and VMware ESX 3.5:
controller 0, port 0 - 192.168.130.101/255.255.255.0
controller 0, port 1 - 192.168.131.101/255.255.255.0
controller 1, port 0 - 192.168.130.102/255.255.255.0
controller 1, port 1 - 192.168.131.102/255.255.255.0

This is the preferred configuration for Microsoft iSCSI initiator, but I am still not sure how to set this up in VMware ESX 3.5. I have a PE2950 with 6 pNIC. I would use 2 pNIC for iSCSI. I have two PC5448 switches for iSCSI. Should I configure two vSwitch? Like this:
controller 0, port 0--pSwitch0--pNIC0--vSwitch0(VMkernel0, ServiceConsole0)
controller 1, port 0-/

controller 0, port 1--pSwitch1--pNIC1--vSwitch1(VMkernel1, ServiceConsole1)
controller 1, port 1--/

VMkernel0: 192.168.130.229/255.255.255.0
VMkernel1: 192.168.131.229/255.255.255.0

I could also use a team for vSwitch0 (pNIC0, pNIC2) and for vSwitch0 (pNIC1, pNIC3), but I would like to use pNIC2 and pNIC3 for DMZ. pNIC4 and pNIC5 is used for LAN.

Is this a reasonable setup or am I missing something?

dell-svein, Everything you described looks correct.
You could consider establishing the iSCSI connections from within the guest OS as well. Current version of ESX only supports failover and not multipathing to a particular LUN. For example if your LUN was on Controller 0, I/O would only use VMKernel0 to Ctrl 0-0. If you used the MS iSCSI Initiator in the guest OS, you could establish 2 sessions, one from NIC1 to ctrl 0-0 and the other from NIC2 to ctrl 1-0. MDSD would configure MPIO to use Round Robin by default. So you might get better performance, I have not tested.

Dave T is correct here. If there is not a phyical connection between the 2 switches they have to be on seperate subnets. This also applies to direct attached storage as well. The IP routing wouldn't know which NIC to use, it would use the first NIC in the binding order on the subnet. Using a seperate subnet lets the OS know it has to use the other NIC.

I have been testing this configuration for failover today to be absolute sure it’s working. Four ping commands to each controller/port from ESX/Linux was started to see what was going on. Network cables was removed and inserted on my server pNICs and MD3000i pNICs. The failover worked better than expected :-) On vSwitch0 and vSwitch1 I also used NIC teaming. I removed pNICs cables (3) until only on was left on my ESX server. Even if I removed the last pNIC cable and inserted another pNIC cable on a different subnet the ongoing large-file-copy process on the iSCSI storage succeed. Of course it was a delay for a while when no network cable was attached to the server for iSCSI. I did the same thing on the MD3000i. The failover was better than expected. I am now calm about setting this in production. This configuration will also work with Microsoft iSCSI initiator with MPIO. (Who would exclude this possibility when setting up MD3000i?)

What I don’t understand is why DELL/VMware does not have a better guide for MPIO configuration. This http://www.dell.com/downloads/global/solutions/md3000i_esx_deploy_guide.pdf setup cannot be used with MS iSCSI MPIO.

I would like to upload some screen dumps the make this easier to understand – but it looks that it’s not possible to do that in this forum.