If you are using Multipath for iSCSI (I am assuming you are using iSCSI) I would keep them separate and not even link them with a LAG or trunk (Not really a need), I only uplink my SAN infrastructure switches to my core network for mgmt. This allows you to update one without taking down the stack and there isnt really a need for cross switch communication.
Thanks for the response! I had seen a few posts about not connecting the two switches at all. What I gathered from those, as well as the documentation was that inter-switch communication needed to occur because of the chattiness of the EQL arrays amongst themselves. In theory, I can visualize that should work, but I'm not sure what oversight I'd be introducing by not trunking them together via a LAG or stacking module. Why would Dell choose to recommend stacking or trunking if no switch interconnect is officially needed? Thoughts?
Yes, all of my ESX hosts, and my VM guest attached volumes use multipathing. I haven't bumped up from 4.0 to 4.1 yet, so I'm not using ESX's multipathing courtesy of the MEM, but just the traditional best practices method.
If these are for iSCSI (Storage Traffic Only) then they do not need to be interconnected or stacked, for normal network cmmunication stacking is worth while, for our core I have a stack of 6248's, but dedicated ones with only the management VLAN connected to the data network. (ie, I keep network and storage traffic separate). You can stack them or trunk them if you like, but if you use multipathing be sure to keep them on separate VLAN's and Subnets.
The stacking interconnect is rated at 48Gbps(2 X 12 full duplex ), and that is enough to support an EQL cluster of up to 12 arrays - the need for the high speed interconnect is that the arrays produce a lot of inter array traffic.
If you link the two switches together with a gig link aggregate then you are limited to the same maximum numner of arrays as with the 5400 switches ( maximum of three arrays ), but if you use the 10gig, then you could go up to 10 arrays.
Either way, it would give you the ability to upgrade the firmware without downing the iSCSI
Thanks for the response Cerbera. I guess the mystery here is the statement that "the arrays produce a lot of inter array traffic." You say it is needed, while the other person who posted said it is not. I believe the other poster said he does not have any interconnects of any sort.
I suppose one could "create" inter-array traffic if you wired the hosts and the EQL's goofy on the switches, but each array's controller and each host's controllers are wired for rendundancy per the best practices guide. So by that I mean, if you imagine one switch that has failed and is down, all systems would still be up, thus the inter array traffic is still flowing as it should. See what I mean? ...that is why I am a bit confused.
I initially moved from the 5424's to the 6224's for two reasons. 1.) I needed a second stack at my CoLo facility for my arrays there (2 arrays), and so with purchasing the 6224's for my primary site (2 going on 3 arrays), and 2.) I thought I could support more arrays (by freeing up the trunked interconnects) and rely on the stacking. Unfortunately I wasn't aware of the issues incured when you stack them, so I'm looking for alternatives.
Interestingly enough, I'm inquiring with the EQL and Powerconnect teams on the same issue, and seem to be getting similar conflicting information. I wanted to get a feel for what some of the users are doing
-4 Dell Poweredge R710's with multiple Network Cards
-Two Dell Powerconnect Switches (5424 but 62XX work also)
Dell Powerconnect switch 1 is Untagged VLAN 15 on all ports but Uplink.
Dell Powerconnect switch 2 is Untagged VLAN 16 on all ports but Uplink.
Uplink ports on Powerconnect are Untagged VLAN 0 only and connected to core network. (mgmt/monitoring)
R710's running Citrix Xenservers, Storage NICs of servers goes Switch 1/2, separate Subnets. MD3000i has two controllers with two network cards per controller, one of each goes to each switch. Hultipath is configured on Xenserver.
With this setup, no communication happens between Switch 1 and 2, I can power off either switch for updates/maintanence and storage traffix works no problem.
Did you end up keeping the stack or did you unstack the switches and use a LAG instead? Based on your experience, what would you recommend for a new setup?
I just noticed your post and the article is very interesting.
We have a pair of PowerConnect M6220 switches (modules for the M1000e Chassis) which are stacked. They connect our ESX hosts to our core network switches - our storage is fibre channel and so have their own switches.
We have decided to break the stack as rebooting the master switch caused the whole stack to go down for 60 seconds (as you discuss).
The major reason that this was not acceptable to us is the loss of communication between the ESX Hosts. This we found had 2 major negative consequences:
VM's on different hosts lost contact - if they were part of the same application (e.g web/database server), this could cause problems with the application.
The ESX hosts can declare themselves isolated and hence turn off all the VM's.
Scheduled maintenance work can be prepared for by turning off the host isolation response but an unscheduled master switch failure cannot in the same way. We have extended the time that an ESX host waits before "declaring" itself isolated from 15 seconds (default) to 3 minutes to prevent this happening.
However, the face that downtime is required for maintenance and that communication is lost between hosts, outweighs any benefits of stacking for us as our environment is 24/7 and hence maintenance is much more difficult to organise.
Out of interest, do you know of any good online resources/blogs on how to split a PowerConnect M6XXX stack? We need to do it so to minimse downtime.
Boy, I definately here you! Isolation alerts are another aspect of this that makes it difficult. ...My take on it through my discovery process was that nobody related to design functionality truly understand how lack of "maintenance redundancy" is such a hinderance. Shoot, 10 years ago I could shut down my whole network over the weekend and nobody would notice. Now, if something is restarted at 3:00am on a Sunday morning, I'm likely to hear about it. ...but I digress.
I am not aware of any good resources that spell out the specifics which incorporate my information, but in a LAG'd approach. Knowing that LAG'ing best practices seem to change as well, that complicates matters. Your environment (number of arrays, which types, etc.) add some variables in there too. I did have many conversations with Dell on the matter. Below is some correspondance I had captured from Dell on the matter:
--------------------------
One important highlight: If a SAN with more than 3 EqualLogic arrays is required, it is recommended that stackable switches such as the Cisco Systems® 3750 family or other models of stacking switches be utilized.
From my experience Stacked switches are always easier to deal with. The stacking cable always have enough bandwidth to support the inter switch traffic.
Customer have very few issues getting stacked switches to work properly.
Non-Stacked switches customer seem to have multiple issues trying to get the correct number of port channels or getting it configured correctly.
Also the recommendations seem to change often about how to configure the port channel. For example presently we want people to enable flow control for the port channel.
Last month this was not the recommendation
---------------------------------
When I was split on my decision, I was building up a LAG configuration that would work if I decided to go in that direction. I can't gaurantee you that the clipped configuration below will work, but its going down the right road I beleive if you want it LAG'd.
Step 1: Reset the switch to defaults if previously configured.
enable
delete startup-config
reload
Step 2: When prompted, follow the setup wizard in order to establish your management IP, SNMP (if applicable to your environment) and default web admin/password.
Step 3: Configure the switch.
Put the switch into admin and configuration mode.
enable
configure
Establish Management Settings
hostname iSCSI-SW01
enable password Password1!
spanning-tree mode rstp
flowcontrol
Add the appropriate VLAN IDs to the database and setup interfaces.
vlan database
vlan 10
vlan 100
exit
interface vlan 1
exit
interface vlan 10
name Management
exit
interface vlan 100
name iSCSI
exit
ip address vlan 10
Create an Etherchannel Group for the LAG
interface port-channel 1
switchport mode trunk
switchport trunk allowed vlan add 100
no storm-control unicast
mtu 9216
exit
NOTE: The management VLAN is not spanned across the LAG and was done intentionally to allow customers to plug port 1 of both switches back into their existing network for management redundancy without causing a spanning-tree violation (loop).
Configure Ports 1-2 as management access Switchports
interface range ethernet 1/g1-1/g2
switchport access vlan 10
spanning-tree portfast
mtu 9216
exit
Add Ports 3-9 to the LAG
interface range ethernet 1/g3-1/g6
channel-group 1 mode on
mtu 9216
exit
Configure Ports 10-24 as iSCSI access Switchports
interface range ethernet 1/g7-1/g24
switchport access vlan 100
no storm-control unicast
spanning-tree portfast
mtu 9216
exit
NOTE: If this is the 6248 switch you can change g24 to g48 in both of the above configuration statements to set all of the remaining ports for iSCSI.
Exit from configuration mode
exit
Save the configuration!!
copy running-config startup-config
DONE!
NOTE: YOU MUST REPEAT THIS PROCESS ON THE SECOND SWITCH BEFORE IMPLEMENTING
Thanks, that was very useful & interesting. I will use it when planning our configuration changes for splitting the stack.
One query I have which regarding your configuration is that you specify "mtu 9216" in both the "interface port-channel 1" and the ports3-9 to which you add the "channel-group". Is this necessary or can it be left out of the ports3-9 configuration, just picking it up from the channel-group?
I ask as we have done the same with vlans on another switch we have, declaring them both in the port-channel and on the interface itself - something I didn't know whether was necessary or not.
bdearlove
65 Posts
0
May 19th, 2011 07:00
If you are using Multipath for iSCSI (I am assuming you are using iSCSI) I would keep them separate and not even link them with a LAG or trunk (Not really a need), I only uplink my SAN infrastructure switches to my core network for mgmt. This allows you to update one without taking down the stack and there isnt really a need for cross switch communication.
sketchy00
203 Posts
0
May 19th, 2011 08:00
Hello BDEARLOVE,
Thanks for the response! I had seen a few posts about not connecting the two switches at all. What I gathered from those, as well as the documentation was that inter-switch communication needed to occur because of the chattiness of the EQL arrays amongst themselves. In theory, I can visualize that should work, but I'm not sure what oversight I'd be introducing by not trunking them together via a LAG or stacking module. Why would Dell choose to recommend stacking or trunking if no switch interconnect is officially needed? Thoughts?
Yes, all of my ESX hosts, and my VM guest attached volumes use multipathing. I haven't bumped up from 4.0 to 4.1 yet, so I'm not using ESX's multipathing courtesy of the MEM, but just the traditional best practices method.
bdearlove
65 Posts
0
May 25th, 2011 05:00
If these are for iSCSI (Storage Traffic Only) then they do not need to be interconnected or stacked, for normal network cmmunication stacking is worth while, for our core I have a stack of 6248's, but dedicated ones with only the management VLAN connected to the data network. (ie, I keep network and storage traffic separate). You can stack them or trunk them if you like, but if you use multipathing be sure to keep them on separate VLAN's and Subnets.
cerbera_a84f2d
176 Posts
0
May 25th, 2011 07:00
The stacking interconnect is rated at 48Gbps(2 X 12 full duplex ), and that is enough to support an EQL cluster of up to 12 arrays - the need for the high speed interconnect is that the arrays produce a lot of inter array traffic.
If you link the two switches together with a gig link aggregate then you are limited to the same maximum numner of arrays as with the 5400 switches ( maximum of three arrays ), but if you use the 10gig, then you could go up to 10 arrays.
Either way, it would give you the ability to upgrade the firmware without downing the iSCSI
sketchy00
203 Posts
0
May 25th, 2011 08:00
Thanks for the response Cerbera. I guess the mystery here is the statement that "the arrays produce a lot of inter array traffic." You say it is needed, while the other person who posted said it is not. I believe the other poster said he does not have any interconnects of any sort.
I suppose one could "create" inter-array traffic if you wired the hosts and the EQL's goofy on the switches, but each array's controller and each host's controllers are wired for rendundancy per the best practices guide. So by that I mean, if you imagine one switch that has failed and is down, all systems would still be up, thus the inter array traffic is still flowing as it should. See what I mean? ...that is why I am a bit confused.
I initially moved from the 5424's to the 6224's for two reasons. 1.) I needed a second stack at my CoLo facility for my arrays there (2 arrays), and so with purchasing the 6224's for my primary site (2 going on 3 arrays), and 2.) I thought I could support more arrays (by freeing up the trunked interconnects) and rely on the stacking. Unfortunately I wasn't aware of the issues incured when you stack them, so I'm looking for alternatives.
Interestingly enough, I'm inquiring with the EQL and Powerconnect teams on the same issue, and seem to be getting similar conflicting information. I wanted to get a feel for what some of the users are doing
bdearlove
65 Posts
0
May 27th, 2011 08:00
Here is what I have setup for my Storage traffic.
-One MD3000i Storage Array with two controllers
-4 Dell Poweredge R710's with multiple Network Cards
-Two Dell Powerconnect Switches (5424 but 62XX work also)
Dell Powerconnect switch 1 is Untagged VLAN 15 on all ports but Uplink.
Dell Powerconnect switch 2 is Untagged VLAN 16 on all ports but Uplink.
Uplink ports on Powerconnect are Untagged VLAN 0 only and connected to core network. (mgmt/monitoring)
R710's running Citrix Xenservers, Storage NICs of servers goes Switch 1/2, separate Subnets. MD3000i has two controllers with two network cards per controller, one of each goes to each switch. Hultipath is configured on Xenserver.
With this setup, no communication happens between Switch 1 and 2, I can power off either switch for updates/maintanence and storage traffix works no problem.
pzero
1 Rookie
•
108 Posts
0
August 26th, 2011 07:00
Did you end up keeping the stack or did you unstack the switches and use a LAG instead? Based on your experience, what would you recommend for a new setup?
Thanks.
sketchy00
203 Posts
0
August 26th, 2011 08:00
Hi PZero,
I ultimately chose to keep it stacked. I explained why, and gave a detailed step-by-step set of instructions at: itforme.wordpress.com/.../reworking-my-powerconnect-6200-switches-for-my-iscsi-san
John1483
20 Posts
0
August 26th, 2011 11:00
Hi Sketchy
I just noticed your post and the article is very interesting.
We have a pair of PowerConnect M6220 switches (modules for the M1000e Chassis) which are stacked. They connect our ESX hosts to our core network switches - our storage is fibre channel and so have their own switches.
We have decided to break the stack as rebooting the master switch caused the whole stack to go down for 60 seconds (as you discuss).
The major reason that this was not acceptable to us is the loss of communication between the ESX Hosts. This we found had 2 major negative consequences:
VM's on different hosts lost contact - if they were part of the same application (e.g web/database server), this could cause problems with the application.
The ESX hosts can declare themselves isolated and hence turn off all the VM's.
Scheduled maintenance work can be prepared for by turning off the host isolation response but an unscheduled master switch failure cannot in the same way. We have extended the time that an ESX host waits before "declaring" itself isolated from 15 seconds (default) to 3 minutes to prevent this happening.
However, the face that downtime is required for maintenance and that communication is lost between hosts, outweighs any benefits of stacking for us as our environment is 24/7 and hence maintenance is much more difficult to organise.
Out of interest, do you know of any good online resources/blogs on how to split a PowerConnect M6XXX stack? We need to do it so to minimse downtime.
Thanks
John
sketchy00
203 Posts
0
August 26th, 2011 11:00
Hi John,
Boy, I definately here you! Isolation alerts are another aspect of this that makes it difficult. ...My take on it through my discovery process was that nobody related to design functionality truly understand how lack of "maintenance redundancy" is such a hinderance. Shoot, 10 years ago I could shut down my whole network over the weekend and nobody would notice. Now, if something is restarted at 3:00am on a Sunday morning, I'm likely to hear about it. ...but I digress.
I am not aware of any good resources that spell out the specifics which incorporate my information, but in a LAG'd approach. Knowing that LAG'ing best practices seem to change as well, that complicates matters. Your environment (number of arrays, which types, etc.) add some variables in there too. I did have many conversations with Dell on the matter. Below is some correspondance I had captured from Dell on the matter:
--------------------------
One important highlight: If a SAN with more than 3 EqualLogic arrays is required, it is recommended that stackable switches such as the Cisco Systems® 3750 family or other models of stacking switches be utilized.
From my experience Stacked switches are always easier to deal with. The stacking cable always have enough bandwidth to support the inter switch traffic.
Customer have very few issues getting stacked switches to work properly.
Non-Stacked switches customer seem to have multiple issues trying to get the correct number of port channels or getting it configured correctly.
Also the recommendations seem to change often about how to configure the port channel. For example presently we want people to enable flow control for the port channel.
Last month this was not the recommendation
---------------------------------
When I was split on my decision, I was building up a LAG configuration that would work if I decided to go in that direction. I can't gaurantee you that the clipped configuration below will work, but its going down the right road I beleive if you want it LAG'd.
Step 1: Reset the switch to defaults if previously configured.
enable
delete startup-config
reload
Step 2: When prompted, follow the setup wizard in order to establish your management IP, SNMP (if applicable to your environment) and default web admin/password.
Step 3: Configure the switch.
Put the switch into admin and configuration mode.
enable
configure
Establish Management Settings
hostname iSCSI-SW01
enable password Password1!
spanning-tree mode rstp
flowcontrol
Add the appropriate VLAN IDs to the database and setup interfaces.
vlan database
vlan 10
vlan 100
exit
interface vlan 1
exit
interface vlan 10
name Management
exit
interface vlan 100
name iSCSI
exit
ip address vlan 10
Create an Etherchannel Group for the LAG
interface port-channel 1
switchport mode trunk
switchport trunk allowed vlan add 100
no storm-control unicast
mtu 9216
exit
NOTE: The management VLAN is not spanned across the LAG and was done intentionally to allow customers to plug port 1 of both switches back into their existing network for management redundancy without causing a spanning-tree violation (loop).
Configure Ports 1-2 as management access Switchports
interface range ethernet 1/g1-1/g2
switchport access vlan 10
spanning-tree portfast
mtu 9216
exit
Add Ports 3-9 to the LAG
interface range ethernet 1/g3-1/g6
channel-group 1 mode on
mtu 9216
exit
Configure Ports 10-24 as iSCSI access Switchports
interface range ethernet 1/g7-1/g24
switchport access vlan 100
no storm-control unicast
spanning-tree portfast
mtu 9216
exit
NOTE: If this is the 6248 switch you can change g24 to g48 in both of the above configuration statements to set all of the remaining ports for iSCSI.
Exit from configuration mode
exit
Save the configuration!!
copy running-config startup-config
DONE!
NOTE: YOU MUST REPEAT THIS PROCESS ON THE SECOND SWITCH BEFORE IMPLEMENTING
John1483
20 Posts
0
August 30th, 2011 10:00
Hi Sketchy
Thanks, that was very useful & interesting. I will use it when planning our configuration changes for splitting the stack.
One query I have which regarding your configuration is that you specify "mtu 9216" in both the "interface port-channel 1" and the ports3-9 to which you add the "channel-group". Is this necessary or can it be left out of the ports3-9 configuration, just picking it up from the channel-group?
I ask as we have done the same with vlans on another switch we have, declaring them both in the port-channel and on the interface itself - something I didn't know whether was necessary or not.
Thanks
John