Solved!
Go to SolutionS
SkyWagona185f
6 Posts
0
19245
June 8th, 2010 08:00
VLAN Routing Performance Issues
Pair of 6224's stacked with several VLAN's. On VLAN 101 we have a large storage system and many servers. On VLAN 102 we many additonal servers. All servers are running Windows 2003. The 6224's are the core of the network with only field switches connected via LAG's. When running disk read/write performance tests from the servers to the NAS storage, we find that any server in VLAN 102 reads/writes at roughly 1/2 the rate of servers in VLAN 101. Example, server A IP 10.1.1.100 writes to NAS at 10.1.1.155 (routes through server field switch through core to NAS field switch) writes at roughly 80Mbps. Server B IP 10.2.2.100 writes to NAS at 10.1.1.155 (routes through server field switch through core to NAS field switch) writes at roughly 35Mbps. We can duplicate this performance across any server in our farms.
My thought is I am having performance issues when routing from the 102 to the 101 network.
We are running an old version of the firmware but before I upgrade I wanted to see if anyone has experienced anything like this.
Config:
show running-config
!Current Configuration:
!System Description "Dell 24 Port Gigabit Ethernet, 2.0.0.12, VxWorks5.5.1"
!System Software Version 2.0.0.12
!
configure
vlan database
vlan 101-103
exit
snmp-server location "xxxxxx"
snmp-server contact "xxxxxxx"
hostname "PC6224"
sntp unicast client enable
sntp server 192.43.244.18
clock timezone -7 minutes 0 zone "mdt"
stack
member 1 1
member 2 1
exit
ip address 192.168.168.1 255.255.255.0
ip name-server 10.1.1.205
ip name-server 10.1.1.206
ip routing
ip route 0.0.0.0 0.0.0.0 10.1.1.1
interface vlan 101
name "internal"
routing
ip address 10.1.1.5 255.255.255.0
exit
interface vlan 102
name "dmz"
routing
ip address 10.2.2.5 255.255.255.0
exit
interface vlan 103
name "public"
routing
ip address 216.38.214.135 255.255.255.128
exit
username "admin" password 1790728c3b2eea2bb079c4c5c98d0fe2 level 15 encrypted
spanning-tree mst configuration
name "00-1E-C9-86-9F-C4"
exit
!
interface ethernet 1/g2
switchport access vlan 101
exit
!
interface ethernet 1/g3
channel-group 5 mode on
switchport access vlan 101
exit
!
interface ethernet 1/g4
channel-group 8 mode on
switchport access vlan 101
exit
!
interface ethernet 1/g5
channel-group 9 mode on
switchport access vlan 101
exit
!
interface ethernet 1/g6
channel-group 9 mode on
switchport access vlan 101
exit
!
interface ethernet 1/g7
channel-group 8 mode on
switchport access vlan 101
exit
!
interface ethernet 1/g8
switchport access vlan 101
exit
!
interface ethernet 1/g9
channel-group 10 mode on
switchport access vlan 102
exit
!
interface ethernet 1/g10
channel-group 10 mode on
switchport access vlan 102
exit
!
interface ethernet 1/g11
switchport access vlan 101
exit
!
interface ethernet 1/g12
switchport access vlan 102
exit
!
interface ethernet 1/g13
channel-group 6 mode on
switchport access vlan 102
exit
!
interface ethernet 1/g14
channel-group 7 mode on
switchport access vlan 102
exit
!
interface ethernet 1/g15
negotiation 1000f
switchport access vlan 102
exit
!
interface ethernet 1/g16
negotiation 1000f
switchport access vlan 102
exit
!
interface ethernet 1/g17
channel-group 1 mode auto
switchport access vlan 102
exit
!
interface ethernet 1/g18
channel-group 2 mode auto
switchport access vlan 102
exit
!
interface ethernet 1/g19
channel-group 3 mode auto
switchport access vlan 103
exit
!
interface ethernet 1/g20
channel-group 4 mode auto
switchport access vlan 103
exit
!
interface ethernet 1/g21
switchport access vlan 103
exit
!
interface ethernet 1/g22
no negotiation
speed 100
switchport access vlan 103
exit
!
interface ethernet 1/g23
switchport access vlan 103
exit
!
interface ethernet 1/g24
spanning-tree cost 20000
switchport access vlan 103
exit
!
interface ethernet 2/g2
switchport access vlan 101
exit
!
interface ethernet 2/g3
channel-group 5 mode on
switchport access vlan 101
exit
!
interface ethernet 2/g4
channel-group 8 mode on
switchport access vlan 101
exit
!
interface ethernet 2/g5
channel-group 9 mode on
switchport access vlan 101
exit
!
interface ethernet 2/g6
channel-group 9 mode on
switchport access vlan 101
exit
!
interface ethernet 2/g7
channel-group 8 mode on
switchport access vlan 101
exit
!
interface ethernet 2/g8
switchport access vlan 101
exit
!
interface ethernet 2/g9
channel-group 10 mode on
switchport access vlan 102
exit
!
interface ethernet 2/g10
channel-group 10 mode on
switchport access vlan 102
exit
!
interface ethernet 2/g11
switchport access vlan 101
exit
!
interface ethernet 2/g12
switchport access vlan 101
exit
!
interface ethernet 2/g13
channel-group 6 mode on
switchport access vlan 102
exit
!
interface ethernet 2/g14
channel-group 7 mode on
switchport access vlan 102
exit
!
interface ethernet 2/g15
negotiation 1000f
switchport access vlan 102
exit
!
interface ethernet 2/g16
negotiation 1000f
switchport access vlan 102
exit
!
interface ethernet 2/g17
channel-group 1 mode auto
exit
!
interface ethernet 2/g18
channel-group 2 mode auto
exit
!
interface ethernet 2/g19
channel-group 3 mode auto
exit
!
interface ethernet 2/g20
channel-group 4 mode auto
exit
!
interface ethernet 2/g21
switchport access vlan 103
exit
!
interface ethernet 2/g22
switchport access vlan 103
exit
!
interface ethernet 2/g23
switchport access vlan 103
exit
!
interface ethernet 2/g24
switchport access vlan 103
exit
!
interface port-channel 1
description 'dmz uplink to F5'
switchport access vlan 102
exit
!
interface port-channel 2
description 'dmz uplink to F5'
switchport access vlan 102
exit
!
interface port-channel 3
description 'pub uplink to F5'
switchport access vlan 103
exit
!
interface port-channel 4
description 'pub uplink to F5'
switchport access vlan 103
exit
!
interface port-channel 5
description 'vlan 101 uplink'
switchport access vlan 101
exit
!
interface port-channel 6
description 'vlan 102 uplink dell 30.2'
switchport access vlan 102
exit
!
interface port-channel 7
description 'vlan 102 uplink 3com 15.4'
switchport access vlan 102
exit
!
interface port-channel 8
description 'vlan 101 uplink dell 15.4'
switchport access vlan 101
exit
!
interface port-channel 9
description '101 LAG to DB Switch'
switchport access vlan 101
exit
!
interface port-channel 10
description 'dmz link to 102 switch'
switchport access vlan 102
exit
snmp-server user admin READ_noAuthNoPriv
snmp-server community lp ro
exit
SkyWagona185f
6 Posts
1551
0
Posted June 10th, 2010 14:00
I assume you mean 10.1.1.100.
The NAS plugs into a field switch which uplinks via LAG to the core switch. All field switches LAG up to the core switches.
First thing to note is that 10.1.1.100 is on the same subnet as the NAS and for that reason there is no VLAN routing taking place. This will a count for some of the loss. Have you done a sniff on the this issue to see if you are getting fragmentation?
SkyWagona185f
6 Posts
1551
0
Posted June 11th, 2010 16:00
I'm confused with your statement. No problems are encountered when going from 10.1.1.x to the NAS, only hosts from 10.2.2.x that go to the NAS. See the included link to an image for more information on how our setup is.
http://picasaweb.google.com/lh/photo/EWyShLJPI4X60kxTI-64qVo8pd0Ww1AOesUgROneWbM?feat=directlink
SkyWagona185f
6 Posts
1551
0
Posted June 14th, 2010 15:00
Given the 2 switches are stacked (with stacking cable), I presume they are acting as 1 switch thus I would need to upgrade them together (unless you think I should unstack and upgrade). The documentation is not 100% clear about this. I intend to open a case with Dell prior to the upgrade. My 1st pass at the upgrade docs says 1st run the firmware upgrade, then run the bootcode upgrade from the boot prompt after the firmware upgrade is complete. Never done it like that before.
I have LAG's built across the switches in the stack for redundancy (say a 2 port LAG is on switch1/15 and switch 2/15). This allows for a full switch failure. The field switches are a mix of 3com 3870's and Dell PC5224's. I haven't tested with one switch for performance.
Definitely UNSTACK the switches and upgrade them separately. Updating the bootcode from the bootcode is key to avoid a chance at corrupting the flash file system of the switch.
3.2 firmware has fast master failover, so since you are counting on the multiple switches in the stack for redundancy, this will be a key feature for you.
SkyWagona185f
6 Posts
1551
0
Posted June 14th, 2010 16:00
Got it. When you say "update the bootcode from the bootcode", what do you mean? Does this differ from the upgrade documents?
SkyWagona185f
6 Posts
250
0
Posted August 13th, 2010 11:00
After performing the upgrade we observed about a 3X performance increase with inter Vlan routing.
bh1633
909 Posts
1551
1
Posted June 14th, 2010 15:00
2.0.0.12 firmware has some performance problems compare to the latest firmware (3.2.0.7). I highly recommend updating to the latest firmware. Be sure and follow the upgrade instructions CARFULLY. Specifically, update each switch separately, with no network traffic and update the bootcode from the bootcode, not the application firm.
Is the vlan 102 traffic traveling across the stacking links? Do you get the same performance when using a single switch? what are the other switches in your network?