Unsolved
This post is more than 5 years old
2 Posts
0
20692
BroadCom 5720 Multi-Switch Connection failing to achieve full bandwidth
We have following Setup:
2 redundant Dell 2848 Switches, identically configured and connected via 8-Port LAG on Ports 41-48.
2 identically setup Dell R730 with BroadCom 5720. Operating System is Microsoft Server 2012-R2.
We wanted to make a fully redundant setup with maximum bandwidth. server S1&S2 port 1&2 are plugged into switch A and server S1&S2 port 3&4 are plugged into switch B.
On the server side, the 4 NIC are teamed together, on the switches the 2 ports each server is plugged in are put into LAG. Since the switches are only PowerConnect 2848 the LAGs are setup static. We only manage to achieve 2 GB/s.
SwitchA:
LAG A1(Server1 p1 Server1 p2)
LAG A2(Server2 p1 Server2 p2)
LAG A3(Switch2 p41..p48)
SwitchB:
LAG B1(Server1 p3 Server1 p4)
LAG B2(Server2 p3 Server3 p4)
LAG B3(Switch1 p41..p48)
If we unplug LAG A1, we have a short drop and go back to 2 GB/s
If we unplug LAG A2, we have a short drop and go back to 2 GB/s
All LAG plugged in again, next test:
We unplug Server1 p1, we have a short drop and go back to 2 GB/s
We unplug Server1 p3, we have a short drop and go back to 2 GB/s
This is where my confusion begins. Up until now i thought the multi-switch connection simply wasn't capable of bandwidth adding, but obviously with only one port plugged into each switch i once again have 2 GB/s. Yet i never had the hoped for 4 GB/s at the beginning.
When trying to find the source of the problem, more confusion arose:
4 NIC teaming on each server, no LAG on switch, all ports plugged into one switch: 1 GB/s
4 NIC teaming on each server, 4 port LAG on switch, all ports plugged into one switch: 4 GB/s
one cable unplugged: drop to 2 GB/s.
I fail to understand this behaviour, maybe i am missing something simple or maybe i am simply dumb, but if you can, please explain it to me or make suggestions how to achieve maximum bandwidth with maximum redundancy.
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
January 23rd, 2015 09:00
What testing method are you using to get these results? What teaming mode is being used? Are you setting up a active/passive type connection? LAGA1 active, LAGB1 passive?
In general teaming/link aggregation will not increase the speed of one connection, but benefit multiple connections to the device.
www.ieee802.org/.../frazier_01_0407.pdf
Two teams connected to two separate switches will serve as two separate connections and will not have their speeds aggregated.
Johannes Wagner
2 Posts
0
January 24th, 2015 02:00
We used the Windows 2012-R2 Datacenter Edition NIC Teaming. The 4 NICs are all active, the team is configured Switch-Independent dynamic without Standby-Adapters. I know that generally LAG do not provide increased bandwidth, but it is a feature of the Broadcom 5720 if i understood correctly.
We did the measurements via large File-Transfers between the servers because it is the actual usage we will need the bandwidth for. As already mentioned, we managed to get full bandwidth (4G, all ports plugged into one switch) and switch independent bandwidth aggregation (1 port per switch, 2 G combined bandwidth) but we failed to achieve full switch independent bandwidth aggregation.
On the server side it is always one virtual NIC that is handling the connection. What baffled me is when we tried an alternative failover configuration with 3 port LAG to at least get 3 GB/s bandwidth with only 1 GB/s failover, we only managed to get 2 GB/s.
PS: We tried the Broadcom Teaming feature with Generic Trunking (FEC/GEC)/802.3ad-Draft Static for similar results in all cases.
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
January 26th, 2015 11:00
These model of switches are pretty basic switches. In this situation I cannot think of any switch setting that would play a role in hindering/improving speeds. The next step you might take, is to use 3rd party software to test the network with. There are several different brands out there, any should do a good job of testing. For example Iperf has the ability to test speeds of a single connection, but can also setup multiple connections.
markdexter
6 Posts
0
March 29th, 2015 16:00
Hey mate,
No guarantees your having the same issue, but we we experiencing lower than expected throughput on some Dell 7 and 920s. Now I'm assuming your models are using Broadcom NICs, so please disregard this if they are not.
Our issue was caused by a driver issue resulting in VMQ's being handled improperly, a quick way to test is to go to your NIC's in device manager then under the "Advanced" tab ensure VMQ is disabled.
Run the test transfer test again and see what happens.
Hope this helps.