LACP configuration problem with 8132F and R515 (10G Intel SFP)

Question

Hello.

I got a problems when configuring LACP with Linux machine now. That is a R515 with Intel Corporation 82599EB 10-Gigabit SFP interfaces. And then connected to 2 stacking 8132F switch respectively. ie. SFP port 1 connected to first 8132F and then port 2 connected to 2nd 8132F.

It strange now is sometimes could not connect those machines but sometimes can. Sometimes bond0 is connect but bond0.20 is cann't. Also, some file transfer test result showing only 10G (I expect is more then 10G) bandwidth can make.

It already changed many different configs but no luck. Please advise about this. The following is those configuration for reference.

Additional what is the relationship between Switch Hash Configuration and xmit_hash_policy under linux? is it they need match it?

Linux: CentOS 6
---- interface -----
/etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
BOOTPROTO=none
#BROADCAST=103.168.235.47
IPADDR=103.168.235.35
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
NETMASK=255.255.255.240
NETWORK=103.168.235.32
ONBOOT=yes
GATEWAY=103.168.235.33
TYPE=Ethernet
USERCTL=no
#MTU=9000
BONDING_OPTS="miimon=100 mode=4 xmit_hash_policy=1"

----
/etc/sysconfig/network-scripts/ifcfg-bond0.20

DEVICE=bond0.20
BOOTPROTO=none
BROADCAST=192.168.20.255
IPADDR=192.168.20.22
NETMASK=255.255.254.0
NETWORK=192.168.20.0
#GATEWAY=202.52.136.193
#DNS1=8.8.8.8
#DNS2=8.8.4.4
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
VLAN=yes
#MTU=9000

----
/etc/sysconfig/network-scripts/ifcfg-p2p1

DEVICE="p2p1"
BOOTPROTO=none
IPV6INIT="no"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
UUID="81fc0465-1601-4493-8b05-fc8e0268e79c"
HWADDR=90:E2:BA:57:E8:20
DOMAIN=abc.hk
#DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
NAME="System p2p1"
MASTER=bond0
SLAVE=yes
USERCTL=no

----
/etc/sysconfig/network-scripts/ifcfg-p2p1

DEVICE="p2p2"
BOOTPROTO="none"
HWADDR="90:E2:BA:57:E8:21"
IPV6INIT="no"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
UUID="dd6b70f3-7ca9-428a-b865-5c19189962cd"
MASTER=bond0
SLAVE=yes
USERCTL=no

---
cat /etc/modprobe.d/bond.conf
alias bond0 bonding

---- status of bonding ----

cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
   Aggregator ID: 1
   Number of ports: 1
   Actor Key: 33
   Partner Key: 1
   Partner Mac Address: 00:00:00:00:00:00

Slave Interface: p2p1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 90:e2:ba:54:09:8c
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: p2p2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 90:e2:ba:54:09:8d
Aggregator ID: 2
Slave queue ID: 0

----------------------
In the switch connection
That is a two 8132F and stacking together already

!
interface Te1/0/3
channel-group 1 mode on
description "pcs01"
switchport mode trunk
exit
!
interface Te1/0/4
channel-group 2 mode on
description "pcs02"
switchport mode trunk
exit
!
interface Te1/0/5
channel-group 3 mode on
description "pcs03"
switchport mode trunk
exit
!
interface Te2/0/3
channel-group 1 mode on
description "pcs01"
switchport mode trunk
exit
!
interface Te2/0/4
channel-group 2 mode on
description "pcs02"
switchport mode trunk
exit
!

interface port-channel 1
hashing-mode 1
switchport mode general
switchport general allowed vlan add 20 tagged
exit
!
interface port-channel 2
hashing-mode 6
spanning-tree disable
spanning-tree mst 0 cost 1000
spanning-tree cost 1000
switchport mode general
switchport general allowed vlan add 20 tagged
exit
!
interface port-channel 3
hashing-mode 6
spanning-tree disable
spanning-tree mst 0 cost 1000
spanning-tree cost 1000
switchport mode general
switchport general allowed vlan add 20 tagged
exit
----------------

Thanks!

Machi

DELL-Josh Cr · Answer

How are you testing the performance? Copying a file from one machine to another is probably not going to use both members of a LAG as it is a single connection on the same tcp/udp port, how the LAG works is that it separates traffic based on the port, so if a copy operation is on one port it is using a single nic and if a second network request is sent for a webpage on a different port it would go out the other nic, but if there are two copy operations on a single port they both will use one nic. You can change this by changing the xmit_hash_policy in linux to change how the computer separates the traffic. https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sec-Using_Channel_Bonding.html

The default method in CentOS/RHEL6 is by MAC address, so connections to a single other system for a file copy are not going to split across both nics. You can try to use layer3/4 information for unfragmented packets and it will transmit per packet, but if the packets are fragmented it will use the default method. The 3rd option is to use MAC and ip addresses, but won’t separate on a single copy operation.

The method that the switch uses and the computer uses do not have to match. You are using hashing mode 1 which is probably the best option for you to see improved performance in copy operations as it will transmit per packet on unfragmented packets. However the switch is going to split based on port and ip address and not per packet and there is not a hashing method on these switches that will.

As for the nics not connecting sometimes, what does it show is the link showing down on the switch and on the server?

machiasiaweb2 · Answer

Hello, Thanks for information.  But my first task is bring up the LACP/LAG between switch and server.  Did any further advise?   Thanks! Machi

DELL-Josh Cr · Answer

To configure the LAG on the switch you select the ports and add them to a channel-group ftp://ftp.dell.com/Manuals/all-products/esuprt_ser_stor_net/esuprt_powerconnect/powerconnect-8100_User%27s%20Guide_en-us.pdf

Page 794

machiasiaweb2 · Answer

Hello,

Yes, I also following those config with example but no luck.

- Additional it also fail when I tried disable either one of NIC from my host machine.

- or change the LAG config to port access mode also fail.

Port Channel summary like following for reference.

console#show interfaces port-channel 5

Channel   Ports                         Ch-Type Hash Type Min-links Local Prf
-------   ----------------------------- -------- --------- --------- ---------
Po5       Active: Te2/0/5, Te1/0/5      Static   3         1         Disabled

console#show interfaces port-channel 7

Channel   Ports                         Ch-Type Hash Type Min-links Local Prf
-------   ----------------------------- -------- --------- --------- ---------
Po7       Active: Te1/0/7, Te2/0/7      Static   1         1         Disabled

Thanks for further advise

DELL-Josh Cr · Answer

Try creating a static LAG instead of LACP and see if that works better. It is possible that something with the LACP packets is the problem.

machiasiaweb2 · Answer

Hello, Unfortunately already setup as a LAG from the first day.  It can see by port-channel status shown 'static' there.

DELL-Josh Cr · Answer

If it is a static LAG than it doesn’t need LACP packets to work and is going to be always on and the issue is not on the switch side.

machiasiaweb2 · Answer

Hello, So, the problem may be from HOST side?  Any others reference? or easy to debug the problem? Thanks!

DELL-Josh Cr · Answer

If the switch is set to a static LAG it treats the ports as part of a LAG regardless of whether there are LACP packets showing up, so if that is not working, the issue is not with the configuration on the switch, it could be a bad port, but a cable or host issue is more likely. Do the ports work individually without a LAG?

Networking General

LACP configuration problem with 8132F and R515 (10G Intel SFP)

Was this post helpful?