6 Posts
0
1780
LACP LAG on S4048 survives reloads but not disabled ports
Hello, I'm not great with switches, particularly Dell switches, so please bear with me.
At one end I have a XenServer with a bonded pair of NIC’s. The bond mode is set to LACP. Each of the NICs in the bond connects to a different S4048 switch, ports Te 1/48 on each, and those two switch ports are configured in a LACP LAG.
Both of the S4048 switches connect via ports Te 1/1 to the rest of our network via a single N4032 switch.
When setting up the LAG on the S4048's, I followed the setup instructions carefully. Everything seemed to go well at first, as when I reloaded each of them independently I lost no connectivity to my XenServer. Next I disabled port Te 1/1 on the first S4048 and lost no connectivity. However, when I disabled port Te 1/1 on the second S4048 I lost connectivity to my XenServer.
I initially thought I had mis-configured that second S4048 switch, but after I reloaded it, then disabled port Te 1/1 again to test, the same issue didn't reoccur. It was only when I then came to disable the port on the first S4048 again, which originally didn’t cause me any connectivity issues, that I realised the switch exhibiting/causing the connectivity issue just changes from one to other between reloads.
I've checked over the config a few times and can't see that I've missed anything obvious. I can see both S4048's sending and receiving PDU's every second (is it supposed to be this frequent?)
Is there anything on the N4032 which I may have missed here?
The problem I'm having seems oddly specific, so I hoped someone might have come across it before. However, either my google-fu is particularly weak or I'm just being unlucky, as I can't seem to find examples of others with the same problem.
I'll share some config details and logs below with certain bits redacted. To my novice eyes I can’t see anything wrong here but hopefully you helpful lot can show me where I’ve made a mistake, or maybe offer some pointers. If any additional info is needed please let me know.
I would really appreciate any assistance with this. Thanks in advance.
TheRealBecks
3 Posts
0
March 1st, 2019 02:00
I'm a litte bit confused as you're having a server using LACP with two physical switches, but both switches are independent. What's missing here is virtual link trunking (VLT) between both switches, so the layer 2 connection via LACP gets redundant. See the manual here: https://downloads.dell.com/manuals/all-products/esuprt_ser_stor_net/esuprt_networking/esuprt_net_fxd_prt_swtchs/force10-s4048-on_concept-guide6_en-us.pdf on site 1055.
That's what I'm seeing here as the initial problem.
guymatthews
6 Posts
0
February 28th, 2019 07:00
SWITCH1#show lacp 1 sys-id
Actor System ID: Priority 1, Address e4f0.04b1.a7dd
Partner System ID: Priority 2, Address e4f0.04b1.addd
SWITCH1#show lacp 1 counters
-----------------------------------------------------------------------------
LACP PDU Marker PDU Unknown Illegal
Port Xmit Recv Xmit Recv Rx Rx
-----------------------------------------------------------------------------
Te 1/48 15589 15624 0 0 0 0
SWITCH1#show interfaces port-channel 1
Port-channel 1 is up, line protocol is up
Created by LACP protocol
Hardware address is xx:xx:xx:xx:xx:xx, Current address is xx:xx:xx:xx:xx:xx
Interface index is 1258291712
Minimum number of links to bring Port-channel up is 1
Internet address is not set
Mode of IPv4 Address Assignment : NONE
DHCP Client-ID :e4f004b1a7df
MTU 1554 bytes, IP MTU 1500 bytes
LineSpeed 10000 Mbit
Members in this channel: Te 1/48(U)
ARP type: ARPA, ARP Timeout 04:00:00
Queueing strategy: fifo
Input Statistics:
16397 packets, 2064384 bytes
538 64-byte pkts, 0 over 64-byte pkts, 15859 over 127-byte pkts
0 over 255-byte pkts, 0 over 511-byte pkts, 0 over 1023-byte pkts
16397 Multicasts, 0 Broadcasts, 0 Unicasts
0 runts, 0 giants, 0 throttles
0 CRC, 0 overrun, 0 discarded
Output Statistics:
16358 packets, 2059392 bytes, 0 underruns
538 64-byte pkts, 0 over 64-byte pkts, 15820 over 127-byte pkts
0 over 255-byte pkts, 0 over 511-byte pkts, 0 over 1023-byte pkts
16358 Multicasts, 0 Broadcasts, 0 Unicasts
0 throttles, 0 discarded, 0 collisions, 0 wreddrops
Rate info (interval 299 seconds):
Input 00.00 Mbits/sec, 1 packets/sec, 0.00% of line-rate
Output 00.00 Mbits/sec, 1 packets/sec, 0.00% of line-rate
Time since last interface status change: 04:27:29
guymatthews
6 Posts
0
February 28th, 2019 09:00
System Logs covering the period where I disable Te 1/1 on the switch, losing connectivity
Feb 28 13:30:49 :
Sending PDU on TenGigabitEthernet 1/48 :
Feb 28 13:30:49 : Partner System Priority: 2, System: e4f004b1addd
Partner Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:49 : Actor System Priority: 1, System: e4f004b1a7dd
Actor Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:48 :
Received PDU on TenGigabitEthernet 1/48 :
Feb 28 13:30:48 : lacpProcessIfmOperUpMsg: Port OPER up, u4Ifindex 200004, port 2097156
Feb 28 13:30:48 : lacpProcessIfmLspeedChgMsg: Port speed change 10000, u4Ifindex 200004, port Te 1/1
Feb 28 13:30:48 %STKUNIT1-M:CP %IFMGR-5-OSTATE_UP: Changed interface state to up: Te 1/1
Feb 28 13:30:48 : Partner System Priority: 1, System: e4f004b1a7dd
Partner Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:48 : Actor System Priority: 2, System: e4f004b1addd
Actor Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:48 :
Sending PDU on TenGigabitEthernet 1/48 :
Feb 28 13:30:48 : Partner System Priority: 2, System: e4f004b1addd
Partner Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:47 : Actor System Priority: 1, System: e4f004b1a7dd
Actor Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:47 :
Received PDU on TenGigabitEthernet 1/48 :
Feb 28 13:30:47 : Partner System Priority: 1, System: e4f004b1a7dd
Partner Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:47 : Actor System Priority: 2, System: e4f004b1addd
Actor Key: 1, Port: 249, Priority: 32768, State: 3f
removed some entries here to keep the post under 20k chars, it was just more of the same PDU send/receives every second in between
Feb 28 13:30:18 :
Received PDU on TenGigabitEthernet 1/48 :
Feb 28 13:30:18 : Partner System Priority: 1, System: e4f004b1a7dd
Partner Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:18 : Actor System Priority: 2, System: e4f004b1addd
Actor Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:18 :
Sending PDU on TenGigabitEthernet 1/48 :
Feb 28 13:30:18 : lacpProcessIfmOperDnMsg: Fetched PortEntry for 200004
Feb 28 13:30:18 : lacpProcessIfmOperDnMsg: Port OPER down, u4Ifindex 200004, port 2097156
Feb 28 13:30:17 %STKUNIT1-M:CP %IFMGR-5-OSTATE_DN: Changed interface state to down: Te 1/1
Feb 28 13:30:17 %STKUNIT1-M:CP %IFMGR-5-ASTATE_DN: Changed interface Admin state to down: Te 1/1
Feb 28 13:30:17 : Partner System Priority: 1, System: e4f004b1a7dd
Partner Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:17 : Actor System Priority: 2, System: e4f004b1addd
Actor Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:17 :
Sending PDU on TenGigabitEthernet 1/48 :
Feb 28 13:30:17 : Partner System Priority: 2, System: e4f004b1addd
Partner Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:17 : Actor System Priority: 1, System: e4f004b1a7dd
Actor Key: 1, Port: 249, Priority: 32768, State: 3f
Feb 28 13:30:17 :
Received PDU on TenGigabitEthernet 1/48 :
Feb 28 13:30:16 : Partner System Priority: 2, System: e4f004b1addd
Partner Key: 1, Port: 249, Priority: 32768, State: 3f
guymatthews
6 Posts
0
February 28th, 2019 09:00
S048 Switch Config (management IP and hostname aside the config is the same across both switches)
Current Configuration ...
! Version 9.14(0.0)
! Last configuration change at Thu Feb 28 12:10:36 2019 by admin
! Startup-config last updated at Thu Feb 28 11:27:51 2019 by admin
!
boot system stack-unit 1 primary system://B
boot system stack-unit 1 secondary system://A
!
logging coredump stack-unit 1
logging coredump stack-unit 2
logging coredump stack-unit 3
logging coredump stack-unit 4
logging coredump stack-unit 5
logging coredump stack-unit 6
!
hostname SWITCH1
!
protocol lldp
!
redundancy auto-synchronize full
!
enable password
lacp system-priority 1
!
stack-unit 1 provision S4048T-ON
!
interface TenGigabitEthernet 1/1
no ip address
switchport
no shutdown
!
interface TenGigabitEthernet 1/2
no ip address
switchport
no shutdown
!
interface TenGigabitEthernet 1/3
no ip address
switchport
no shutdown
!
interface TenGigabitEthernet 1/4
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/5
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/6
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/7
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/8
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/9
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/10
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/11
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/12
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/13
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/14
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/15
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/16
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/17
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/18
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/19
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/20
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/21
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/22
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/23
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/24
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/25
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/26
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/27
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/28
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/29
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/30
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/31
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/32
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/33
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/34
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/35
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/36
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/37
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/38
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/39
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/40
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/41
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/42
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/43
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/44
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/45
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/46
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/47
no ip address
switchport
shutdown
!
interface TenGigabitEthernet 1/48
no ip address
!
port-channel-protocol LACP
port-channel 1 mode active
no shutdown
!
interface fortyGigE 1/49
no ip address
shutdown
!
interface fortyGigE 1/50
no ip address
shutdown
!
interface fortyGigE 1/51
no ip address
shutdown
!
interface fortyGigE 1/52
no ip address
shutdown
!
interface fortyGigE 1/53
no ip address
shutdown
!
interface fortyGigE 1/54
no ip address
shutdown
!
interface ManagementEthernet 1/1
ip address 10.0.1.204/24
no shutdown
!
interface ManagementEthernet 2/1
shutdown
!
interface ManagementEthernet 3/1
shutdown
!
interface ManagementEthernet 4/1
shutdown
!
interface ManagementEthernet 5/1
shutdown
!
interface ManagementEthernet 6/1
shutdown
!
interface Port-channel 1
no ip address
no shutdown
!
interface Vlan 1
!untagged TenGigabitEthernet 1/1-1/47
!
management route 0.0.0.0/0 10.0.1.254
!
logging 10.1.10.58
!
ntp server 10.1.10.5
!
ip ssh server enable
!
line console 0
line vty 0
line vty 1
line vty 2
line vty 3
line vty 4
line vty 5
line vty 6
line vty 7
line vty 8
line vty 9
!
http-server secure-http
!
reload-type
boot-type normal-reload
config-scr-download enable
!
end
guymatthews
6 Posts
0
March 5th, 2019 09:00
Thanks for the input. This helped me resolve this issue