Unsolved
This post is more than 5 years old
35 Posts
0
37313
Unexplained packets discarded
The last two weeks we had about 3 times 30 seconds of downtime caused by out switches. The problem is that a lot of packets are being discared. These packets are received on our core switch. We have configured two 6224's in a stack connected to multiple 3500's. They are connected via two cables and configured as port channel on both sides.
The regular port channel does not show any errors:
CoreSwitch#show interfaces counters port-channel 8
Ch InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
ch8 2214803607 11553709 144015 8191
Ch OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
ch8 12432260369 13633306 1196509 655365
Alignment Errors: ............................. 0
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Multiple Collision Frames: .................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Oversize Packets: ............................. 288931
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
AccessSwitch# show interfaces counters port-channel 1
Ch InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
ch1 966097929 93105839 49047587 831619282606
Ch OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
ch1 862523206 12199637 1535178 211770915270
FCS Errors: 0
Single Collision Frames: 0
Late Collisions: 0
Excessive Collisions: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0
However, looking at the interfaces more specific show something different:
CoreSwitch#show interfaces counters ethernet 1/g8
Port InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
1/g8 586862096 2984293 96750 4030
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
1/g8 6678995559 6414005 356659 246406
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Oversize Packets: ............................. 106394
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 0
Transmit Packets Discarded: ................... 0
LANStack2#show interfaces counters ethernet 2/g8
Port InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
2/g8 1652287248 8681279 47883 4190
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
2/g8 5849332071 7335827 843155 410456
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Oversize Packets: ............................. 183346
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 2167
Transmit Packets Discarded: ................... 0
Unit 2 discarded some frames! I see this behavior on all switches connected to the second unit.
Port configuration:
CoreSwitch#show running-config interface ethernet 1/g8
channel-group 8 mode auto
CoreSwitch#show running-config interface ethernet 2/g8
channel-group 8 mode auto
CoreSwitch#show running-config interface port-channel 8
service-policy in VOIP
spanning-tree guard root
switchport mode general
switchport general allowed vlan add 8,10,12,20,120,200 tagged
switchport general allowed vlan add 1 tagged
AccessSwitch#
interface port-channel 1
switchport mode general
switchport general allowed vlan add 8,10,12,20,120,200 tagged
exit
interface ethernet g3
switchport mode general
channel-group 1 mode auto
interface ethernet g4
switchport mode general
channel-group 1 mode auto
A soon as we lost connection I rapidly connected to the switch and did a show process cpu command to see what what going on. The cpu utilisation is higher than normale but not unacceptable high. Normally it's below 10% and now the 1 min timer raised to 32%. 18% was caused by the ipMapForwardingTask, which might indicate a buffer issue?
Any troubleshoot tips? Vlans are the same. The stranged thing is that only unit2 is discarding packets. Vlans are configured the same so that doens't explain the discards. We also notice a connection loss for a short time so that's not the issue.
DELL-Josh Cr
Moderator
Moderator
•
8.7K Posts
0
March 27th, 2015 12:00
Hi,
ipMapForwardingTask is related to the IP stack, so typically when it is high there is too much traffic for the switch to keep up. Is the switch firmware up to date? Is spanning tree enabled? If it is switch 2 the spanning-tree root switch?
vwijhe
35 Posts
0
March 30th, 2015 01:00
The switch is running at almost the latest firmware version (3.3.11.2) so I would not expect any fixes for this issue regarding the release notes on the latest version (3.3.12.1). De stack is the root switch, configured with the spanning-tree priority 4096 command. Spanning-tree looks normal for me. For example, channel 3 is connected to 1/g3 and 2/g3 and so on. The only exception for this rule is po1, which is bound to 1/g21 and 2/g22. These ports are not connected via port channel:
port 1,2, 17, 19, 20, 22
LANStack#show switch
Management Standby Preconfig Plugged-in Switch Code
SW Status Status Model ID Model ID Status Version
--- ---------- --------- ------------- ------------- ------------- -----------
1 Stack Mbr Oper Stby PCT6224 PCT6224 OK 3.3.11.2
2 Mgmt Sw PCT6224 PCT6224 OK 3.3.11.2
LANStack#show stack-port
Configured Running
Stack Stack Link Link
Unit Interface Mode Mode Status Speed (Gb/s)
---- ---------------- ---------- ---------- ------------ ------------
1 xg1 Stack Stack Link Up 12
1 xg2 Stack Stack Link Up 12
1 xg3 Ethernet Ethernet Link Down Unknown
1 xg4 Ethernet Ethernet Link Down Unknown
2 xg1 Stack Stack Link Up 12
2 xg2 Stack Stack Link Up 12
2 xg3 Ethernet Ethernet Link Down Unknown
2 xg4 Ethernet Ethernet Link Down Unknown
LANStack#show spanning-tree
Spanning tree :Enabled - BPDU Flooding :Disabled - Portfast BPDU filtering :Disabled - mode :rstp
CST Regional Root: 10:00:D0:67:E5:75:6D:D9
Regional Root Path Cost: 0
ROOT ID
Address D0:67:E5:75:6D:D9
This Switch is the Root.
Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec
Interfaces
Name State Prio.Nbr Cost Sts Role PortFast Restricted
------ -------- --------- ---------- ---- ----- -------- -------
1/g1 Enabled 128.1 0 DIS Disb No No
1/g2 Enabled 128.2 0 DIS Disb No No
1/g3 Enabled 128.3 0 FWD Disb No No
1/g4 Enabled 128.4 0 FWD Disb No No
Name State Prio.Nbr Cost Sts Role PortFast Restricted
------ -------- --------- ---------- ---- ----- -------- -------
1/g5 Enabled 128.5 0 FWD Disb No No
1/g6 Enabled 128.6 0 FWD Disb No No
1/g7 Enabled 128.7 0 FWD Disb No No
1/g8 Enabled 128.8 0 FWD Disb No No
1/g9 Enabled 128.9 0 FWD Disb No No
1/g10 Enabled 128.10 0 FWD Disb No No
1/g11 Enabled 128.11 0 FWD Disb No No
1/g12 Enabled 128.12 0 FWD Disb No No
1/g13 Enabled 128.13 0 DIS Disb No No
1/g14 Enabled 128.14 0 DIS Disb No No
1/g15 Enabled 128.15 0 DIS Disb No No
1/g16 Enabled 128.16 0 DIS Disb No No
1/g17 Enabled 128.17 0 DIS Disb No No
1/g18 Enabled 128.18 0 DIS Disb No No
1/g19 Enabled 128.19 20000 FWD Desg No No
1/g20 Enabled 128.20 20000 FWD Desg No No
Name State Prio.Nbr Cost Sts Role PortFast Restricted
------ -------- --------- ---------- ---- ----- -------- -------
1/g21 Enabled 128.21 0 FWD Disb No No
1/g22 Enabled 128.22 20000 FWD Desg No No
1/g23 Enabled 128.23 20000 FWD Desg No No
1/g24 Enabled 128.24 0 FWD Disb No No
1/xg1 Enabled 128.25 0 DIS Disb No No
1/xg2 Enabled 128.26 0 DIS Disb No No
1/xg3 Enabled 128.27 0 DIS Disb No No
1/xg4 Enabled 128.28 0 DIS Disb No No
2/g1 Enabled 128.53 20000 FWD Desg No No
2/g2 Enabled 128.54 20000 FWD Desg No No
2/g3 Enabled 128.55 0 FWD Disb No No
2/g4 Enabled 128.56 0 FWD Disb No No
2/g5 Enabled 128.57 0 FWD Disb No No
2/g6 Enabled 128.58 0 FWD Disb No No
2/g7 Enabled 128.59 0 FWD Disb No No
2/g8 Enabled 128.60 0 FWD Disb No No
Name State Prio.Nbr Cost Sts Role PortFast Restricted
------ -------- --------- ---------- ---- ----- -------- -------
2/g9 Enabled 128.61 0 FWD Disb No No
2/g10 Enabled 128.62 0 FWD Disb No No
2/g11 Enabled 128.63 0 FWD Disb No No
2/g12 Enabled 128.64 0 FWD Disb No No
2/g13 Enabled 128.65 0 DIS Disb No No
2/g14 Enabled 128.66 0 DIS Disb No No
2/g15 Enabled 128.67 0 DIS Disb No No
2/g16 Enabled 128.68 0 DIS Disb No No
2/g17 Enabled 128.69 0 DIS Disb No No
2/g18 Enabled 128.70 0 DIS Disb No No
2/g19 Enabled 128.71 20000 FWD Desg No No
2/g20 Enabled 128.72 20000 FWD Desg No No
2/g21 Enabled 128.73 0 FWD Disb No No
2/g22 Enabled 128.74 0 DIS Disb No No
2/g23 Enabled 128.75 0 DIS Disb No No
2/g24 Enabled 128.76 0 FWD Disb No No
Name State Prio.Nbr Cost Sts Role PortFast Restricted
------ -------- --------- ---------- ---- ----- -------- -------
2/xg1 Enabled 128.77 0 DIS Disb No No
2/xg2 Enabled 128.78 0 DIS Disb No No
2/xg3 Enabled 128.79 0 DIS Disb No No
2/xg4 Enabled 128.80 0 DIS Disb No No
ch1 Enabled 96.626 10000 FWD Desg No No
ch2 Enabled 96.627 0 DIS Disb No Yes
ch3 Enabled 96.628 10000 FWD Desg No Yes
ch4 Enabled 96.629 10000 FWD Desg No Yes
ch5 Enabled 96.630 10000 FWD Desg No Yes
ch6 Enabled 96.631 10000 FWD Desg No Yes
ch7 Enabled 96.632 10000 FWD Desg No Yes
ch8 Enabled 96.633 10000 FWD Desg No Yes
ch9 Enabled 96.634 10000 FWD Desg No Yes
ch10 Enabled 96.635 10000 FWD Desg No Yes
ch11 Enabled 96.636 10000 FWD Desg No Yes
Name State Prio.Nbr Cost Sts Role PortFast Restricted
------ -------- --------- ---------- ---- ----- -------- -------
ch12 Enabled 96.637 10000 FWD Desg No Yes
ch13 Enabled 96.638 0 DIS Disb No Yes
ch14 Enabled 96.639 0 DIS Disb No Yes
ch15 Enabled 96.640 0 DIS Disb No No
ch16 Enabled 96.641 0 DIS Disb No No
ch17 Enabled 96.642 0 DIS Disb No No
ch18 Enabled 96.643 0 DIS Disb No No
ch19 Enabled 96.644 0 DIS Disb No No
ch20 Enabled 96.645 0 DIS Disb No No
ch21 Enabled 96.646 0 DIS Disb No No
ch22 Enabled 96.647 0 DIS Disb No No
ch23 Enabled 96.648 0 DIS Disb No No
ch24 Enabled 96.649 10000 FWD Desg No No
ch25 Enabled 96.650 0 DIS Disb No No
ch26 Enabled 96.651 0 DIS Disb No No
ch27 Enabled 96.652 0 DIS Disb No No
Name State Prio.Nbr Cost Sts Role PortFast Restricted
------ -------- --------- ---------- ---- ----- -------- -------
ch28 Enabled 96.653 0 DIS Disb No No
ch29 Enabled 96.654 0 DIS Disb No No
ch30 Enabled 96.655 0 DIS Disb No No
ch31 Enabled 96.656 0 DIS Disb No No
ch32 Enabled 96.657 0 DIS Disb No No
ch33 Enabled 96.658 0 DIS Disb No No
ch34 Enabled 96.659 0 DIS Disb No No
ch35 Enabled 96.660 0 DIS Disb No No
ch36 Enabled 96.661 0 DIS Disb No No
ch37 Enabled 96.662 0 DIS Disb No No
ch38 Enabled 96.663 0 DIS Disb No No
ch39 Enabled 96.664 0 DIS Disb No No
ch40 Enabled 96.665 0 DIS Disb No No
ch41 Enabled 96.666 0 DIS Disb No No
ch42 Enabled 96.667 0 DIS Disb No No
ch43 Enabled 96.668 0 DIS Disb No No
Name State Prio.Nbr Cost Sts Role PortFast Restricted
------ -------- --------- ---------- ---- ----- -------- -------
ch44 Enabled 96.669 0 DIS Disb No No
ch45 Enabled 96.670 0 DIS Disb No No
ch46 Enabled 96.671 0 DIS Disb No No
ch47 Enabled 96.672 0 DIS Disb No No
ch48 Enabled 96.673 0 DIS Disb No No
DELL-Josh Cr
Moderator
Moderator
•
8.7K Posts
0
March 30th, 2015 09:00
Do the logs show any errors during the time when the issue is occuring?
vwijhe
35 Posts
0
March 30th, 2015 23:00
Good point. The logs are still in there so I've taken a look. There are some intrestring lines:
<189> MAR 27 10:11:14 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4455 %% Spanning Tree Topology Change: 0, Unit: 1
<189> MAR 27 10:11:18 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4456 %% Spanning Tree Topology Change: 0, Unit: 1
<189> MAR 27 10:11:22 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4457 %% Spanning Tree Topology Change: 0, Unit: 1
<189> MAR 27 10:11:26 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4458 %% Spanning Tree Topology Change: 0, Unit: 1
<189> MAR 27 10:11:30 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4459 %% Spanning Tree Topology Change: 0, Unit: 1
<189> MAR 27 10:11:34 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4460 %% Spanning Tree Topology Change: 0, Unit: 1
<189> MAR 27 10:11:37 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4461 %% Spanning Tree Topology Change: 0, Unit: 1
<189> MAR 27 10:11:41 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4462 %% Spanning Tree Topology Change: 0, Unit: 1
<189> MAR 27 10:11:45 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4463 %% Spanning Tree Topology Change: 0, Unit: 1
<189> MAR 27 10:11:49 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4464 %% Spanning Tree Topology Change: 0, Unit: 1
I see this kind of messages pretty frequently. So the problem might be something in the spannning tree topology again... I am going to do some research on this one and let you know the results.
vwijhe
35 Posts
0
March 31st, 2015 00:00
I am not sure anymore if spanning tree is the problem. A topogy change will occur even when a client turns on so these messages are very common (or is it only showing local topology changes?). Also, most ports are configured with the spanning tree root command, so STP root switch changes should be very rare or even never happen....
All the switches are configured with RSTP.
Because the switch is configured as stack, there is no need to configure STP on both switches? (it's even not possible I guess).
The only strange thing I see so far is that one access switch is connected via three cables (two UTP one fiber). The fiber is not in use anymore (administrative shutdown) but to make sure it doesn't harm anything I'll remove the cable tommorow. Despite of the admin shutdown the port is still up on other side.
DELL-Josh Cr
Moderator
Moderator
•
8.7K Posts
0
March 31st, 2015 09:00
Try changing the root priority to something really low for spanning-tree so that it is less likely to change root.
vwijhe
35 Posts
0
April 7th, 2015 02:00
Priority is already set to 4096.