Skip to main content
Start a Conversation

Unsolved

This post is more than 5 years old

vwijhe

35 Posts

37164

March 27th, 2015 05:00

Unexplained packets discarded

The last two weeks we had about 3 times 30 seconds of downtime caused by out switches. The problem is that a lot of packets are being discared. These packets are received on our core switch. We have configured two 6224's in a stack connected to multiple 3500's. They are connected via two cables and configured as port channel on both sides. 

The regular port channel does not show any errors:

CoreSwitch#show interfaces counters port-channel 8

Ch InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
ch8 2214803607 11553709 144015 8191


Ch OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
ch8 12432260369 13633306 1196509 655365


Alignment Errors: ............................. 0
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Multiple Collision Frames: .................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Oversize Packets: ............................. 288931
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0

AccessSwitch# show interfaces counters port-channel 1

Ch InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
ch1 966097929 93105839 49047587 831619282606

Ch OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
ch1 862523206 12199637 1535178 211770915270

FCS Errors: 0
Single Collision Frames: 0
Late Collisions: 0
Excessive Collisions: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0

However, looking at the interfaces more specific show something different:

CoreSwitch#show interfaces counters ethernet 1/g8

Port InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
1/g8 586862096 2984293 96750 4030


Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
1/g8 6678995559 6414005 356659 246406


FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Oversize Packets: ............................. 106394
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 0
Transmit Packets Discarded: ................... 0

LANStack2#show interfaces counters ethernet 2/g8

Port InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
2/g8 1652287248 8681279 47883 4190


Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
2/g8 5849332071 7335827 843155 410456


FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Oversize Packets: ............................. 183346
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 2167
Transmit Packets Discarded: ................... 0

Unit 2 discarded some frames! I see this behavior on all switches connected to the second unit. 

Port configuration:

CoreSwitch#show running-config interface ethernet 1/g8
channel-group 8 mode auto

CoreSwitch#show running-config interface ethernet 2/g8
channel-group 8 mode auto

CoreSwitch#show running-config interface port-channel 8
service-policy in VOIP
spanning-tree guard root
switchport mode general
switchport general allowed vlan add 8,10,12,20,120,200 tagged
switchport general allowed vlan add 1 tagged

AccessSwitch#

interface port-channel 1
switchport mode general
switchport general allowed vlan add 8,10,12,20,120,200 tagged
exit

interface ethernet g3
switchport mode general
channel-group 1 mode auto

interface ethernet g4
switchport mode general
channel-group 1 mode auto

A soon as we lost connection I rapidly connected to the switch and did a show process cpu command to see what what going on. The cpu utilisation is higher than normale but not unacceptable high. Normally it's below 10% and now the 1 min timer raised to 32%. 18% was caused by the ipMapForwardingTask, which might indicate a buffer issue? 

Any troubleshoot tips? Vlans are the same. The stranged thing is that only unit2 is discarding packets. Vlans are configured the same so that doens't explain the discards. We also notice a connection loss for a short time so that's not the issue. 

DELL-Josh Cr

Moderator

 • 

8.5K Posts

March 27th, 2015 12:00

Hi,

ipMapForwardingTask is related to the IP stack, so typically when it is high there is too much traffic for the switch to keep up. Is the switch firmware up to date? Is spanning tree enabled? If it is switch 2 the spanning-tree root switch?

vwijhe

35 Posts

March 30th, 2015 01:00

The switch is running at almost the latest firmware version (3.3.11.2) so I would not expect any fixes for this issue regarding the release notes on the latest version (3.3.12.1). De stack is the root switch, configured with the spanning-tree priority 4096 command. Spanning-tree looks normal for me. For example, channel 3 is connected to 1/g3 and 2/g3 and so on. The only exception for this rule is po1, which is bound to 1/g21 and 2/g22. These ports are not connected via port channel:

port 1,2, 17, 19, 20, 22

LANStack#show switch

   Management Standby   Preconfig     Plugged-in    Switch        Code

SW  Status     Status    Model ID      Model ID      Status        Version

--- ---------- --------- ------------- ------------- ------------- -----------

1   Stack Mbr  Oper Stby PCT6224       PCT6224       OK            3.3.11.2

2   Mgmt Sw              PCT6224       PCT6224       OK            3.3.11.2

LANStack#show stack-port

                     Configured  Running

                       Stack      Stack      Link       Link

Unit    Interface       Mode       Mode       Status     Speed (Gb/s)

---- ---------------- ---------- ---------- ------------ ------------

1    xg1              Stack      Stack      Link Up      12

1    xg2              Stack      Stack      Link Up      12

1    xg3              Ethernet   Ethernet   Link Down    Unknown

1    xg4              Ethernet   Ethernet   Link Down    Unknown

2    xg1              Stack      Stack      Link Up      12

2    xg2              Stack      Stack      Link Up      12

2    xg3              Ethernet   Ethernet   Link Down    Unknown

2    xg4              Ethernet   Ethernet   Link Down    Unknown

LANStack#show spanning-tree

Spanning tree :Enabled - BPDU Flooding :Disabled - Portfast BPDU filtering :Disabled - mode :rstp

CST Regional Root:        10:00:D0:67:E5:75:6D:D9

Regional Root Path Cost:  0

ROOT ID

             Address         D0:67:E5:75:6D:D9

             This Switch is the Root.

             Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec

Interfaces

Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

------  -------- --------- ----------  ---- ----- --------  -------

1/g1    Enabled  128.1             0   DIS  Disb      No      No

1/g2    Enabled  128.2             0   DIS  Disb      No      No

1/g3    Enabled  128.3             0   FWD  Disb      No      No

1/g4    Enabled  128.4             0   FWD  Disb      No      No

Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

------  -------- --------- ----------  ---- ----- --------  -------

1/g5    Enabled  128.5             0   FWD  Disb      No      No

1/g6    Enabled  128.6             0   FWD  Disb      No      No

1/g7    Enabled  128.7             0   FWD  Disb      No      No

1/g8    Enabled  128.8             0   FWD  Disb      No      No

1/g9    Enabled  128.9             0   FWD  Disb      No      No

1/g10   Enabled  128.10            0   FWD  Disb      No      No

1/g11   Enabled  128.11            0   FWD  Disb      No      No

1/g12   Enabled  128.12            0   FWD  Disb      No      No

1/g13   Enabled  128.13            0   DIS  Disb      No      No

1/g14   Enabled  128.14            0   DIS  Disb      No      No

1/g15   Enabled  128.15            0   DIS  Disb      No      No

1/g16   Enabled  128.16            0   DIS  Disb      No      No

1/g17   Enabled  128.17            0   DIS  Disb      No      No

1/g18   Enabled  128.18            0   DIS  Disb      No      No

1/g19   Enabled  128.19        20000   FWD  Desg      No      No

1/g20   Enabled  128.20        20000   FWD  Desg      No      No

Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

------  -------- --------- ----------  ---- ----- --------  -------

1/g21   Enabled  128.21            0   FWD  Disb      No      No

1/g22   Enabled  128.22        20000   FWD  Desg      No      No

1/g23   Enabled  128.23        20000   FWD  Desg      No      No

1/g24   Enabled  128.24            0   FWD  Disb      No      No

1/xg1   Enabled  128.25            0   DIS  Disb      No      No

1/xg2   Enabled  128.26            0   DIS  Disb      No      No

1/xg3   Enabled  128.27            0   DIS  Disb      No      No

1/xg4   Enabled  128.28            0   DIS  Disb      No      No

2/g1    Enabled  128.53        20000   FWD  Desg      No      No

2/g2    Enabled  128.54        20000   FWD  Desg      No      No

2/g3    Enabled  128.55            0   FWD  Disb      No      No

2/g4    Enabled  128.56            0   FWD  Disb      No      No

2/g5    Enabled  128.57            0   FWD  Disb      No      No

2/g6    Enabled  128.58            0   FWD  Disb      No      No

2/g7    Enabled  128.59            0   FWD  Disb      No      No

2/g8    Enabled  128.60            0   FWD  Disb      No      No

Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

------  -------- --------- ----------  ---- ----- --------  -------

2/g9    Enabled  128.61            0   FWD  Disb      No      No

2/g10   Enabled  128.62            0   FWD  Disb      No      No

2/g11   Enabled  128.63            0   FWD  Disb      No      No

2/g12   Enabled  128.64            0   FWD  Disb      No      No

2/g13   Enabled  128.65            0   DIS  Disb      No      No

2/g14   Enabled  128.66            0   DIS  Disb      No      No

2/g15   Enabled  128.67            0   DIS  Disb      No      No

2/g16   Enabled  128.68            0   DIS  Disb      No      No

2/g17   Enabled  128.69            0   DIS  Disb      No      No

2/g18   Enabled  128.70            0   DIS  Disb      No      No

2/g19   Enabled  128.71        20000   FWD  Desg      No      No

2/g20   Enabled  128.72        20000   FWD  Desg      No      No

2/g21   Enabled  128.73            0   FWD  Disb      No      No

2/g22   Enabled  128.74            0   DIS  Disb      No      No

2/g23   Enabled  128.75            0   DIS  Disb      No      No

2/g24   Enabled  128.76            0   FWD  Disb      No      No

Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

------  -------- --------- ----------  ---- ----- --------  -------

2/xg1   Enabled  128.77            0   DIS  Disb      No      No

2/xg2   Enabled  128.78            0   DIS  Disb      No      No

2/xg3   Enabled  128.79            0   DIS  Disb      No      No

2/xg4   Enabled  128.80            0   DIS  Disb      No      No

ch1     Enabled   96.626       10000   FWD  Desg      No      No

ch2     Enabled   96.627           0   DIS  Disb      No      Yes

ch3     Enabled   96.628       10000   FWD  Desg      No      Yes

ch4     Enabled   96.629       10000   FWD  Desg      No      Yes

ch5     Enabled   96.630       10000   FWD  Desg      No      Yes

ch6     Enabled   96.631       10000   FWD  Desg      No      Yes

ch7     Enabled   96.632       10000   FWD  Desg      No      Yes

ch8     Enabled   96.633       10000   FWD  Desg      No      Yes

ch9     Enabled   96.634       10000   FWD  Desg      No      Yes

ch10    Enabled   96.635       10000   FWD  Desg      No      Yes

ch11    Enabled   96.636       10000   FWD  Desg      No      Yes

Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

------  -------- --------- ----------  ---- ----- --------  -------

ch12    Enabled   96.637       10000   FWD  Desg      No      Yes

ch13    Enabled   96.638           0   DIS  Disb      No      Yes

ch14    Enabled   96.639           0   DIS  Disb      No      Yes

ch15    Enabled   96.640           0   DIS  Disb      No      No

ch16    Enabled   96.641           0   DIS  Disb      No      No

ch17    Enabled   96.642           0   DIS  Disb      No      No

ch18    Enabled   96.643           0   DIS  Disb      No      No

ch19    Enabled   96.644           0   DIS  Disb      No      No

ch20    Enabled   96.645           0   DIS  Disb      No      No

ch21    Enabled   96.646           0   DIS  Disb      No      No

ch22    Enabled   96.647           0   DIS  Disb      No      No

ch23    Enabled   96.648           0   DIS  Disb      No      No

ch24    Enabled   96.649       10000   FWD  Desg      No      No

ch25    Enabled   96.650           0   DIS  Disb      No      No

ch26    Enabled   96.651           0   DIS  Disb      No      No

ch27    Enabled   96.652           0   DIS  Disb      No      No

Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

------  -------- --------- ----------  ---- ----- --------  -------

ch28    Enabled   96.653           0   DIS  Disb      No      No

ch29    Enabled   96.654           0   DIS  Disb      No      No

ch30    Enabled   96.655           0   DIS  Disb      No      No

ch31    Enabled   96.656           0   DIS  Disb      No      No

ch32    Enabled   96.657           0   DIS  Disb      No      No

ch33    Enabled   96.658           0   DIS  Disb      No      No

ch34    Enabled   96.659           0   DIS  Disb      No      No

ch35    Enabled   96.660           0   DIS  Disb      No      No

ch36    Enabled   96.661           0   DIS  Disb      No      No

ch37    Enabled   96.662           0   DIS  Disb      No      No

ch38    Enabled   96.663           0   DIS  Disb      No      No

ch39    Enabled   96.664           0   DIS  Disb      No      No

ch40    Enabled   96.665           0   DIS  Disb      No      No

ch41    Enabled   96.666           0   DIS  Disb      No      No

ch42    Enabled   96.667           0   DIS  Disb      No      No

ch43    Enabled   96.668           0   DIS  Disb      No      No

Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

------  -------- --------- ----------  ---- ----- --------  -------

ch44    Enabled   96.669           0   DIS  Disb      No      No

ch45    Enabled   96.670           0   DIS  Disb      No      No

ch46    Enabled   96.671           0   DIS  Disb      No      No

ch47    Enabled   96.672           0   DIS  Disb      No      No

ch48    Enabled   96.673           0   DIS  Disb      No      No

DELL-Josh Cr

Moderator

 • 

8.5K Posts

March 30th, 2015 09:00

Do the logs show any errors during the time when the issue is occuring?

vwijhe

35 Posts

March 30th, 2015 23:00

Good point. The logs are still in there so I've taken a look. There are some intrestring lines:

<189> MAR 27 10:11:14 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4455 %% Spanning Tree Topology Change: 0, Unit: 1

<189> MAR 27 10:11:18 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4456 %% Spanning Tree Topology Change: 0, Unit: 1

<189> MAR 27 10:11:22 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4457 %% Spanning Tree Topology Change: 0, Unit: 1

<189> MAR 27 10:11:26 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4458 %% Spanning Tree Topology Change: 0, Unit: 1

<189> MAR 27 10:11:30 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4459 %% Spanning Tree Topology Change: 0, Unit: 1

<189> MAR 27 10:11:34 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4460 %% Spanning Tree Topology Change: 0, Unit: 1

<189> MAR 27 10:11:37 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4461 %% Spanning Tree Topology Change: 0, Unit: 1

<189> MAR 27 10:11:41 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4462 %% Spanning Tree Topology Change: 0, Unit: 1

<189> MAR 27 10:11:45 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4463 %% Spanning Tree Topology Change: 0, Unit: 1

<189> MAR 27 10:11:49 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4464 %% Spanning Tree Topology Change: 0, Unit: 1

I see this kind of messages pretty frequently. So the problem might be something in the spannning tree topology again... I am going to do some research on this one and let you know the results. 

vwijhe

35 Posts

March 31st, 2015 00:00

I am not sure anymore if spanning tree is the problem. A topogy change will occur even when a client turns on so these messages are very common (or is it only showing local topology changes?). Also, most ports are configured with the spanning tree root command, so STP root switch changes should be very rare or even never happen....

All the switches are configured with RSTP.

Because the switch is configured as stack, there is no need to configure STP on both switches? (it's even not possible I guess).

The only strange thing I see so far is that one access switch is connected via three cables (two UTP one fiber). The fiber is not in use anymore (administrative shutdown) but to make sure it doesn't harm anything I'll remove the cable tommorow. Despite of the admin shutdown the port is still up on other side. 

DELL-Josh Cr

Moderator

 • 

8.5K Posts

March 31st, 2015 09:00

Try changing the root priority to something really low for spanning-tree so that it is less likely to change root.

vwijhe

35 Posts

April 7th, 2015 02:00

Priority is already set to 4096.

View More

View All

No Events found!

Top