Help

Start a discussion
UNSOLVED

vwijhe

updated

12 years ago

V

vwijhe

1 Rookie

•

35 Posts

0

37818

March 27th, 2015 05:00

Unexplained packets discarded

The last two weeks we had about 3 times 30 seconds of downtime caused by out switches. The problem is that a lot of packets are being discared. These packets are received on our core switch. We have configured two 6224's in a stack connected to multiple 3500's. They are connected via two cables and configured as port channel on both sides. 

The regular port channel does not show any errors:

CoreSwitch#show interfaces counters port-channel 8

Ch InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
ch8 2214803607 11553709 144015 8191


Ch OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
ch8 12432260369 13633306 1196509 655365


Alignment Errors: ............................. 0
FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Multiple Collision Frames: .................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Oversize Packets: ............................. 288931
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0

AccessSwitch# show interfaces counters port-channel 1

Ch InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
ch1 966097929 93105839 49047587 831619282606

Ch OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
ch1 862523206 12199637 1535178 211770915270

FCS Errors: 0
Single Collision Frames: 0
Late Collisions: 0
Excessive Collisions: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0

However, looking at the interfaces more specific show something different:

CoreSwitch#show interfaces counters ethernet 1/g8

Port InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
1/g8 586862096 2984293 96750 4030


Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
1/g8 6678995559 6414005 356659 246406


FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Oversize Packets: ............................. 106394
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 0
Transmit Packets Discarded: ................... 0

LANStack2#show interfaces counters ethernet 2/g8

Port InOctets InUcastPkts InMcastPkts InBcastPkts
---------------- ---------- ----------- ----------- -----------
2/g8 1652287248 8681279 47883 4190


Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
---------------- ---------- ------------ ------------ ------------
2/g8 5849332071 7335827 843155 410456


FCS Errors: ................................... 0
Single Collision Frames: ...................... 0
Late Collisions: .............................. 0
Excessive Collisions: ......................... 0
Multiple Collisions: .......................... 0
Oversize Packets: ............................. 183346
Internal MAC Rx Errors: ....................... 0
Received Pause Frames: ........................ 0
Transmitted Pause Frames: ..................... 0
Receive Packets Discarded: .................... 2167
Transmit Packets Discarded: ................... 0

Unit 2 discarded some frames! I see this behavior on all switches connected to the second unit. 

Port configuration:

CoreSwitch#show running-config interface ethernet 1/g8
channel-group 8 mode auto

CoreSwitch#show running-config interface ethernet 2/g8
channel-group 8 mode auto

CoreSwitch#show running-config interface port-channel 8
service-policy in VOIP
spanning-tree guard root
switchport mode general
switchport general allowed vlan add 8,10,12,20,120,200 tagged
switchport general allowed vlan add 1 tagged

AccessSwitch#

interface port-channel 1
switchport mode general
switchport general allowed vlan add 8,10,12,20,120,200 tagged
exit

interface ethernet g3
switchport mode general
channel-group 1 mode auto

interface ethernet g4
switchport mode general
channel-group 1 mode auto

A soon as we lost connection I rapidly connected to the switch and did a show process cpu command to see what what going on. The cpu utilisation is higher than normale but not unacceptable high. Normally it's below 10% and now the 1 min timer raised to 32%. 18% was caused by the ipMapForwardingTask, which might indicate a buffer issue? 

Any troubleshoot tips? Vlans are the same. The stranged thing is that only unit2 is discarding packets. Vlans are configured the same so that doens't explain the discards. We also notice a connection loss for a short time so that's not the issue. 

  • DJ

    DELL-Josh Cr

    Community Manager

    •

    9745 Posts

    •

    44407 Points

    4541

    0

    Posted March 27th, 2015 12:00

    Hi,

    ipMapForwardingTask is related to the IP stack, so typically when it is high there is too much traffic for the switch to keep up. Is the switch firmware up to date? Is spanning tree enabled? If it is switch 2 the spanning-tree root switch?

  • V

    vwijhe

    1 Rookie

    •

    35 Posts

    4541

    0

    Posted March 30th, 2015 01:00

    The switch is running at almost the latest firmware version (3.3.11.2) so I would not expect any fixes for this issue regarding the release notes on the latest version (3.3.12.1). De stack is the root switch, configured with the spanning-tree priority 4096 command. Spanning-tree looks normal for me. For example, channel 3 is connected to 1/g3 and 2/g3 and so on. The only exception for this rule is po1, which is bound to 1/g21 and 2/g22. These ports are not connected via port channel:

    port 1,2, 17, 19, 20, 22

    LANStack#show switch

       Management Standby   Preconfig     Plugged-in    Switch        Code

    SW  Status     Status    Model ID      Model ID      Status        Version

    --- ---------- --------- ------------- ------------- ------------- -----------

    1   Stack Mbr  Oper Stby PCT6224       PCT6224       OK            3.3.11.2

    2   Mgmt Sw              PCT6224       PCT6224       OK            3.3.11.2

    LANStack#show stack-port

                         Configured  Running

                           Stack      Stack      Link       Link

    Unit    Interface       Mode       Mode       Status     Speed (Gb/s)

    ---- ---------------- ---------- ---------- ------------ ------------

    1    xg1              Stack      Stack      Link Up      12

    1    xg2              Stack      Stack      Link Up      12

    1    xg3              Ethernet   Ethernet   Link Down    Unknown

    1    xg4              Ethernet   Ethernet   Link Down    Unknown

    2    xg1              Stack      Stack      Link Up      12

    2    xg2              Stack      Stack      Link Up      12

    2    xg3              Ethernet   Ethernet   Link Down    Unknown

    2    xg4              Ethernet   Ethernet   Link Down    Unknown

    LANStack#show spanning-tree

    Spanning tree :Enabled - BPDU Flooding :Disabled - Portfast BPDU filtering :Disabled - mode :rstp

    CST Regional Root:        10:00:D0:67:E5:75:6D:D9

    Regional Root Path Cost:  0

    ROOT ID

                 Address         D0:67:E5:75:6D:D9

                 This Switch is the Root.

                 Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec

    Interfaces

    Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

    ------  -------- --------- ----------  ---- ----- --------  -------

    1/g1    Enabled  128.1             0   DIS  Disb      No      No

    1/g2    Enabled  128.2             0   DIS  Disb      No      No

    1/g3    Enabled  128.3             0   FWD  Disb      No      No

    1/g4    Enabled  128.4             0   FWD  Disb      No      No

    Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

    ------  -------- --------- ----------  ---- ----- --------  -------

    1/g5    Enabled  128.5             0   FWD  Disb      No      No

    1/g6    Enabled  128.6             0   FWD  Disb      No      No

    1/g7    Enabled  128.7             0   FWD  Disb      No      No

    1/g8    Enabled  128.8             0   FWD  Disb      No      No

    1/g9    Enabled  128.9             0   FWD  Disb      No      No

    1/g10   Enabled  128.10            0   FWD  Disb      No      No

    1/g11   Enabled  128.11            0   FWD  Disb      No      No

    1/g12   Enabled  128.12            0   FWD  Disb      No      No

    1/g13   Enabled  128.13            0   DIS  Disb      No      No

    1/g14   Enabled  128.14            0   DIS  Disb      No      No

    1/g15   Enabled  128.15            0   DIS  Disb      No      No

    1/g16   Enabled  128.16            0   DIS  Disb      No      No

    1/g17   Enabled  128.17            0   DIS  Disb      No      No

    1/g18   Enabled  128.18            0   DIS  Disb      No      No

    1/g19   Enabled  128.19        20000   FWD  Desg      No      No

    1/g20   Enabled  128.20        20000   FWD  Desg      No      No

    Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

    ------  -------- --------- ----------  ---- ----- --------  -------

    1/g21   Enabled  128.21            0   FWD  Disb      No      No

    1/g22   Enabled  128.22        20000   FWD  Desg      No      No

    1/g23   Enabled  128.23        20000   FWD  Desg      No      No

    1/g24   Enabled  128.24            0   FWD  Disb      No      No

    1/xg1   Enabled  128.25            0   DIS  Disb      No      No

    1/xg2   Enabled  128.26            0   DIS  Disb      No      No

    1/xg3   Enabled  128.27            0   DIS  Disb      No      No

    1/xg4   Enabled  128.28            0   DIS  Disb      No      No

    2/g1    Enabled  128.53        20000   FWD  Desg      No      No

    2/g2    Enabled  128.54        20000   FWD  Desg      No      No

    2/g3    Enabled  128.55            0   FWD  Disb      No      No

    2/g4    Enabled  128.56            0   FWD  Disb      No      No

    2/g5    Enabled  128.57            0   FWD  Disb      No      No

    2/g6    Enabled  128.58            0   FWD  Disb      No      No

    2/g7    Enabled  128.59            0   FWD  Disb      No      No

    2/g8    Enabled  128.60            0   FWD  Disb      No      No

    Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

    ------  -------- --------- ----------  ---- ----- --------  -------

    2/g9    Enabled  128.61            0   FWD  Disb      No      No

    2/g10   Enabled  128.62            0   FWD  Disb      No      No

    2/g11   Enabled  128.63            0   FWD  Disb      No      No

    2/g12   Enabled  128.64            0   FWD  Disb      No      No

    2/g13   Enabled  128.65            0   DIS  Disb      No      No

    2/g14   Enabled  128.66            0   DIS  Disb      No      No

    2/g15   Enabled  128.67            0   DIS  Disb      No      No

    2/g16   Enabled  128.68            0   DIS  Disb      No      No

    2/g17   Enabled  128.69            0   DIS  Disb      No      No

    2/g18   Enabled  128.70            0   DIS  Disb      No      No

    2/g19   Enabled  128.71        20000   FWD  Desg      No      No

    2/g20   Enabled  128.72        20000   FWD  Desg      No      No

    2/g21   Enabled  128.73            0   FWD  Disb      No      No

    2/g22   Enabled  128.74            0   DIS  Disb      No      No

    2/g23   Enabled  128.75            0   DIS  Disb      No      No

    2/g24   Enabled  128.76            0   FWD  Disb      No      No

    Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

    ------  -------- --------- ----------  ---- ----- --------  -------

    2/xg1   Enabled  128.77            0   DIS  Disb      No      No

    2/xg2   Enabled  128.78            0   DIS  Disb      No      No

    2/xg3   Enabled  128.79            0   DIS  Disb      No      No

    2/xg4   Enabled  128.80            0   DIS  Disb      No      No

    ch1     Enabled   96.626       10000   FWD  Desg      No      No

    ch2     Enabled   96.627           0   DIS  Disb      No      Yes

    ch3     Enabled   96.628       10000   FWD  Desg      No      Yes

    ch4     Enabled   96.629       10000   FWD  Desg      No      Yes

    ch5     Enabled   96.630       10000   FWD  Desg      No      Yes

    ch6     Enabled   96.631       10000   FWD  Desg      No      Yes

    ch7     Enabled   96.632       10000   FWD  Desg      No      Yes

    ch8     Enabled   96.633       10000   FWD  Desg      No      Yes

    ch9     Enabled   96.634       10000   FWD  Desg      No      Yes

    ch10    Enabled   96.635       10000   FWD  Desg      No      Yes

    ch11    Enabled   96.636       10000   FWD  Desg      No      Yes

    Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

    ------  -------- --------- ----------  ---- ----- --------  -------

    ch12    Enabled   96.637       10000   FWD  Desg      No      Yes

    ch13    Enabled   96.638           0   DIS  Disb      No      Yes

    ch14    Enabled   96.639           0   DIS  Disb      No      Yes

    ch15    Enabled   96.640           0   DIS  Disb      No      No

    ch16    Enabled   96.641           0   DIS  Disb      No      No

    ch17    Enabled   96.642           0   DIS  Disb      No      No

    ch18    Enabled   96.643           0   DIS  Disb      No      No

    ch19    Enabled   96.644           0   DIS  Disb      No      No

    ch20    Enabled   96.645           0   DIS  Disb      No      No

    ch21    Enabled   96.646           0   DIS  Disb      No      No

    ch22    Enabled   96.647           0   DIS  Disb      No      No

    ch23    Enabled   96.648           0   DIS  Disb      No      No

    ch24    Enabled   96.649       10000   FWD  Desg      No      No

    ch25    Enabled   96.650           0   DIS  Disb      No      No

    ch26    Enabled   96.651           0   DIS  Disb      No      No

    ch27    Enabled   96.652           0   DIS  Disb      No      No

    Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

    ------  -------- --------- ----------  ---- ----- --------  -------

    ch28    Enabled   96.653           0   DIS  Disb      No      No

    ch29    Enabled   96.654           0   DIS  Disb      No      No

    ch30    Enabled   96.655           0   DIS  Disb      No      No

    ch31    Enabled   96.656           0   DIS  Disb      No      No

    ch32    Enabled   96.657           0   DIS  Disb      No      No

    ch33    Enabled   96.658           0   DIS  Disb      No      No

    ch34    Enabled   96.659           0   DIS  Disb      No      No

    ch35    Enabled   96.660           0   DIS  Disb      No      No

    ch36    Enabled   96.661           0   DIS  Disb      No      No

    ch37    Enabled   96.662           0   DIS  Disb      No      No

    ch38    Enabled   96.663           0   DIS  Disb      No      No

    ch39    Enabled   96.664           0   DIS  Disb      No      No

    ch40    Enabled   96.665           0   DIS  Disb      No      No

    ch41    Enabled   96.666           0   DIS  Disb      No      No

    ch42    Enabled   96.667           0   DIS  Disb      No      No

    ch43    Enabled   96.668           0   DIS  Disb      No      No

    Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted

    ------  -------- --------- ----------  ---- ----- --------  -------

    ch44    Enabled   96.669           0   DIS  Disb      No      No

    ch45    Enabled   96.670           0   DIS  Disb      No      No

    ch46    Enabled   96.671           0   DIS  Disb      No      No

    ch47    Enabled   96.672           0   DIS  Disb      No      No

    ch48    Enabled   96.673           0   DIS  Disb      No      No

  • DJ

    DELL-Josh Cr

    Community Manager

    •

    9745 Posts

    •

    44407 Points

    4541

    0

    Posted March 30th, 2015 09:00

    Do the logs show any errors during the time when the issue is occuring?

  • V

    vwijhe

    1 Rookie

    •

    35 Posts

    4541

    0

    Posted March 30th, 2015 23:00

    Good point. The logs are still in there so I've taken a look. There are some intrestring lines:

    <189> MAR 27 10:11:14 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4455 %% Spanning Tree Topology Change: 0, Unit: 1

    <189> MAR 27 10:11:18 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4456 %% Spanning Tree Topology Change: 0, Unit: 1

    <189> MAR 27 10:11:22 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4457 %% Spanning Tree Topology Change: 0, Unit: 1

    <189> MAR 27 10:11:26 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4458 %% Spanning Tree Topology Change: 0, Unit: 1

    <189> MAR 27 10:11:30 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4459 %% Spanning Tree Topology Change: 0, Unit: 1

    <189> MAR 27 10:11:34 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4460 %% Spanning Tree Topology Change: 0, Unit: 1

    <189> MAR 27 10:11:37 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4461 %% Spanning Tree Topology Change: 0, Unit: 1

    <189> MAR 27 10:11:41 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4462 %% Spanning Tree Topology Change: 0, Unit: 1

    <189> MAR 27 10:11:45 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4463 %% Spanning Tree Topology Change: 0, Unit: 1

    <189> MAR 27 10:11:49 192.168.20.1-2 TRAPMGR[124132592]: traputil.c(611) 4464 %% Spanning Tree Topology Change: 0, Unit: 1

    I see this kind of messages pretty frequently. So the problem might be something in the spannning tree topology again... I am going to do some research on this one and let you know the results. 

  • V

    vwijhe

    1 Rookie

    •

    35 Posts

    4541

    0

    Posted March 31st, 2015 00:00

    I am not sure anymore if spanning tree is the problem. A topogy change will occur even when a client turns on so these messages are very common (or is it only showing local topology changes?). Also, most ports are configured with the spanning tree root command, so STP root switch changes should be very rare or even never happen....

    All the switches are configured with RSTP.

    Because the switch is configured as stack, there is no need to configure STP on both switches? (it's even not possible I guess).

    The only strange thing I see so far is that one access switch is connected via three cables (two UTP one fiber). The fiber is not in use anymore (administrative shutdown) but to make sure it doesn't harm anything I'll remove the cable tommorow. Despite of the admin shutdown the port is still up on other side. 

  • DJ

    DELL-Josh Cr

    Community Manager

    •

    9745 Posts

    •

    44407 Points

    4541

    0

    Posted March 31st, 2015 09:00

    Try changing the root priority to something really low for spanning-tree so that it is less likely to change root.

  • V

    vwijhe

    1 Rookie

    •

    35 Posts

    4541

    0

    Posted April 7th, 2015 02:00

    Priority is already set to 4096.

Dell Support Resources

Latest Solutions

View More