UNSOLVED

WallieNgti

updated

8 years ago

W

WallieNgti

21 Posts

0

4970

May 14th, 2018 03:00

Dell P6224 STP issues

For some reason, our switches decide to change ports from forwarding state to blocking state.

 

Without clear reason, one of the switches seem to elect a new STP root, as the switch itself. Situation is causing the ports the router is connected too, to be blocked, causing all network traffic to fail.

Switches are PC 6224 and 6224P and related, running Software Version 3.3.7.3 and 3.3.17.1.

Any ideas where to start, to get this sorted out???

 

 

 
  • WallieNgti

    21 Posts

    3970

    0

    Posted May 15th, 2018 03:00

    There are 4 physical connections from the router to the switch, which are all in one channel-group.

    Original the priority of the first switch was 8192, which was the lowest of all the switches.

    Issue happened again today, changed for now the priority of the first switch to the suggested 4096, without success.

    The other switches all have higher priority, like 16384, 20480, 24576, 28672 and 36864.

    On some moments, connection between switches are being closed (1/g19 is the port sw-5 is connected to, from sw-0) on some moments one of the connections with the router are being closed, causing the whole network to breakdown.

  • WallieNgti

    21 Posts

    3752

    0

    Posted May 18th, 2018 01:00


    As router we are using openbsd, running on a Dell PowerEdge R430

    Further details:

    sw-0: stack of 2, 6224P, System Software Version 3.3.17.1
    sw-1: stack of 4 6248, System Software Version 3.3.17.1
    sw-2: stack of 2 6248P, System Software Version 3.3.17.1
    sw-3: 6224, System Software Version 3.3.17.1
    sw-4: 6224P, System Software Version 3.3.17.1
    sw-5: 6224, System Software Version 3.3.17.1


    sw-0 connects to the internet

    All other switches connect to sw-0 directly

    Connection between sw-0 and sw-4 are using sw-0 1/g20, 2/g20 and port-channel 20, sw-4 1/g23, 1/g24, port-channel 1
    sw-0 => sw-4:

    interface ethernet 1/g20
    channel-group 20 mode auto
    description "sw-4 (1/g23)"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    interface ethernet 2/g20
    channel-group 20 mode auto
    description "sw-4 (1/g24)"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    interface port-channel 20
    description "sw-4"
    switchport mode general
    switchport general allowed vlan add 28-29,192,199-207,300,304-305 tagged
    switchport general allowed vlan remove 1
    mtu 9216
    exit

    On sw-4 side:

    interface ethernet 1/g23
    channel-group 1 mode auto
    description "sw-0"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface ethernet 1/g24
    channel-group 1 mode auto
    description "sw-0"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface port-channel 1
    description "sw-0"
    switchport mode general
    switchport general allowed vlan add 10-11,28-29,192,199-207,300,304-305 tagged
    switchport general allowed vlan remove 1
    mtu 9216
    exit


    Regular connection to external is using sw-0 1/g24:

    interface ethernet 1/g24
    no negotiation
    description "Cogent"
    spanning-tree portfast
    mtu 9216
    switchport access vlan 10
    dot1x port-control force-authorized
    no lldp transmit
    no isdp enable
    power inline never
    exit


    Router is an openbsd machine, which is connected using 4 connections, sw-0 1/g1, 1/g2, 2/g1, 2/g2 and port-channel 1

    interface ethernet 1/g1
    channel-group 1 mode auto
    shutdown
    description "rt-0 em0"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface ethernet 1/g2
    channel-group 1 mode auto
    shutdown
    description "rt-0 em1"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface ethernet 2/g1
    channel-group 1 mode auto
    description "rt-0 em2"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface ethernet 2/g2
    channel-group 1 mode auto
    description "rt-0 em3"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface port-channel 1
    description "rt-0"
    switchport mode general
    switchport general allowed vlan add 10-11,28-29,172,192,199-207,300 tagged
    switchport general allowed vlan add 304-305,400-401 tagged
    switchport general allowed vlan remove 1
    mtu 9216
    exit

    At the moment two of the interfaces with the router are shutdown, as this seems to be reducing the issue for now.

    On switch sw-0, the internet connections are connected to, as well as all the other switches, router (rt-0), authentication servers and wifi access points.

    The configuration has been running for month without any major changes or issues, and recently no changes has been performed on the configuration, which could explain the current issue.
    After the issue was noticed, all the switches have been upgraded to the latest software version, without resolving the problem.

    Last time issue log messages:

    May 15 08:16:48 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 872 %% 2/0/2 status is authorized
    May 15 08:16:48 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 873 %% Link Up: 2/0/2
    May 15 08:16:48 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 874 %% 2/0/2 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:17:09 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 879 %% 2/0/1 status is authorized
    May 15 08:17:09 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 880 %% Link Up: 2/0/1
    May 15 08:17:09 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 881 %% 2/0/1 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:17:10 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 882 %% 2/0/2 is transitioned from the Learning state to the Forwarding state in instance 0
    May 15 08:17:10 192.168.200.100-1 TRAPMGR[124789904]: traputil.c(610) 883 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:10 192.168.200.101-4 TRAPMGR[126268512]: traputil.c(610) 3787 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:11 192.168.200.105-1 TRAPMGR[124790160]: traputil.c(610) 2659261 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:10 192.168.200.106-1 TRAPMGR[124790384]: traputil.c(610) 836 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:15 192.168.200.105-1 TRAPMGR[124790160]: traputil.c(610) 2659278 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:14 192.168.200.101-4 TRAPMGR[126268512]: traputil.c(610) 3788 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:14 192.168.200.106-1 TRAPMGR[124790384]: traputil.c(610) 837 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:30 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 884 %% 2/0/1 is transitioned from the Learning state to the Forwarding state in instance 0
    May 15 08:17:30 192.168.200.100-1 TRAPMGR[124789904]: traputil.c(610) 885 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:30 192.168.200.101-4 TRAPMGR[126268512]: traputil.c(610) 3789 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:31 192.168.200.105-1 TRAPMGR[124790160]: traputil.c(610) 2659347 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:30 192.168.200.106-1 TRAPMGR[124790384]: traputil.c(610) 838 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:33 192.168.200.105-1 TRAPMGR[151808944]: traputil.c(610) 2659366 %% 1/0/24 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:17:33 192.168.200.105-1 TRAPMGR[151808944]: traputil.c(610) 2659367 %% Link Up: 1/0/24
    May 15 08:17:35 192.168.200.105-1 TRAPMGR[124790160]: traputil.c(610) 2659369 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:34 192.168.200.101-4 TRAPMGR[126268512]: traputil.c(610) 3790 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:34 192.168.200.106-1 TRAPMGR[124790384]: traputil.c(610) 839 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:36 192.168.200.105-1 TRAPMGR[151808944]: traputil.c(610) 2659379 %% Link Up: 1/0/24
    May 15 08:17:36 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 886 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:17:36 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 887 %% Link Up: 2/0/20
    May 15 08:17:55 192.168.200.105-1 TRAPMGR[124790160]: traputil.c(610) 2659470 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:55 192.168.200.105-1 TRAPMGR[151808944]: traputil.c(610) 2659474 %% 1/0/24 is transitioned from the Learning state to the Forwarding state in instance 0
    May 15 08:17:55 192.168.200.100-1 TRAPMGR[124789904]: traputil.c(610) 888 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:55 192.168.200.101-4 TRAPMGR[126268512]: traputil.c(610) 3791 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:54 192.168.200.106-1 TRAPMGR[124790384]: traputil.c(610) 840 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:17:56 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 889 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:17:58 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 890 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:17:58 192.168.200.106-1 TRAPMGR[124790384]: traputil.c(610) 841 %% Spanning Tree Topology Change: 0, Unit: 1
    May 15 08:18:00 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 891 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:02 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 892 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:04 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 893 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:06 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 894 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:08 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 895 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:10 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 896 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:12 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 897 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:14 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 898 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:16 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 899 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:18 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 900 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:20 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 901 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:22 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 902 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:24 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 903 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:26 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 904 %% 2/0/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:18:27 192.168.200.105-1 TRAPMGR[151808944]: traputil.c(610) 2659618 %% Unit 1 elected as the new STP root
    May 15 08:18:27 192.168.200.105-1 TRAPMGR[151808944]: traputil.c(610) 2659619 %% Instance 0 has elected a new STP root: 7000:a4ba:db7a:8208
    May 15 08:18:26 192.168.200.100-1 TRAPMGR[151807264]: traputil.c(610) 905 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 15 08:31:30 192.168.200.101-4 TRAPMGR[126268512]: traputil.c(610) 4120 %% Spanning Tree Topology Change: 0, Unit: 1

     

     

     

  • WallieNgti

    21 Posts

    3918

    0

    Posted May 18th, 2018 01:00


    As router we are using openbsd, running on a Dell PowerEdge R430

    Further details:

    sw-0: stack of 2, 6224P, System Software Version 3.3.17.1
    sw-1: stack of 4 6248, System Software Version 3.3.17.1
    sw-2: stack of 2 6248P, System Software Version 3.3.17.1
    sw-3: 6224, System Software Version 3.3.17.1
    sw-4: 6224P, System Software Version 3.3.17.1
    sw-5: 6224, System Software Version 3.3.17.1


    sw-0 connects to the internet

    All other switches connect to sw-0 directly

    Connection between sw-0 and sw-4 are using sw-0 1/g20, 2/g20 and port-channel 20, sw-4 1/g23, 1/g24, port-channel 1
    sw-0 => sw-4:

    interface ethernet 1/g20
    channel-group 20 mode auto
    description "sw-4 (1/g23)"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    interface ethernet 2/g20
    channel-group 20 mode auto
    description "sw-4 (1/g24)"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    interface port-channel 20
    description "sw-4"
    switchport mode general
    switchport general allowed vlan add 28-29,192,199-207,300,304-305 tagged
    switchport general allowed vlan remove 1
    mtu 9216
    exit
    
    

    On sw-4 side:

    interface ethernet 1/g23
    channel-group 1 mode auto
    description "sw-0"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface ethernet 1/g24
    channel-group 1 mode auto
    description "sw-0"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface port-channel 1
    description "sw-0"
    switchport mode general
    switchport general allowed vlan add 10-11,28-29,192,199-207,300,304-305 tagged
    switchport general allowed vlan remove 1
    mtu 9216
    exit
    
    


    Regular connection to external is using sw-0 1/g24:

    interface ethernet 1/g24
    no negotiation
    description "Cogent"
    spanning-tree portfast
    mtu 9216
    switchport access vlan 10
    dot1x port-control force-authorized
    no lldp transmit
    no isdp enable
    power inline never
    exit
    
    
    

    Router is an openbsd machine, which is connected using 4 connections, sw-0 1/g1, 1/g2, 2/g1, 2/g2 and port-channel 1

    interface ethernet 1/g1
    channel-group 1 mode auto
    shutdown
    description "rt-0 em0"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface ethernet 1/g2
    channel-group 1 mode auto
    shutdown
    description "rt-0 em1"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface ethernet 2/g1
    channel-group 1 mode auto
    description "rt-0 em2"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface ethernet 2/g2
    channel-group 1 mode auto
    description "rt-0 em3"
    mtu 9216
    dot1x port-control force-authorized
    power inline never
    exit
    !
    interface port-channel 1
    description "rt-0"
    switchport mode general
    switchport general allowed vlan add 10-11,28-29,172,192,199-207,300 tagged
    switchport general allowed vlan add 304-305,400-401 tagged
    switchport general allowed vlan remove 1
    mtu 9216
    exit
    
    

    At the moment two of the interfaces with the router are shutdown, as this seems to be reducing the issue for now.

    On switch sw-0, the internet connections are connected to, as well as all the other switches, router (rt-0), authentication servers and wifi access points.

    The configuration has been running for month without any major changes or issues, and recently no changes has been performed on the configuration, which could explain the current issue.
    After the issue was noticed, all the switches have been upgraded to the latest software version, without resolving the problem.

     

  • WallieNgti

    21 Posts

    3901

    0

    Posted May 18th, 2018 06:00

    On sw-0, following BPDU are received:

    port 1/g17: BPDU: sent 4, received 3 (connection with sw-2)
    port 1/g24: BPDU: sent 137679, received 0 (internet connection)
    port 2/g5: BPDU: sent 137725, received 0 (connection with powerrail)
    port 2/g6: BPDU: sent 137736, received 0 (Access point)
    port 2/g12: BPDU: sent 137792, received 0 (camera device)
    port 2/g17: BPDU: sent 3, received 0 (connection with sw-2)
    port 2/g20: BPDU: sent 234, received 208 (connection with sw-4)
    port 2/g24: BPDU: sent 137950, received 0 (backup internet connection)
    port-channel 1: BPDU: sent 137512, received 0 (connection with router)
    port-channel 3: BPDU: sent 137986, received 0 (connection with authentication server)
    port-channel 17: BPDU: sent 138037, received 4 (connection with sw-2)
    port-channel 19: BPDU: sent 138057, received 2 (connection with sw-3)
    port-channel 20: BPDU: sent 93796, received 100 (connection with sw-4)

     

    Seen that at the moment today the issue occurred, there was a high spike on traffic on port-channel 20, incoming on sw-0, from sw-4. Was shown as 100M bits per second. Not sure how accurate that measurement is in cacti.

    Logs don't show that much, nothing before the issue, just at the moment the issue is there:

    May 18 09:04:29 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22267 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:31 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22268 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:33 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22269 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:35 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22270 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:37 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22271 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:39 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22272 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:41 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22273 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:43 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22274 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:45 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22275 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:47 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22276 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 18 09:04:49 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22277 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0

    General spanning tree settings on the switches are set in the config as below: (can't post the full config, due to max character counts)

    spanning-tree portfast bpdufilter default
    spanning-tree bpdu-protection
    spanning-tree priority 4096

    Appart from the priority count, all switches are configured regarding this the same.

  • WallieNgti

    21 Posts

    3787

    0

    Posted May 28th, 2018 06:00

    Took a bit of time the issue occurred again. This time I was able to disable communication with the failing switch, without shutting it down, resulting in having access to the logs on the  switch still...

    Noticed on my syslog server the following (filtered all the messages from all switches):

    May 28 07:23:50 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1075 %% Unit 1 elected as the new STP root
    May 28 07:23:49 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22681 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:51 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22682 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:53 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22683 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:55 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22684 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:57 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22685 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:59 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22686 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:24:01 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22687 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:24:03 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22688 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0

    Before the new STP root election noting mentioned in here.

    From sw-4, got following log messages:

    <190> MAY 28 02:15:56 192.168.200.105-1 UNKN[152778208]: sntp_client.c(1769) 1063 %% SNTP: system clock synchronized on MON MAY 28 02:15:56 2018 UTC. Indicates that SNTP has successfully synchronized the time of the box with the server.
    <190> MAY 28 05:23:40 192.168.200.105-1 UNKN[152778208]: sntp_client.c(1769) 1064 %% SNTP: system clock synchronized on MON MAY 28 05:23:40 2018 UTC. Indicates that SNTP has successfully synchronized the time of the box with the server.
    <189> MAY 28 06:56:53 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1065 %% 1/0/17 status is Unauthorized
    <189> MAY 28 06:56:53 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1066 %% Link Up: 1/0/17
    <189> MAY 28 06:56:56 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1067 %% 1/0/17 status is Unauthorized
    <189> MAY 28 06:56:56 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1068 %% Link Down: 1/0/17
    <189> MAY 28 06:56:56 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1069 %% Link on 1/0/17 is failed
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1070 %% 1/0/17 status is Unauthorized
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1071 %% Link Up: 1/0/17
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1072 %% 1/0/17 status is authorized
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1073 %% 1/0/17 is transitioned from the Forwarding state to the Blocking state in instance 0
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1074 %% 1/0/17 is transitioned from the Learning state to the Forwarding state in instance 0
    <189> MAY 28 07:23:50 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1075 %% Unit 1 elected as the new STP root
    <189> MAY 28 07:23:50 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1076 %% Instance 0 has elected a new STP root: 7000:a4ba:db7a:8208
    <189> MAY 28 07:25:09 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1077 %% 1/0/23 is transitioned from the Forwarding state to the Blocking state in instance 0
    <189> MAY 28 07:25:09 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1078 %% Link Up: 1/0/23
    <189> MAY 28 07:25:09 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1079 %% 0/3/1 is transitioned from the Forwarding state to the Blocking state in instance 0
    <190> MAY 28 07:25:09 192.168.200.105-1 UNKN[98662720]: osapi_ipeak.c(2126) 1080 %% ipstkArpFlush: sending delete failed
    <189> MAY 28 07:25:12 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1081 %% Link Up: 1/0/23
    <189> MAY 28 07:25:31 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1082 %% 1/0/23 is transitioned from the Learning state to the Forwarding state in instance 0
    <189> MAY 28 07:25:31 192.168.200.105-1 TRAPMGR[124790192]: traputil.c(610) 1083 %% Spanning Tree Topology Change: 0, Unit: 1
    <189> MAY 28 08:30:25 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1084 %% Instance 0 has elected a new STP root: 1000:d067:e5a9:76ef
    <189> MAY 28 08:30:28 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1085 %% 1/0/17 status is Unauthorized
    <189> MAY 28 08:30:28 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1086 %% 1/0/17 status is Unauthorized
    <189> MAY 28 08:30:28 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1087 %% Link Down: 1/0/17

    Around 07:42, I did perform a shutdown on sw-0 1/g20, causing the rest of the network to keep working properly. In the logs on sw-4, nothing is shown around this time.

    On sw-, following information regarding BPDU:

    port 1/g1: BPDU: sent 30, received 0
    port 1/g5: BPDU: sent 1492, received 0
    port 1/g6: BPDU: sent 11219, received 0
    port 1/g7: BPDU: sent 1379, received 0
    port 1/g10: BPDU: sent 138582, received 0
    port 1/g11: BPDU: sent 66, received 0
    port 1/g17: BPDU: sent 4050, received 0
    port-channel 1: BPDU: sent 66, received 425637 (connected with sw-0)

     

  • WallieNgti

    21 Posts

    881

    0

    Posted May 30th, 2018 02:00

    Took a bit of time the issue occurred again. This time I was able to disable communication with the failing switch, without shutting it down, resulting in having access to the logs on the switch still...

    Noticed on my syslog server the following (filtered all the messages from all switches):

    May 28 07:23:50 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1075 %% Unit 1 elected as the new STP root
    May 28 07:23:49 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22681 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:51 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22682 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:53 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22683 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:55 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22684 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:57 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22685 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:59 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22686 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:24:01 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22687 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:24:03 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22688 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0

    Before the new STP root election noting mentioned in here.

    From sw-4, got following log messages:

    <190> MAY 28 02:15:56 192.168.200.105-1 UNKN[152778208]: sntp_client.c(1769) 1063 %% SNTP: system clock synchronized on MON MAY 28 02:15:56 2018 UTC. Indicates that SNTP has successfully synchronized the time of the box with the server.
    <190> MAY 28 05:23:40 192.168.200.105-1 UNKN[152778208]: sntp_client.c(1769) 1064 %% SNTP: system clock synchronized on MON MAY 28 05:23:40 2018 UTC. Indicates that SNTP has successfully synchronized the time of the box with the server.
    <189> MAY 28 06:56:53 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1065 %% 1/0/17 status is Unauthorized
    <189> MAY 28 06:56:53 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1066 %% Link Up: 1/0/17
    <189> MAY 28 06:56:56 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1067 %% 1/0/17 status is Unauthorized
    <189> MAY 28 06:56:56 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1068 %% Link Down: 1/0/17
    <189> MAY 28 06:56:56 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1069 %% Link on 1/0/17 is failed
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1070 %% 1/0/17 status is Unauthorized
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1071 %% Link Up: 1/0/17
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1072 %% 1/0/17 status is authorized
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1073 %% 1/0/17 is transitioned from the Forwarding state to the Blocking state in instance 0
    <189> MAY 28 06:56:59 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1074 %% 1/0/17 is transitioned from the Learning state to the Forwarding state in instance 0
    <189> MAY 28 07:23:50 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1075 %% Unit 1 elected as the new STP root
    <189> MAY 28 07:23:50 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1076 %% Instance 0 has elected a new STP root: 7000:a4ba:db7a:8208
    <189> MAY 28 07:25:09 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1077 %% 1/0/23 is transitioned from the Forwarding state to the Blocking state in instance 0
    <189> MAY 28 07:25:09 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1078 %% Link Up: 1/0/23
    <189> MAY 28 07:25:09 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1079 %% 0/3/1 is transitioned from the Forwarding state to the Blocking state in instance 0
    <190> MAY 28 07:25:09 192.168.200.105-1 UNKN[98662720]: osapi_ipeak.c(2126) 1080 %% ipstkArpFlush: sending delete failed
    <189> MAY 28 07:25:12 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1081 %% Link Up: 1/0/23
    <189> MAY 28 07:25:31 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1082 %% 1/0/23 is transitioned from the Learning state to the Forwarding state in instance 0
    <189> MAY 28 07:25:31 192.168.200.105-1 TRAPMGR[124790192]: traputil.c(610) 1083 %% Spanning Tree Topology Change: 0, Unit: 1
    <189> MAY 28 08:30:25 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1084 %% Instance 0 has elected a new STP root: 1000:d067:e5a9:76ef
    <189> MAY 28 08:30:28 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1085 %% 1/0/17 status is Unauthorized
    <189> MAY 28 08:30:28 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1086 %% 1/0/17 status is Unauthorized
    <189> MAY 28 08:30:28 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1087 %% Link Down: 1/0/17

    Around 07:42, I did perform a shutdown on sw-0 1/g20, causing the rest of the network to keep working properly. In the logs on sw-4, nothing is shown around this time.

    Regarding the BPDU, only received are found on port-channel 1: 

    BPDU: sent 103, received 472365

    All the others all have zero received.

     

     

     

  • WallieNgti

    21 Posts

    879

    0

    Posted May 30th, 2018 02:00

    Took a bit of time the issue occurred again. This time I was able to disable communication with the failing switch, without shutting it down, resulting in having access to the logs on the switch still...

    Noticed on my syslog server the following (filtered all the messages from all switches):

    May 28 07:23:50 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1075 %% Unit 1 elected as the new STP root
    May 28 07:23:49 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22681 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0
    May 28 07:23:51 192.168.200.100-1 TRAPMGR[151837152]: traputil.c(610) 22682 %% 0/3/20 is transitioned from the Forwarding state to the Blocking state in instance 0

    Before the new STP root election noting mentioned in here.

    From sw-4, got following log messages:

    <189> MAY 28 07:23:50 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1075 %% Unit 1 elected as the new STP root
    <189> MAY 28 07:23:50 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1076 %% Instance 0 has elected a new STP root: 7000:a4ba:db7a:8208
    <189> MAY 28 07:25:09 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1077 %% 1/0/23 is transitioned from the Forwarding state to the Blocking state in instance 0
    <189> MAY 28 07:25:09 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1078 %% Link Up: 1/0/23
    <189> MAY 28 07:25:09 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1079 %% 0/3/1 is transitioned from the Forwarding state to the Blocking state in instance 0
    <190> MAY 28 07:25:09 192.168.200.105-1 UNKN[98662720]: osapi_ipeak.c(2126) 1080 %% ipstkArpFlush: sending delete failed
    <189> MAY 28 07:25:12 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1081 %% Link Up: 1/0/23
    <189> MAY 28 07:25:31 192.168.200.105-1 TRAPMGR[151554592]: traputil.c(610) 1082 %% 1/0/23 is transitioned from the Learning state to the Forwarding state in instance 0
    <189> MAY 28 07:25:31 192.168.200.105-1 TRAPMGR[124790192]: traputil.c(610) 1083 %% Spanning Tree Topology Change: 0, Unit: 1

    Around 07:42, 

    I did perform a shutdown on sw-0 1/g20, causing the rest of the network to keep working properly. In the logs on sw-4, nothing is shown around this time.

    Regarding the BPDU, only received are found on port-channel 1:

    BPDU: sent 103, received 472365

    All the others all have zero received.

     

  • WallieNgti

    21 Posts

    885

    0

    Posted May 30th, 2018 03:00

    One more try, after all the others seems to be marked as **bleep** and disappear into the black hole..

     

    Happened again, all to be seen in the syslog server is a message that sw-4 elected a new STP root.

    Afterh this (with a time stamp of 1 second earlier???) the message the portchannel transiioned from the forwarding state to the blocking state...

    Before the new STP root election noting mentioned in here. Only thing shown in the logs of sw-4 is only the election of the new STP root, and related. Only seeing one message regarding ipstkArpFlush: sending delete failed

    Regarding the BPDU, only received are found on port-channel 1:

    BPDU: sent 103, received 472365

    All the others all have zero received.

     

     

  • WallieNgti

    21 Posts

    866

    0

    Posted May 31st, 2018 00:00

    Daniel,

    Those addresses are the CST Regional Root addresses mentioned on the spanning tree command on sw-0 and sw-4.

    sw-0#show spanning-tree 
    Spanning tree :Enabled - BPDU Flooding :Disabled - Portfast BPDU filtering :Enabled - mode :rstp
    CST Regional Root:        10:00:D0:67:E5:A9:76:EF
    Regional Root Path Cost:  0
    ROOT ID
                  Address         D0:67:E5:A9:76:EF
                  This Switch is the Root. 
                  Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec
    Interfaces
    
     Name    State   Prio.Nbr    Cost       Sts  Role PortFast  Restricted
    ------  -------- --------- ----------  ---- ----- --------  -------
     1/g1    Enabled  128.1             0   DIS  Disb      No      No  
     1/g2    Enabled  128.2             0   DIS  Disb      No      No  
     1/g3    Enabled  128.3             0   DIS  Disb      No      No  
     1/g4    Enabled  128.4             0   DIS  Disb      No      No  

    and for sw-4:

    sw-4#show spanning-tree 
    Spanning tree :Enabled - BPDU Flooding :Disabled - Portfast BPDU filtering :Enabled - mode :rstp
    CST Regional Root:        70:00:A4:BA:DB:7A:82:08
    Regional Root Path Cost:  0
    ROOT ID
                  Address         D0:67:E5:A9:76:EF
                  Path Cost       20000
                  Root Port      ch1    
                  Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec
    Bridge ID
                  Priority        28672
                  Address         A4:BA:DB:7A:82:08
                  Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec