5548P Switch port going down causing RSTP issues

Question

Hi,

We have several 3448P, 3548P and 5548P devices connected utilizing RSTP (Long).

We just recently added the 5548P switches and since then noticed that our logs show fowarding and blocking messages when an interswitch connection on one of our 5548P devices transitions from an UP to DOWN state.

We have configured all the switch's interconnection ports to disable portfast.

When looking through the RAM logs on all the switches it appears that the DOWN/UP occurs FIRST on the offending switch which triggers the RSTP occurrances.

We are scheduling changing out the cabling between switches but haven't been granted an outage yet.

Is there any way to toubleshoot this further? I have turned logging to debug but it doesn't show me any reason for the port cycling.

Ideas?

DELL-Willy M · Answer

If a port is transitioning from UP to DOWN you should see spanning tree messages in the logs showing the transition from forwarding to blocking. When the port goes down it is setting the spanning tree protocol into convergence.

Have you isolated the UP/DOWN to a single port connection?

What is physically connected to this port?

Do you have portfast configure for all the end user PCs? Then not configured on the switch to switch connections.

Moby16 · Answer

Yes we always see the ports that interconnect two 5548P switches affected. The one 5548P port goes down which takes down the other port it is directly connected to. We have portfast configured for all the non switch (PC) connected ports - all switch interconnected ports are set to portfast disabled.

The one thing that puzzles me also is on the switch that the first port goes down the other ports that are connected to the other switches transition to blocking/forwarding even though portfast is disabled on their ports.

Here is the log:

12 2147483557 03~Apr~2013 11:45:32 Warning %STP-W-PORTSTATUS: gi1/0/45: STP status Forwarding

13 2147483558 03~Apr~2013 11:45:32 Warning %STP-W-PORTSTATUS: gi1/0/46: STP status Forwarding

14 2147483559 03~Apr~2013 11:45:32 Warning %STP-W-PORTSTATUS: gi1/0/47: STP status Forwarding

15 2147483560 03~Apr~2013 11:45:32 Warning %STP-W-PORTSTATUS: gi1/0/46: STP status Blocking

16 2147483561 03~Apr~2013 11:45:32 Warning %STP-W-PORTSTATUS: gi1/0/45: STP status Blocking

17 2147483562 03~Apr~2013 11:45:30 Info %LINK-I-Up: gi1/0/47

18 2147483563 03~Apr~2013 11:45:27 Warning %LINK-W-Down: gi1/0/47

gi1/0/47 connects to another 5548P switch.

gi1/0/45 and gi1/0/46 connect to 3448P and 3548P switch

The whole cascade starts (according to the time stamps with gi1/0/47) or occasionally the port on the switch connected to this port.

DELL-Willy M · Answer

Do you have the current firmware installed on the different switches? I would plan on updating them when you have your scheduled down time.

If you have the same configurations on the other ports connecting switch to switch and they are not going up and down. I would want to confirm that we have good cabling on that connection. Along with the current firmware installed.

If it is at all possible to change the physical port that you are using for this connection to see if you get the same reaction or if it is resolved.

1. Firmware updated

2. Try known good cables

3. Try know good port

Firmware downloads:

3348 - v2.0.0.34 - www.dell.com/.../powerconnect-3448

3548 - v2.0.0.51,A08 - www.dell.com/.../powerconnect-3548

5548 - v4.1.0.8,A06 - www.dell.com/.../powerconnect-5548

Moby16 · Answer

That's what we've planned. I've uploaded all the current firmware to our 3548 and 3448 switches. It looks like the 5548's are up to date. We are also planning to replace the switch interconnect cables and use a different port between floors to eliminate this as a potential root cause.

Unfortunately we're in a month end freeze so can't schedule anything for a week. I'll update when we get a chance to do the recabling.

Moby16 · Answer

I have one more question.... Is setting the STP port settings on the switch for PC connected ports to AUTO sufficient or shoulld they all be set to ENABLED?

Moby16 · Answer

Sorry...that should be two more questions.....

Is a MITEL 5224 IP Phone (where you can actually plug your PC into the back to get ethernet) considered a switch?

If so we have a lot of work to figure out which ports have phones and therefore sudo-switches. And if we turn off portfast for the phone connect ports and someone moves their phone we could ave a LOT of management issues.

DELL-Willy M · Answer

1. spanning tree is enable by default and is best practice to leave it that way. I'm not aware of an "auto" option in the spanning tree commands. Are you speaking about the speed negotiation set to auto possibly? Portfast can be set on the end user ports where a PC can be removed and added frequently. Portfast helps smooth the transition when a port is going up and down.

2. The phones are considered a type of switch. That is the reason for setting up a trunk connection with the switch. Some people get colored cables that are only used for the phones and the rest for data. That can help with the confusion when moving phones and PCs around.

If you have PC or phones being moved around on one of the switches (and not being configured properly) it could be a cause of the link going down between the switches.

Moby16 · Answer

I see thast this is a legacy setting for the older switch image. There is an option in STP settings to set the port to ENABLE/AUTO/DISABLE. This is no longer available on the newer switch image. (It's now a check box labelled Enable Fastport)

We have set the VLANs for the switches to:

switchport trunk allowed vlan add 2 (Voice) on all the ports except where the master switch is.

I'll upgrade the images and switch cables when we get a window and update this thread with the results.

Before the 5548P switches were added everything was stable. That's why I'm pursuing the root cause.

Moby16 · Answer

I am still waiting for a maintenance window to allow me to upgrade the switch firmware but in the meantime something puzzles me.

In the logs on switch 65 we are seeing STP blocking, forwarding on ports (45 and 46) connected to other switches (switch 69 and switch 68) when one of the interconnected ports (47) goes down but on the switch that the interconnection is also going down on (switch 62) there is no blocking, forwarding on either port 21 or 47.

The ports that seem to keep going down are 47 on switch 65 and 46 on switch 62. I'm trying to understand why those other switches ports (21 and 47) don't also go into blocking/forwarding if switch 62's port goes down as well.

Moby16 · Answer

Follow up:

Now that you can see the topology here is the log file from switch 65. This ONLY appears on switch 65. Switch 62 which is directly connected only shows port 46 going up and down.

2 2147483559 15~Apr~2013 17:54:29 Warning %STP-W-PORTSTATUS: gi1/0/45: STP status Forwarding

3 2147483560 15~Apr~2013 17:54:29 Warning %STP-W-PORTSTATUS: gi1/0/46: STP status Forwarding

4 2147483561 15~Apr~2013 17:54:29 Warning %STP-W-PORTSTATUS: gi1/0/47: STP status Forwarding

5 2147483562 15~Apr~2013 17:54:29 Warning %STP-W-PORTSTATUS: gi1/0/46: STP status Blocking

6 2147483563 15~Apr~2013 17:54:29 Warning %STP-W-PORTSTATUS: gi1/0/45: STP status Blocking

7 2147483564 15~Apr~2013 17:54:27 Info %LINK-I-Up: gi1/0/47

8 2147483565 15~Apr~2013 17:54:25 Warning %LINK-W-Down: gi1/0/47

9 2147483566 15~Apr~2013 17:00:58 Warning %SNMP-W-SNMPAUTHFAIL: Access attempted by unauthorized NMS

10 2147483569 13~Apr~2013 21:28:48 Warning %STP-W-PORTSTATUS: gi1/0/45: STP status Forwarding

11 2147483570 13~Apr~2013 21:28:48 Warning %STP-W-PORTSTATUS: gi1/0/46: STP status Forwarding

12 2147483571 13~Apr~2013 21:28:48 Warning %STP-W-PORTSTATUS: gi1/0/47: STP status Forwarding

13 2147483572 13~Apr~2013 21:28:48 Warning %STP-W-PORTSTATUS: gi1/0/46: STP status Blocking

14 2147483573 13~Apr~2013 21:28:48 Warning %STP-W-PORTSTATUS: gi1/0/45: STP status Blocking

15 2147483574 13~Apr~2013 21:28:45 Info %LINK-I-Up: gi1/0/47

16 2147483575 13~Apr~2013 21:28:42 Warning %LINK-W-Down: gi1/0/47

17 2147483576 13~Apr~2013 09:40:52 Warning %STP-W-PORTSTATUS: gi1/0/45: STP status Forwarding

18 2147483577 13~Apr~2013 09:40:52 Warning %STP-W-PORTSTATUS: gi1/0/46: STP status Forwarding

19 2147483578 13~Apr~2013 09:40:52 Warning %STP-W-PORTSTATUS: gi1/0/47: STP status Forwarding

20 2147483579 13~Apr~2013 09:40:52 Warning %STP-W-PORTSTATUS: gi1/0/46: STP status Blocking

21 2147483580 13~Apr~2013 09:40:52 Warning %STP-W-PORTSTATUS: gi1/0/45: STP status Blocking

22 2147483581 13~Apr~2013 09:40:51 Info %LINK-I-Up: gi1/0/47

23 2147483582 13~Apr~2013 09:40:48 Warning %LINK-W-Down: gi1/0/47

Moby16 · Answer

Thanks. The switches are a floor apart interconnected via an interconnection cable run. We already have a spare floor to floor ports set aside as well as new data cables but we're still waiting for a maintenance window. I was just musing about the reasons for our issues while we waited.

I think I get why you say to set portfast enabled but we have set all the interconnected switch ports to portfast disabled as was recommended. I guess we could temporarily set it to enabled while we wait for the window.

By the way....This posting is related to my other posting about trying to find the native MAC addresses on our switches as those odd MAC addresses keep showing up too.

Moby16 · Answer

I have scanned my local area network six ways to Sunday with a litany of scanners and can't find those MAC addresses anywhere but they keep showing up on the switch's trunk ports. The fact that they are SO close to the existing switch's MAC addresses makes me wonder.

For instance one is d0:67:e5:yy:xx:88 and the other d0:67:e5:yy:xx:86 ; whereas xx and yy in this case is the same octet on each switch.(and we've never had any other 5548P switches touch this network other than the ones we have currently installed - and the ones installed are d0:67:e5:yy:xx:59 and d0:67:e5:yy:xx:c3

Hopefully we will be granted our window tomorrow night so we can upgrade all the firmware, change the cables and change the runs between floors. If that doesn't fix it perhaps we have a flakey switch or port on the switch?

I want to keep the ports on the switch the same after the outage so we can determine this.

Moby16 · Answer

OK...I figured it out! [The MAC address part, anyway)

With LanSweeper I was able to scan my switches and the main MAC address of the switch is assigned to the "internal interface" and the VLANS. Starting from port one the MAC addresses assigned to each port increments up IN HEX from there.

So for instance from my example if my main MAC address for my switch was listed as:

d0:67:e5:yy:xx:19 the ports, starting at port 1, would be d0:67:e5:yy:xx:1a, port 2 d0:67:e5:yy:xx:1b, etc.

So....my interconnected ports are 45 and 47 physical ports up from the first port [59H] therefore: (86 hex - 59 hex= 2d hex = 45 dec) and (88 hex - 59 hex= 2f hex = 47dec) [don't forget this is hex not decimal]

Which is exactly right - my interconnect ports are 46 and 48 on this switch (remember to add one for starting at port 1).

Mystery solved.

See the diagram in the thread earlier

It may take a couple of reads of this entry to get your head around this. ;-)

Now on to the root cause of the switch port issue.

Moby16 · Answer

OK...I figured it out! [The MAC address part, anyway)

With LanSweeper I was able to scan my switches and the main MAC address of the switch is assigned to the "internal interface" and the VLANS. Starting from port one the MAC addresses assigned to each port increments up IN HEX from there.

So for instance from my example if my main MAC address for my switch was listed as:

d0:67:e5:yy:xx:19 the ports, starting at port 1, would be d0:67:e5:yy:xx:1a, port 2 d0:67:e5:yy:xx:1b, etc.

So....my interconnected ports are 45 and 47 physical ports up from the first port [59H] therefore: (86 hex - 59 hex= 2d hex = 45 dec) and (88 hex - 59 hex= 2f hex = 47dec) [don't forget this is hex not decimal]

Which is exactly right - my interconnect ports are 46 and 48 on this switch (remember to add one for starting at port 1).

See the diagram earlier in this thread.

Mystery solved.

It may take a couple of reads of this entry to get your head around this. ;-)

Now on to the root cause of my port up/down issue.

Moby16 · Answer

Now that I have the MAC address and port assignment issue straightened out it brings up an interesting fact that is revealed in my spreadsheet that lists all the MAC addresses on my network as outputted from each switch.

The 5548P switches see the Internal (perhaps native VLAN) MAC address AND the Uplink MAC address of the 3448P and 3548P switches but the 3448P and 3548P switches do not see the internal (perhaps native VLAN) MAC address of the 5548P switches. All but switch 69. It ONLY sees switch 65's Uplink MAC address. Let's look at the diagram again:

I read on another thread this:

Just one comment - native (untagged) vlans on trunks are the common source of misconfigurations, loops and also serious security issue (vlan hopping attack).

Thus for switch-to-switch interconnections I'd strongly recommend using 'switchport mode trunk' which on 62xx enforces tagging for all vlans and disallows the user to configure any untagged vlan.

Here's my output from the interconnected switches:

2 Floor (Switch 62)# show interfaces switchport gigabitethernet gi1/0/47

Gathering information...

Name: gi1/0/47 [to Cisco Switch]
Switchport: enable
Administrative Mode: trunk
Operational Mode: up
Access Mode VLAN: 1
Access Multicast TV VLAN: none
Trunking Native Mode VLAN: 1
Trunking VLANs Enabled: 1-3
4-4094 (Inactive)
General PVID: 1
General VLANs Enabled: none
General Egress Tagged VLANs Enabled: none
General Forbidden VLANs: none
General Ingress Filtering: enabled
General Acceptable Frame Type: all
General GVRP status: disabled
Customer Mode VLAN: none
Private-vlan promiscuous-association primary VLAN: none
Private-vlan promiscuous-association Secondary VLANs Enabled: none
Private-vlan host-association primary VLAN: none

Private-vlan host-association Secondary VLAN Enabled: none
DVA: disable

Classification rules:

Classification type Group ID VLAN ID
------------------- -------- -------

2 Floor# show interfaces switchport gi1/0/46 [to Dell switch 65]
Gathering information...

Name: gi1/0/46
Switchport: enable
Administrative Mode: trunk
Operational Mode: up
Access Mode VLAN: 1
Access Multicast TV VLAN: none
Trunking Native Mode VLAN: 1
Trunking VLANs Enabled: 1-3
4-4094 (Inactive)
General PVID: 1
General VLANs Enabled: none
General Egress Tagged VLANs Enabled: none
General Forbidden VLANs: none
General Ingress Filtering: enabled
General Acceptable Frame Type: all
General GVRP status: disabled
Customer Mode VLAN: none
Private-vlan promiscuous-association primary VLAN: none
Private-vlan promiscuous-association Secondary VLANs Enabled: none
Private-vlan host-association primary VLAN: none
Private-vlan host-association Secondary VLAN Enabled: none
DVA: disable

Classification rules:

Classification type Group ID VLAN ID
------------------- -------- -------

3rd Floor# show interfaces switchport gi1/0/47

Gathering information...

Name: gi1/0/47 [to Dell switch 62]
Switchport: enable
Administrative Mode: trunk
Operational Mode: up
Access Mode VLAN: 1
Access Multicast TV VLAN: none
Trunking Native Mode VLAN: 1
Trunking VLANs Enabled: 1-3
4-4094 (Inactive)
General PVID: 1
General VLANs Enabled: none
General Egress Tagged VLANs Enabled: none
General Forbidden VLANs: none
General Ingress Filtering: enabled
General Acceptable Frame Type: all
General GVRP status: disabled
Customer Mode VLAN: none
Private-vlan promiscuous-association primary VLAN: none
Private-vlan promiscuous-association Secondary VLANs Enabled: none
Private-vlan host-association primary VLAN: none
[0mMore: , Quit: q or CTRL+Z, One line:

Private-vlan host-association Secondary VLAN Enabled: none
DVA: disable

Classification rules:

Classification type Group ID VLAN ID
------------------- -------- -------

3rd Floor# exit

DellSwitches70# show interfaces switchport ethernet 1/e2

Port : 1/e2 [to Dell switch 62]
Port Mode: Trunk
Gvrp Status: disabled
Ingress Filtering: true
Acceptable Frame Type: admitAll
Ingress UnTagged VLAN ( NATIVE ): 1

Port is member in:

Vlan               Name               Egress rule Port Membership Type
---- -------------------------------- ----------- --------------------
1                  1                  Untagged          System
2                  2                   Tagged           Static
3               Wireless               Tagged           Static

Forbidden VLANS:

Vlan Name
---- --------------------------------

DellSwitches70# exit

2 Floor# show interfaces switchport gi1/0/21
Gathering information...

Name: gi1/0/21 [to Dell switch 70]
Switchport: enable
Administrative Mode: trunk
Operational Mode: up
Access Mode VLAN: 1
Access Multicast TV VLAN: none
Trunking Native Mode VLAN: 1
Trunking VLANs Enabled: 1-3
4-4094 (Inactive)
General PVID: 1
General VLANs Enabled: none
General Egress Tagged VLANs Enabled: none
General Forbidden VLANs: none
General Ingress Filtering: enabled
General Acceptable Frame Type: all
General GVRP status: disabled
Customer Mode VLAN: none
Private-vlan promiscuous-association primary VLAN: none
Private-vlan promiscuous-association Secondary VLANs Enabled: none
Private-vlan host-association primary VLAN: none
Private-vlan host-association Secondary VLAN Enabled: none
DVA: disable

Classification rules:

Classification type Group ID VLAN ID
------------------- -------- -------

DellSwitches69# show interfaces switchport ethernet 1/e48
Port : 1/e48 [to Dell switch 65]
Port Mode: Trunk
Gvrp Status: disabled
Ingress Filtering: true
Acceptable Frame Type: admitAll
Ingress UnTagged VLAN ( NATIVE ): 1

Port is member in:

Vlan               Name               Egress rule Port Membership Type
---- -------------------------------- ----------- --------------------
1                  1                  Untagged          System
2                  2                   Tagged           Static
3               Wireless               Tagged           Static

Forbidden VLANS:

Vlan Name
---- --------------------------------

DellSwitches69#

Can anyone spot anything that might suggest a problem??

Networking General

5548P Switch port going down causing RSTP issues

Was this post helpful?