PowerConnect 6248 PCT6248 stops forwarding traffic intermittently

Hello GregT_at_FCSG,

I would like to gather some more information.

Can you provide a topology layout?

Also run this and provide the output : show tech-support

Upload the  tech-support  report here under your service tag  https://dell.to/3VvGNHA

Then Private Message me the service tag for me to retrieve the report.

You can click on my name : MOD Dell -Charles R

Look in the Right hand column:

     Contact Me

          Send this user a private message with service tag

Hello GregT_at_FCSG,

I received the private message. Thank you for the attachments. I'll get them and review and update you.

History:

Three switch domains/networks each terminating to their respective phsyicaly firewall port.  The three domains are private, public, and voice.  Multiple wiring closets with each closet have a switch in each domain and an uplink to their respective top level switch.

There was not a connection between any of the domains.  All switches had an out of box (default)  configuration with only a host name, IP address, gateway, username and password added.

All switches had single link between upstream and downstream switches.  The deepest level is three switches. Most switches are 6248 but overtime, a few other Cisco Small Business (SG series) have been added as well as a few off brands.

Some users have 4-port unmanaged switches at the desktop.

Network was not monitored, logs were not reviewed.  Any network problems may have been passed off as an anomoly or self resolved, thus unnoticed.  I have to assume this network has existed for 10+ years.

June 2022, I became involved.  I was asked to allow the PCs to attach to the VoIP phone data port.  This would mean create VLANs and combining the data network and voice network into a single domain.  These networks had different IP addresses.

The simple solution was to keep the data on VLAN1 and create a new VLAN (110) for the voice network.  Then, start moving the phones from the dedicated voice switch to the data switch.

The voice VLAN would still use the firewall as the gateway so a VLAN 110 interface was not needed on the data switch.  In fact, none of the switches were running layer 3, the firewall was the default gateway for every network.

To join the data and the voice network, I configured a port on the data switch as an access port with this configuration:

CoreSwitch#show running-config interface gi1/0/40

switchport access vlan 110
lldp transmit-tlv port-desc sys-name sys-desc sys-cap
lldp transmit-mgmt
lldp notification

CoreSwitch#show isdp neighbors det

Device ID PHSWITCH1
Address(es):
IP Address: xx.xx.110.3
Capability Router
Platform PCT6248P
Interface Gi1/0/40
Port ID 1/0/40
Holdtime 156
Advertisement Version 2
Time when last changed 0 days 00:01:04
Version :
3.2.0.7

The configuration of PHSWITCH1 is unknow at this time but assumed to be default.

This work for a month without apparent issue but since this a school, there was not much activity.

In August, the network had one switch drop off line im,mediately after trunking was enabled to allow the voice vlan to propagate.  Problem woukd resolve when a docking station/iMAC attached was removed from the user port.  Symptom was the downstream switch became ROOT.  Moving the dock station/iMAC to an access port on the core switch resolved the issue.

In September, the same scenario happened on another switch.  The symptom was the switch would stop responing to ping and traffic would not flow.  Found disconnecting a specific user (port 44) on the downstream switch resolved the issue.  Once resolved, the dock could be re-attached.  Unlike the August failure, reattaching the dock to port 44 did not cause the problem to return.  In the Augsust failure, the problem was reproducable as soon as the dock station was attached to the access switch (model 6248) but did cuase an issue when attached to the CoreSwitch (model 2048).

At this time, syslogging was enabled, spanning tree settings applied, network monitoring (ping tool and SNMP) added.  Now that more information was available, I found that multiple switches would go offline and then reappear.

Spanning tree topology changes were frequent.  More tuning with spanning tree happened, tyical settings now are:

global

spanning-tree portfast bpdufilter default

access ports:

spanning-tree portfast
spanning-tree tcnguard

Not all switches have been touched yet but this seems to be calming down the  log messages

"Spanning Tree Topology Change: 0, Unit: 1"

and eliminated any:

"Spanning Tree Topology Change Received: MSTID: 0 Gi1/0/xx"

The Cisco SG switches are configured with

SWITCH-37#sho spanning-tree

Spanning tree enabled mode RSTP
Default port cost method: long

I am investigating (without much luck), what the setting is for the Dell 2048 and 6248 switches for the port cost method.

Hi @GregT_at_FCSG,

Based on your switch logs, I found that the switch is on 3.3.5.5. There is a fix in the firmware on 3.3.18.1 about voipTask that might be related to your situation. Try updating the firmware and check if the issue persist. 

https://dell.to/3g2s0ni

Hello GregT_at_FCSG,

This may be a good resource for you:

Route Preferences  Pages 95-98

https://dell.to/3SedUwp

PowerConnect 6248P Release Notes:

https://dell.to/3Tquz0W

We are putting a freeze any changes for a few weeks so I will not be able to try anything until mid- November.

I do need to understand the spanning-tree port/path cost method for the 6248 (short or long).  We have both a mixed model and a mix vendor environment and i want to make sure they all match.

I can find this info on various Dell switches but have been unable to locate this for the PCT6248 and also did it change with any firmware revisions.  Do you know where I can get this information?

Not sure why I didn't think of this earlier.

The newer version (a.k.a. long) will calcualte the cost of a 1G link to be 20,000 versus the older version (a.k.a. short) would calculate the cost to be 4.  So all I had to do is look at the cost of a port and sure enough a 1G link is 20,000.

To answer my own question, the PCT6248 is using the long method.

I have not made any firmware updates or changes.  Acess switch SWITCH_Rm113 (PCT6248P) downlinked from my CoreSwitch (N2048) decided to become STP Root for no reason.  This caused the CoreSwitch to set the STP state to discard on the downlink port (Port 7) which is connected to SWITCH_Rm113 port 48.

The CoreSwitch STP priority is 4096.  MAC address: F8B1.5654.8321

The SWITCH_Rm113 STP priority is 32768. MAC address: D067.E5AE.8712. Firmware 3.3.1.10

Why would a switch decide to become STP root for no apparent reason?  No strange log entries around this same time.

Log from SWITCH_Rm113:

<189> NOV 29 13:37:26 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2831 %% Link Down: 1/0/6
<189> NOV 29 13:37:26 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2832 %% Link on 1/0/6 is failed
<189> NOV 29 13:37:26 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2833 %% 1/0/6 is transitioned from the Forwarding state to the Blocking state in instance 0
<189> NOV 29 14:49:46 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2834 %% Link Down: 1/0/23
<189> NOV 29 14:49:46 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2835 %% Link on 1/0/23 is failed
<189> NOV 29 14:49:46 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2836 %% 1/0/23 is transitioned from the Forwarding state to the Blocking state in instance 0
<189> NOV 29 14:52:58 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2837 %% Unit 1 elected as the new STP root
<189> NOV 29 14:52:58 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2838 %% Instance 0 has elected a new STP root: 8000:d067:e5ae:8712

<189> NOV 29 15:01:58 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2839 %% Link Down: 1/0/48
<189> NOV 29 15:01:58 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2840 %% Link on 1/0/48 is failed
<189> NOV 29 15:01:58 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2841 %% 1/0/48 is transitioned from the Forwarding state to the Blocking state in instance 0
<189> NOV 29 15:02:03 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2842 %% Link Up: 1/0/48
<189> NOV 29 15:02:03 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2843 %% 1/0/48 is transitioned from the Forwarding state to the Blocking state in instance 0
<189> NOV 29 15:02:25 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2844 %% 1/0/48 is transitioned from the Learning state to the Forwarding state in instance 0
<189> NOV 29 15:02:25 192.168.0.34-1 TRAPMGR[125666288]: traputil.c(611) 2845 %% Spanning Tree Topology Change: 0, Unit: 1
<189> NOV 29 15:02:49 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2846 %% Link Down: 1/0/48

<189> NOV 29 15:01:58 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2839 %% Link Down: 1/0/48
<189> NOV 29 15:01:58 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2840 %% Link on 1/0/48 is failed
<189> NOV 29 15:01:58 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2841 %% 1/0/48 is transitioned from the Forwarding state to the Blocking state in instance 0
<189> NOV 29 15:02:03 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2842 %% Link Up: 1/0/48

    :

    :

<189> NOV 29 15:06:23 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2855 %% 1/0/48 is transitioned from the Forwarding state to the Blocking state in instance 0
<189> NOV 29 15:06:32 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2856 %% Link Up: 1/0/48
<189> NOV 29 15:06:32 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2857 %% 1/0/48 is transitioned from the Forwarding state to the Blocking state in instance 0
<189> NOV 29 15:06:39 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2858 %% Instance 0 has elected a new STP root: 1000:f8b1:5654:8321
<189> NOV 29 15:06:39 192.168.0.34-1 TRAPMGR[152677424]: traputil.c(611) 2859 %% 1/0/48 is transitioned from the Learning state to the Forwarding state in instance 0
<189> NOV 29 15:06:39 192.168.0.34-1 TRAPMGR[125666288]: traputil.c(611) 2860 %% Spanning Tree Topology Change: 0, Unit: 1

I shut/no shut CoreSwitch port 7 multiple times. After 13 min and 13 seconds, SWITCH_Rm113 changed STP root to the CoreSwith. Not sure if the multiple shut/no shut assisted with this change.

SWITCH_Rm113#show spanning-tree
Spanning tree :Enabled - BPDU Flooding :Disabled - Portfast BPDU filtering :Enabled - mode :rstp
CST Regional Root: 80:00:D0:67:E5:AE:87:12
Regional Root Path Cost: 0
ROOT ID
Address F8:B1:56:54:83:21
Path Cost 20000
Root Port 1/g48
Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec
Bridge ID
Priority 32768
Address D0:67:E5:AE:87:12

CoreSwitch#show spanning-tree

Spanning tree :Enabled - BPDU Flooding :Disabled - Portfast BPDU filtering :Disabled - mode :rstp
CST Regional Root: 10:00:F8:B1:56:54:83:21
Regional Root Path Cost: 0
ROOT ID
Priority 4096
Address F8B1.5654.8321
This Switch is the Root.
Hello Time 2 Sec Max Age 20 sec Forward Delay 15 sec TxHoldCount 6 sec
Bridge Max Hops 20

Hello thanks for choosing Dell. 

I suggest bringing  firmware to up to date on both switches and check the status. It could be local loop occurred in the setup.