3548 CPU Utilization

I swapped it with another 3548p unit and used the same config.  When unracking the problematic one, I unplugged one port at a time to see if the CPU load and high ping times went down when any particular device was disconnected, and it did not: at the end, with only a serial cable plugged into it for console access and a single ethernet cable plugged into the firewall for internet access, CPU times were still high.

Turning it off, racking a replacement and reconfiguring the new one did indeed fix the problem: CPU util went back down to normal (under 10%), and ping times were normal as well.  What's strange is as soon as we did this (and not before), a second switch at the same site starting exhibiting the same behavior as the first problem!  This lead me to believe the problem has something to do with spanning-tree, as perhaps there was a problem that was propagating to whatever the root master was.  Of course, disconnecting that first switch (which was a core) would have forced an election process, and the second switch (an edge) may have inherited the bad state.  

Currently, the edge switch is still in this state.  It is showing itself as the STP bridge, not the root (the new core holds root).  It's CPU is running at max, and ping times are high:

From the core switch that is directly connected via fiber:

TOCoreSW# ping tocamerasw
Pinging tocamerasw.britzinc.com. (10.229.100.252) with 56 bytes of data:

56 bytes from 10.229.100.252: icmp_seq=1. time=1400 ms
PING: no reply from 10.229.100.252
PING: timeout
56 bytes from 10.229.100.252: icmp_seq=3. time=620 ms
56 bytes from 10.229.100.252: icmp_seq=4. time=680 ms

----10.229.100.252 PING Statistics----
4 packets transmitted, 3 packets received, 25% packet loss
round-trip (ms) min/avg/max = 620/900/1400

The port does show some multicast traffic, but a very small percent and without any errors:

TOCoreSW# show int counter eth g1

Port InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
g1 611656928 14345 64008 914038994303

Port OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
g1 307397488 168974 563212 19821710826

FCS Errors: 0
Single Collision Frames: 0
Late Collisions: 0
Excessive Collisions: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0
TOCoreSW#

Flow control is off.  

As far as VOIP goes, yes the phones have built-in switches and workstations are daisy-chained off of them.  Because a dumb switch could theoretically be plugged into this same port, we cannot use portfast or disable STP on ports feeding phones, as they are technically not end-points.  We have such a small network that we have not split voice and workstation traffic into distinct VLANs: there is only 1 VLAN on the switch, the default VLAN 1.  We are not using the voice VLAN and have no OUI defined for our phones. This simplifies switch conf as well as phone conf, but we can test creating a voice vlan if necessary.

I actually did already take the original switch that showed this behavior and booted it up in an isolated environment, and the CPU utilization looked fine.  So I agree there is probably something unique to that network causing this.  Unfortunately it does not appear as if simply unplugging this device resolves the problem.  Rebooting while each device is unplugged is an option, but will take an extremely long time!

Wireshark is a good idea, I will capture and analyze packets on that switch.  What is the CLI command to enable port-mirroring, to put an interface in promiscuous mode where it gets copies of all traffic?  I see:

config

int eth e1

port monitor

where can be any single port, and I can add up to 4 ports to this config, but that's far too low of a limit.  Is there no way to make a port fully promiscuous?