Skip to main content
Start a Conversation

Unsolved

This post is more than 5 years old

J

JSH_IN

3 Posts

29942

October 28th, 2004 19:00

ARP Storm Control

We have PowerConnect 5224 switches at the edge of our LAN with Catalyst 3750G-24TS switches at the core.  Recently we experienced an ARP Storm coming from a PC that was infected with the SQL Slammer worm.  The ARP Storm brought the entire network virtually to a halt.
 
The PC 5224's are set to the default of 256 broadcasts per second on all of the switch ports.  When the ARP Storm started, all of the PC 5224 switches reported STP Root changes and Topology changes continuously until the offending PC was disconnected from the LAN:
 
2004-10-14 10:10:17 Local7.Info psi7734lsw3 psi7734lsw3 trapmgmt:STP root change notification.
2004-10-14 10:10:17 Local7.Info psi7734lsw10 psi7734lsw10 trapmgmt:STP root change notification.
2004-10-14 10:10:17 Local7.Info psi7734lsw5 psi7734lsw5 trapmgmt:STP root change notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw4 psi7734lsw4 trapmgmt:STP root change notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw6 psi7734lsw6 trapmgmt:STP root change notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw6 psi7734lsw6 trapmgmt:STP root change notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw5 psi7734lsw5 trapmgmt:VLAN 3 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw5 psi7734lsw5 trapmgmt:VLAN 5 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw10 psi7734lsw10 trapmgmt:VLAN 2 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw7 psi7734lsw7 trapmgmt:VLAN 2 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw5 psi7734lsw5 trapmgmt:VLAN 10 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw11 psi7734lsw11 trapmgmt:VLAN 2 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw9 psi7734lsw9 trapmgmt:STP root change notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw6 psi7734lsw6 trapmgmt:VLAN 2 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw10 psi7734lsw10 trapmgmt:VLAN 3 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw7 psi7734lsw7 trapmgmt:VLAN 3 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw11 psi7734lsw11 trapmgmt:VLAN 3 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw9 psi7734lsw9 trapmgmt:STP topology change notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw6 psi7734lsw6 trapmgmt:VLAN 3 link-up notification.
2004-10-14 10:10:18 Local7.Info psi7734lsw10 psi7734lsw10 trapmgmt:VLAN 5 link-up notification.
 
The log looks like that for over 30 minutes.  Never ending STP Root changes and Topology changes.
 
Now my theory is that the ARP Storm was rate-limited to 256 packets/sec on the edge switch, but those 256 packets crossed the entire LAN every second causing all of the other switch ports to rate limit other legitimate broadcasts (ARPs, STP, etc).
 
I'm thinking of changing the Broadcast Control settings to 64 packets/sec on host ports and leaving it at 256 packets/sec on Trunk ports to the Catalysts.  That way if this happens again, the offending PC will be limited to pumping out 64 broadcasts/sec, allowing the trunk ports to pass 192 legitimate broadcasts/sec thereby keeping the LAN functioning at a reasonable level until the offending PC is found and removed.

Thoughts?

JSH_IN

3 Posts

November 11th, 2004 14:00

Nobody has any thoughts or ideas? Nobody has ever experienced this type of problem with PowerConnects?

JSH_IN

3 Posts

November 15th, 2004 12:00

I tested my theory this weekend using packETH to generate an ARP storm across the LAN. Removing broadcast control from the trunking interfaces didn't resolve the problem, as STP still flapped continuously during the storm.

This brings me to my latest theory. The ARP storm is occuring on the same LAN as the management interfaces of the switches. I believe this may be causing the PowerConnect 5224's to become CPU bound as it has to look at each ARP request. I'm going to try moving the management interface to a seperate VLAN and perform the ARP storm again.

It sure would be nice if the PowerConnect 5224's presented more statistics, like CPU usage. Is there an SNMP OID for CPU usage on the 5224's? Does anyone even care about this problem? Dell???

DELL-GregG

812 Posts

November 15th, 2004 12:00

The private MIBs do not support an object for monitoring CPU utilization. There is no way to see the CPU utilization on any of our current PowerConnect products.
 
The broadcast suppression inplemented in the 3.1.0.8 firmware was a software-based solution specifically intended to prevent the CPU from locking up during a broadcast storm. It is still entirely possible to lock the processor with a broadcast traffic percentage greater than 30% on a large network. Unfortunately, the ASIC does not support a more robust broadcast suppression feature.
 
If you are using a firmware prior to 3.1.0.8, the broadcast suppression did not function properly. I would suggest updating to the latest firmware, 3.1.0.16.

View More

View All

No Events found!

Top