nic bonding through 2 unstacked 6284 switches

Bonding in considered the same as teaming the Nic's together or Link Aggregation on the switch side..  When teaming the individual physical ports need to end at the same device.  In your case this is not happening.  If the 6248 switches were stacked you can have one cable going to each physical switch.  This will give you physical redundancy if a switch were to go down for some reason.

This would be the reason for the dropped packets.

Well, according to the linux docs on the matter the balance-alb bonding/teaming mode does not require anything special in terms of switch config. So, I'm not using LAG / LACP on the switches.

As far as I understand the linux algorithm chooses a NIC mac address for every outgoing connection and also replies to ARP queries in a similar way - the goal is to spread peers equally over all NICs (every peer has its NIC but not all peers share the same NIC). Which gives you both redundancy and load sharing. And I like that :)

Anyway, I'm not really used to Dell switches, if I would like to get deeper insight on what exactly is happening ... can you give me some suggestions on where to look? Any network diagnostic tools I can use with the switches? Or logging, packet capture, etc? ARP monitoring, maybe the linux ARP 'tricks' are somehow messing up the switches?

Right, ok, yesterday I found some settings on the switches that control dynamic ARP inspection, seems to be some sort of a rate-limiter for ARP traffic. I disabled it (thinking that linux does GARP and stuff) and that solved most of the problems. For instance, at the moment both switches see the correct mac address at their ports, if I unplug one of the cables the arp tables on the switches change and traffic is not interrupted.

Unfortunately, I still have a problem - traffic between some host pairs is impossible. When I do a ping from one server to another tcpdump shows that requests are received but replies disappear somewhere ... Even though the arp tables on the switches seem to be correct.

Any thoughts? Ideas about other arp-related settings on the switches?

How are you physically connecting the cables and hardware at this point?  How exactly did you disable the garp?  The "no garp timer" command just resets the parameters to default based on this information I pulled from the CLI User Guide.

garp timer

Use the garp timer command in Interface Configuration mode to adjust the

GARP application join, leave, and leaveall GARP timer values. To reset the

timer to default values, use the no form of this command.

Syntax

garp timer {join | leave | leaveall} timer_value

no garp timer

I'm not seeing many options in manipulating the arp feature other than changing the timers.  I will keep looking at what options we may have on configuring it differently.

Nooo, perhaps I wan't as clear as I could. I didn't disable the GARP timer, what I was doing was browsing the web interface of the switch ... looking for inspiration or something :) And I found this 'dynamic ARP inspection' section under 'switching'. And in it there's some filter, that apparently applies limits to ARP traffic, by default. The help says you can turn it off by setting the port to 'trusted' state. This is what I did.

Just trying to gather some more information.

What firmware are the switches at?

Are we only seeing the lost replies when using this specific bond mode? If we do a single connection without bonding do the replies make it back properly?

To confirm, you are seeing the replies leave the server? Or a reply from the server is never seen?

Thanks