I bought a used S6100 switch with 64 40gbe ports and a bunch of ConnectX-3 NICs to replace the gigabit network I have been running in my server room. With the NICs in the servers and directly connected to each other, NICs and DAC cables worked perfectly. Configuring the S6100 has me pulling my hair out. Each port with a DAC inserted is set to: no ip address, mtu 9001, switchport, no shutdown. All active interfaces are in VLAN 1 and it is mtu 9001, untagged, no shutdown. I have 1gbe management interface assigned 192.168.1.2/24. All the 40GBe NIC interfaces are assigned 192.168.40.0/24 or 192.168.41.0/24. I can ping all the 192.168.40.0/24 and 192.168.41.0/24 addresses successfully from the management port, but trying to ping a server from an other server doesn't work. The only way to get to the 40 and 41 networks is through the switch, so the connections between the servers and switch are obviously fine. Also, all the servers are Ubuntu 22.04. Eventually I would like to get RoCEv2 running, but i can't even get the servers to see each other. Any thoughts on what I missed?