Hello,

We have several Dell PowerEdge Servers equipped with Broadcom Adv. Dual 10Gb Ethernet cards. They run Ubuntu 18.04 (Kernel 5.4.0-144-generic) with an OpenStack cloud installation using VXLAN tunnel for tenant networking.

Since the recent update of the NIC firmware to 22.31.13.70, all TCP connections inside VXLAN tunnels are not working anymore. ICMP packets can be sent and received by virtual machines, but TCP data connections are stuck. The kernel log shows the following messages:

kernel: bnxt_en 0000:19:00.0 data: hwrm req_type 0xa1 seq id 0x5716 error 0xffff
kernel: bnxt_en 0000:19:00.0 data: hwrm_tunnel_dst_port_alloc failed. rc:-95

We tried resetting the NIC firmware by physically unplugging the server from all power for 10 minutes, but this did not help too. Further investigation indicated that disabling TX checksum offloading avoids the issue:

# ethtool --offload data tx off
Actual changes:
tx-checksumming: off
	tx-checksum-ipv4: off
	tx-checksum-ipv6: off
tcp-segmentation-offload: off
	tx-tcp-segmentation: off [requested on]
	tx-tcp6-segmentation: off [requested on]

With this workaround, connectivity was restored.

Not offloading, of course, has an impact on performance and latency. Would you recommend a downgrade to the previous firmware, and disabling iDRAC firmware updates, or will there be a fix soon?

Thank you.