Hello,

This is the first I've heard of the issue and I'm working on confirming OVS offload support, but would you mind sharing what BIOS revision you're on? 

Hi, Did you resolve that?

I have the same problem with R660xs and Broadcom BCM57508 2x100G QSFP PCIE adapter running under Ubuntu 24.04. An attempt to enable OVS hardware offloading according to the instructions here and here

results in the similar error messages:

# echo 1 > /sys/class/net/enp23s0f0np0/device/sriov_numvfs
# devlink dev eswitch set pci/0000:17:00.0 mode switchdev
# dmesg | tail
[ 1749.554291] bnxt_en 0000:18:00.0 eth0: Broadcom BCM5750X NetXtreme-E Ethernet Virtual Function found at mem 21fffc2b0000, node addr 12:ea:00:e8:10:3a
[ 1749.554312] bnxt_en 0000:18:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 1749.558379] bnxt_en 0000:18:00.0 enp23s0f0v0: renamed from eth0
[ 1761.134566] bnxt_en 0000:17:00.0 enp23s0f0np0: hwrm req_type 0xfd seq id 0x15dc error 0xffff
[ 1761.134631] bnxt_en 0000:17:00.0 enp23s0f0np0: hwrm_cfa_vfr_alloc error rc=-95
[ 1761.134638] bnxt_en 0000:17:00.0 enp23s0f0np0: bnxt_vf_reps_create error=-67

According to the source code of bnxt_en the request type 0xfd stands for HWRM_CFA_VFR_ALLOC, I guess, a request to create a VF representor.

Here are my BIOS, driver, and firmware versions:

System BIOS Version     = 2.5.4

Firmware Name                       : PRIMATE_FW
Firmware Version                    : 232.0.156.9
RoCE Firmware Version               : 232.0.156.0
HWRM Interface Spec                 : 1.10.3
Kong mailbox channel                : Not Applicable
Active Package Version              : 23.21.13.39
Package Version on NVM              : 23.21.13.39
Active NVM config version           : 232.0.0 

BIOS settings (also tried different things without much success):

racadm>>get NIC.DeviceLevelConfig.5
[Key=NIC.Slot.1-1-1#DeviceLevelConfig]
Broadcom_AdapterErrorRecovery=Enabled
Broadcom_Bar2Size=64M
Broadcom_DCBX=Disabled
Broadcom_DefaultEVBMode=VEB
Broadcom_FlowOffload=Enabled
Broadcom_LLDPNearestBridge=Enabled
Broadcom_LLDPNearestNonTPMRBridge=Enabled
Broadcom_MaxPfMsixVectors=74
Broadcom_MsixVectorsPerVf=8
Broadcom_PerformanceProfile=Default
Broadcom_PortEnablement=Enable_All_Ports
Broadcom_SupportRdmaOnVFs=Enabled
#NParEP=Disabled
NumberVFAdvertised=32
#NumberVFSupported=64
#PartitionStateInterpretation=Fixed
#RDMAProtocolSupport=RoCE
#RDMASupport=Available
#SRIOVSupport=Available
#VFAllocBasis=Device
#VFAllocMult=8
VirtualizationMode=SRIOV

Also, the latest release notes mention:

Notables
========
- OVS has been deprecated for 5741x and 5750x devices in this release
- OVS-DPDK (Flow Offload) feature added for 5741x only.
	- Not currently supported for 575xx device
	- Currently as TechPreview or Early Access
- RDMA over SRIOV(VF) has been deprecated for 574xx devices

Could this be clarified? What exactly for OVS has been deprecated? For the flow offload, does it mean that setting Broadcom_FlowOffload to Enabled does not have any effect on 575xx? Similarly, for Broadcom_SupportRdmaOnVFs on 574xx?

Hi,

Okay i'm just brainstorming, so about those OVS offloading issues with the Broadcom 57508... The errors you're seeing, plus the vendor's own release notes, point to them dropping support for switchdev mode on these 5750x cards. Even enabling things like FlowOffload in the BIOS won't help if the underlying feature is gone from the firmware/driver. Basically, I believe it looks like OVS switchdev is off the table for this card and setup because Broadcom doesn't support it anymore.

Hope that helps!

I looked at the upstream release notes from broadcom (for the last 2 releases), and I do not see mentioning any drop of support for OVS:

https://docs.broadcom.com/doc/Release_Notes-233.1.135.7
https://docs.broadcom.com/doc/Release_Notes-231.1.162.1

Also, this issue was first reported over 5 years ago. It does not seem to me like a recent change.

Setting switchdev mode for VF was a step to enable TrueFlow, which is advertised for all BCM95750X adapters. Could it be that some broadcom firmware features have been disabled by Dell? E.g., I do not find TrueFlow mentioned anyware in the documentation provided for this network card by Dell.

Hey,

I've just read through all of this since I was trying to find out how I mix'n'match VEB with OVS (LPE16002 here). One thing that VERY much started coming up in the back of my head during reading was the DCB-X part.

In the "classic" BIOS menu docs you also find more info about constraints and it seems to match my suspicion. So, after a long nightmare session on Intel XXV-710 or so, hyper-intelligent cloud NICs connected to era-matching datacenter switches (Cisco Nexus 5K iirc) I kept coming back to DCB-X and in the end, we had problems caused (traffic loss because the NIC and switch kept coordinating automatically and without the OS via LLDP. Basically, they agreed on switch control of VEB or similar mechanisms on the NIC but noone of the admins or OS designers had expected that. So it had no actual policy.

I very strongly suspect your cases are the reverse of what I described - to use VEB you would need to enable the LLDP Sub-TLVs for virtualization. I've run to my bookshelf, but i'm not sure where I found it back then. Searched for half an hour now, to no avail. as I said before it was a nightmare session where I discovered that, and I was sick so I don't have the notes at usual quality.

The whole stuff is underdocumented, as DCB and TRILLL and all that stuff pre 2015 and pre VXLAN sadly didn't take off - since one needed to read docs :-)

I think back then the client admins just disabled the NIC builtin LLDP agent or replaced the NICs instead of adding one line of switch config.

I hope some of you can make it work for your case. I'm really sure it should work if you enable the protocols and kick over some policy.

I've seen more recent switches talk dcb-x+lldp proper in recent time still, i.e. a QFX5110 had it.

If you need to lab it out, get an old Brocade VDX since they had all the ethernet fabrics stuff, it should be much easier to work out from there. 

See below URL and the quote I put underneath the link.

https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/installation/bios-comprehensive-configuration-management.html

Here with some formatting that got lost:

– Configures the Edge Virtual Bridging (EVB) mode, which is an IEEE standard that involves the interaction between virtual switching environments in a hypervisor and the first layer of the physical switching infrastructure.

First layer here means the first layer of switches in the topology, i.e. ToR or blade chassis' internal switches. it does not refer to a physical layer (PHY) or the switch on the VEB NIC.

Hi,

Mixing and matching VEB (Virtual Ethernet Bridge) with OVS (Open vSwitch) can be a powerful but nuanced setup, especially in environments where performance, visibility, and policy enforcement. 

Thanks for sharing such a detailed and insightful experience—this is a goldmine for anyone dealing with VEB, OVS, and DCBX interactions in virtualized environments!

@DELL-Joey C I didn't post that yesterday, but later I thought the most plausible course of action is to start by only enabling the flow feature and disabling VEB. do integration tests with OVS there, this will not need switch interaction. If there's no driver callback errors in that situation, then flip on VEB.

I also wish best of luck to anyone who tries, it is a good path to highly efficient virtualization envs.

I tried setting

Broadcom_DefaultEVBMode=None

without much success (same error message). As I wrote, I tried many different settings and they did not appear to have any effect on creating OVS VF representors. OVS without representors (eswitch legacy mode) appers to be working fine.

B.t.w., I contacted Broadcom regarding this issue, and got a reply: "Dell-branded products differ in design and or firmware and features". So I guess Broadcom instructions may not apply to Dell-branded cards, even with of the same model name.

Following an advice of Dell FAE I then opened an Dell Service Request (ISG Support Service) with the report of this bug, but it seems that the Dell contractors only resolve hardware issues and could not help me with this problem.

Hello.

I'm also having trouble creating a representor
Are there any additional packages I need to install?
I can change it using switchdev, but VFR0 (eth0) isn't being created.
The Broadcom technical documentation states the following:

"ip link" should show a new VFR0 (eth0) interface.

Tests were conducted on Broadcom 57414 and 57504.

@mmx21​ it gets close to trolling them but since you already were in contact, would you mind asking broadcom to name a nic model and firmware version that is supposed to work?

i still got a (iirc stock) broadcom 25g nic somewhere, my lpe16004 was emc^2 oem but i think the burden of proof in this story needs to be with the vendors who need to come forward and name their tested setups including firmware revision. otherwise we are never able to trace back firmware regressions which (i feel pretty certain by now) the vendors themselves are unaware of.

so, in case of Dell - there's mentions in manuals of the setup, there's surely also a presentation here and there.

if we can collect those for your support staff as references, can they help out by tracking down the people who did the testing and then have those people make at least an educated guess what and when they did the tests with?

Then the community can help reproduce Dell's original results, cross validated with newer firmware(s) and with that, Dell would be able to make things work again by involving Broadcom or whomever knows who changed what and where.

@Darkfader feel free to ask broadcom if your card is supported:

https://www.broadcom.com/support/request-tech-support/ethernet-nic-support-request

They check using the card's serial number if the card is broadcom-branded or not.

(Be aware that the form is a bit flaky: do not write too much text in the description)

As far as I understand, Broadcom still provides firmware for Dell-branded cards, but they do not provide any technical support for them directly.

Let us know if you get any progress on this issue!