PowerEdge: ESXi 7.0 U3 - Network Port Missing Interrupt and Cannot Initialize Link

Servers with a large quantity of Network Interface Cards (NICs) installed are noticing link down on NIC ports after booting ESXi 7.0.

The network card configuration is as follows:

Analysis:

1. Confirmed that the functionality works with slot7 network card by performing a loopback test in the BIOS.
2. Collected OS logs and analyzed the vobd log, found that slot7 port3 and port4 corresponding to vmnic16 and 17 is not registered as a device.
3. Check the VMkernel log and found that the corresponding hexadecimal ID b4:00.3 reported "failed to allocate 64 MSIX interrupts."

OS version：

• VMware ESXi 7.0.3 build-23794027
• VMware ESXi 7.0 U3

The NIC driver info:

bnxtnet                        231.0.153.0-1OEM.700.1.0.15843807      BCM     VMwareCertified   2025-03-28


180 : 00 : 00        Broadcom Inc. and subsidiaries        NetXtreme-E P425D BCM57504 4x25G SFP28 PCIE        NIC.Slot.7-1-1

180 : 00 : 01        Broadcom Inc. and subsidiaries        NetXtreme-E P425D BCM57504 4x25G SFP28 PCIE        NIC.Slot.7-2-1

180 : 00 : 02        Broadcom Inc. and subsidiaries        NetXtreme-E P425D BCM57504 4x25G SFP28 PCIE        NIC.Slot.7-3-1

180 : 00 : 03        Broadcom Inc. and subsidiaries        NetXtreme-E P425D BCM57504 4x25G SFP28 PCIE        NIC.Slot.7-4-1

vmnic：

0000:b4:00.0 Ethernet controller: Broadcom NetXtreme-E P425D BCM57504 4x25G SFP28 PCIE Ethernet [vmnic14]  in  PCIe Slot 7

0000:b4:00.1 Ethernet controller: Broadcom NetXtreme-E P425D BCM57504 4x25G SFP28 PCIE Ethernet [vmnic15]  in  PCIe Slot 7

0000:b4:00.2 Ethernet controller: Broadcom NetXtreme-E P425D BCM57504 4x25G SFP28 PCIE Ethernet [vmnic16]  in  PCIe Slot 7

0000:b4:00.3 Ethernet controller: Broadcom NetXtreme-E P425D BCM57504 4x25G SFP28 PCIE Ethernet [vmnic17]  in  PCIe Slot 7

vobd log:

2025-04-03T14:58:31.645Z: [netCorrelator] 13121287320us: [vob.net.vmnic.linkstate.up] vmnic vmnic13 linkstate up

2025-04-03T15:03:56.295Z: [netCorrelator] 13445937328us: [vob.net.vmnic.linkstate.down] vmnic vmnic13 linkstate down

2025-04-03T16:57:06.411Z: [netCorrelator] 20236053468us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up

2025-04-03T17:25:17.682Z: [netCorrelator] 21927324575us: [vob.net.vmnic.linkstate.up] vmnic vmnic14 linkstate up

2025-04-03T17:26:56.888Z: [netCorrelator] 22026529736us: [vob.net.vmnic.linkstate.up] vmnic vmnic15 linkstate up

VMkernel：

Line 42990: 2025-04-01T18:42:40.775Z cpu47:2097967)WARNING: VMK_PCI: 598: device 0000:b4:00.3 failed to allocate 64 MSIX interrupts

Line 43540: 2025-04-01T18:42:48.232Z cpu2:2099020)WARNING: VMK_PCI: 598: device 0000:b4:00.2 failed to allocate 64 MSIX interrupts

Line 43589: 2025-04-01T18:42:48.378Z cpu2:2099020)WARNING: VMK_PCI: 598: device 0000:b4:00.3 failed to allocate 64 MSIX interrupts

Line 54191: 2025-04-03T11:20:14.710Z cpu0:2097969)WARNING: VMK_PCI: 598: device 0000:b4:00.2 failed to allocate 64 MSIX interrupts

Line 54254: 2025-04-03T11:20:14.767Z cpu6:2097969)WARNING: VMK_PCI: 598: device 0000:b4:00.3 failed to allocate 64 MSIX interrupts

Line 54801: 2025-04-03T11:20:22.225Z cpu48:2099006)WARNING: VMK_PCI: 598: device 0000:b4:00.2 failed to allocate 64 MSIX interrupts

Line 54850: 2025-04-03T11:20:22.398Z cpu48:2099006)WARNING: VMK_PCI: 598: device 0000:b4:00.3 failed to allocate 64 MSIX interrupts

ESXi does not detect all the vmnic adapters, but, those vmnic adapters are seen in "lspci" and not in esxcfg-nics -l. 

In the vmkernel.log file of the ESXi host, you see entries similar to:

xxxx:xx:xx.x failed to allocate 64 MSIX interrupts

See the Broadcom knowledge article How to increase maxIntrCookies if default limit is reached in ESXi  

The "maxIntrCookies" default value was increased to 4096 in ESXi version 8 to address this issue. Update host "maxIntrCookies" settings to 4096 where values are found to be set below 4096.

Workaround:
To work around this issue, adjust the boot option "maxIntrCookies" to 4096 to increase the interrupt limit.

The default value for "maxIntrCookies" depends on the ESXi version in use and whether the advanced settings value was manually set higher before a host was upgraded to ESXi version 8 where the default value was increased.  

• ESXi version 7.x "maxIntrCookies" default value is set to 1024
• The default value in ESXi version 8 was increased to 4096.

Note: The maximum interrupt cookies are capped to 4096 even if you set it to a value that is > 4096.
For more information about instructions on how to configure boot options, see Configuring advanced options for ESXi/ESX (310338). 


SSH to ESXi CLI:

esxcli system settings kernel set -s maxIntrCookies -v 4096

/sbin/auto-backup.sh