Solved!
Go to SolutionCritical: network (bnxt_en module) crashes on 14G servers
Hi,
We dozen of new 14g servers, they were running (with no or minimal load) in last few weeks without any issues.
After we deployed BIOS upgrade (1.3.7), and rebooted (even power off/on), some of the servers (every 3rd) got locked with below kernel messages.
Doesn't matter if BIOS 1.3.7 or BIOS 1.2.11 are installed, problem persist. Even after few days of running (minimal or no load at all) they've got locked.
How we should proceed from here, as the servers should get more load, and we lost faith in stability of them. Running fulling updated RHEL v7 and firmwares.
bnxt_en modules are dies, and lock network access and spews constantly
00:03:34 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0x12be} len:0
00:03:34 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x51 0x12d6} len:0
More about the kernel message on paste bin url: Kernel messages of crashed bnxt_en
Please let us know which way to proceed from here.
Thank you!
DELL-Josh Cr
Community Manager
•
9699 Posts
•
43744 Points
12097
0
Posted March 9th, 2018 13:00
Hi,
Have you tried flashing back to an older bios? What version were you running previously? Are they all the same model of NIC?
Why are my follow-ups erased from this forum?
Below are more lockups with latest DELL software.
DELL-Josh Cr
Community Manager
•
9699 Posts
•
43744 Points
12080
0
Posted March 9th, 2018 14:00
Can you private message me the followups if they are not posting?
Yes. All hardware is same across the server base.
I reverted back on couple of servers to 1.2.11 BIOS. We did upgrade LifeCycle / iDRAC as well, but this surely is not connected with iDRAC.
It seems that we should go with full support from DELL route.
If you have any other valuable resource, particularly with the network card (Broadcom BCM57412 NetXtreme-E 10Gb Ethernet), please let me know.
Thanks!
DELL-Josh Cr
Community Manager
•
9699 Posts
•
43744 Points
12076
0
Posted March 9th, 2018 15:00
If reverting does fix it, we can look into issues with the update, if it doesn’t fix it we can try to find out what is causing the problem. What version were you running previously? Are all the servers using same model of NIC? Can you create a support assist bundle? https://www.dell.com/support/article/us/en/04/sln306670/poweredge-14g-how-to-manually-create-the-supportassist-collection-?lang=en
Unfortunately, reverting to BIOS which was delivered with the servers doesn't work.
Again, same lock ups. Two more lockups with two different server with BIOS 1.2.11 on the link below.
Thank you
DELL-Josh Cr
Community Manager
•
9699 Posts
•
43744 Points
11997
0
Posted March 12th, 2018 09:00
Can you create a support assist bundle on one server on the older bios and one on the newer one? https://www.dell.com/support/article/us/en/04/sln306670/poweredge-14g-how-to-manually-create-the-supportassist-collection-?lang=en
NonOnNon
16 Posts
16619
1
Posted May 8th, 2018 23:00
Solution was to power down the server(s) and *remove* power cords for 5 minutes.
This will force re-init of the NIC firmware.