PowerEdge Hardware General

Last reply by 12-07-2016 Solved
Start a Discussion
2 Bronze
2 Bronze
7447

PowerEdge M610 and BCM57111 firmware update troubles

Hi,

I am currently doing a maintenance on our m1000e chassis, flashing the firmwares to the latest versions.

Our setup is:

- m1000e chassis
A1 - Dell PowerConnect M6220
A2 - Dell PowerConnect M6220
B1 - Dell PowerConnect M6220
B2 - Dell PowerConnect M6220
C1 - Dell PowerConnect M8024
C2 - Dell PowerConnect M8024

- 16 x PowerEdge M610 blades
Fabric A - Embedded LOM (1 gig)
Fabric B - Broadcom M5709t (1 gig)
Fabric C - Broadcom M57711 (10 gig)

Updating the chassis went well, everything is up to date.

But I'm having a lot of trouble updating the Broadcom M57711 on the blades.

I first tried to update them from OMSA "dsu" utility, without success.
Updating everything else on the blades went well, except the M57711 NIC's.

Then I was told that it should be updated from an updater live cd that I grabbed from:
www.dell.com/.../SLN296511 (update repositories)
Most specifically: dell.app.box.com/.../8303280249

It detected the NIC and started flashing them, but then this happened:

Device: BCM57711 10Gigabit PCIe (eth4)
Application: BCM57711 10Gigabit PCIe (eth4)
Update failure. Firmware programming utility returned an error. Original firmware restored

Device: BCM57711 10Gigabit PCIe (eth5)
Application: BCM57711 10Gigabit PCIe (eth5)
Update failure. Firmware programming utility returned an error. Original firmware restored

So I rebooted the blades to their OS, and ... Both M57711 interfaces are gone.

When bnx2x driver is loading it's throwing this:
[ 659.758218] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.713.04 ($DateTime: 2016/01/24 22:14:19 $)
[ 659.758383] bnx2x 0000:05:00.0: msix capability found
[ 659.758394] bnx2x: probe of 0000:05:00.0 failed with error -22
[ 659.758399] bnx2x 0000:05:00.1: msix capability found
[ 659.758403] bnx2x: probe of 0000:05:00.1 failed with error -22

Lspci is seeing the cards:
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57711 10-Gigabit PCIe
05:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57711 10-Gigabit PCIe

However, on the iDRAC of the blades they do not show up anymore. Only the other 1gig NICs are seen.

So I tried to reflash the firmware from the CMC update server components page (it's using the lifecycle controller to flash it).
With the latest firmware proposed by Dell for this card (using the blades service tag):
www.dell.com/.../DriversDetails
The blade rebooted, tried to flash and reported a failure.

I tried again with a less recent firmware:
www.dell.com/.../DriversDetails
Same problem.

I'm now stuck with the blades iDRAC and OS not seeing the cards anymore.

Anyone experienced something like this ?
I'm out of ideas what to try next.

Thanks for your help!
Sébastien

Solution (1)

Accepted Solutions
2 Bronze
2 Bronze
4480

Note: it is not a matter of stability but compatibility. Dell erroneously released the 8.X.X FW update in the Qlogic package with the BCM5771 listed in the catalog. They have since removed the BCM57711 from the web package. Unfortunately they have not fixed the SUU package deployment yet, nor the Lifecycle Controller web update.Also note that your cards will become useless after the firmware update failure on version 8.07.25. Only package 7.12.19 is the latest compatible FW package for that Dell BCM57711Mezz card.

View solution in original post

Replies (11)
Moderator
Moderator
5046

Hello

I don't recall an update that allows the M1000e or 11th generation blades to run 64 bit Dell Update Packages. I would suggest trying the update again using the 32 bit DUP:

http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=WR5R5

Thanks

Daniel Mysinger
Dell EMC, Enterprise Engineer

2 Bronze
2 Bronze
5057

Hello Daniel,

Thanks for taking time to answer my post.

I had already tried with the 32 bit version of the DUP, it failed on 5 blades out of 6 i've tried.

Only one has succeeded, but now the "dsu" utility on this blade (xen-blade04, see screenshot) thinks the firmware is outdated and want to flash it back to the previous version. Looks like the firmware version numbering has changed between latest and previous update packages (due to broadcom- > qlogic rebranding?).

Screenshot: http://imgur.com/ucM4GGA

Latest: Version 08.07.25, A00-00
www.dell.com/.../DriversDetails

Previous: Version 7.12.19, 7.12.19
www.dell.com/.../DriversDetails

So dsu thinks 08.07.25 is lower than 7.12.19.

--

Another thing is that before trying to flash the NICs from the CMC I tried to do it from the live update CD for the M610 blades, on the first two blades. The flash failed and it looks like the NIC completly died. It's not seen anymore. 

I tried to reseat and a bit later tried to hard reset the idrac (it worked but the NIC still don't show anymore).

Commands were:

racadm serveraction –m server-1 reseat –f

and

winrm i LCWipe schemas.dmtf.org/.../DCIM_LCService -u:[iDRAC_root] -p:[iDRAC_PWD] -r:https://[iDRAC_IP]/wsman -SkipCNcheck -SkipCAcheck -encoding:utf-8 -a:basic

screenshot: http://imgur.com/BYozkcR

There's something weird going on with these BCM57711 cards and their firmware and I don't know what to do next, especially for the two first blades where the cards seems to be dead after the flash attempt with the updater live cd.

I think those need to be replaced...

Thanks for your help.

Sébastien

Moderator
Moderator
5057

I had already tried with the 32 bit version of the DUP, it failed on 5 blades out of 6 i've tried.

The LCC/iDRAC update has built-in timeouts which can cause updates to fail if they take too long. Since you have so many network controllers that it is trying to update it may be triggering this time out. If the updates are failing then I would suggest updating from an operating system. I would try booting to Support Live Image again and run the updates designed for RHEL from that operating system.

There's something weird going on with these BCM57711 cards and their firmware and I don't know what to do next, especially for the two first blades where the cards seems to be dead after the flash attempt with the updater live cd.

I think those need to be replaced...

Yes, if those cards will no longer intialize then they will need to be replaced. There is a minimum level of functionality required for the card to receive a FW update. You can try booting the system with only those cards to see if they will initialize. You can do this by disabling the integrated LAN on Motherboard and physically removing the other add-on NIC. I would then boot to the Support Live Image and attempt to update the firmware.

Thanks

Daniel Mysinger
Dell EMC, Enterprise Engineer

4 Beryllium
5057

FYI 64bit DUP's work Fine on M1000e/M610's. The NICs and/or their firmware have an issue that DELL has NOT fixed. You supposedly have to do them individually booting from usb flash or cd/dvd. Ridiculous :(

2 Bronze
2 Bronze
5057

BEWARE UPDATE 8.07.25!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

We have a similar configuration as the gal above:
(16) X M610
a - m6220
b - m8024
C - m8024-K

Each Blade has
A - LOM (BCM5709S)
B - BCM57711
C -BCM8242-K 

We ran the DELL update on half our chassis (top 8) using the recommended 08.07.25 update (from 7.12.19) and the FW update bricked all 8 cards. We were blown away. Our production environment was down for two days until we borrowed 8 BCM57711's from another company. 

We then tried it using the DUP (from Dell DRM) and it did the same thing to card #9 (non-usable).

So it is obvious that this update IS NOT COMPATIBLE with the BCM57711 cards and it fails them permanently. DELL -You need to publish something on your website about this and get your updates in order. DELL -This is going to cost me over $3K in replacement hardware.

As a test, we installed WinOS to bare metal (on another M610 and card #10) and ran the Dell Windows x64 update packages from the product website. We updated on older 7.12.17 to 7.12.19 (Network_Firmware_35RF5_WN64_7.12.19.EXE) successfully. When we tried running the latest version 08.07.25 (Network_Firmware_WR5R5_WN64_08.07.25_A00-00_01.EXE) from the Dell website, it read "No products compatible with this update package".  The card was spared from devistation. Obviously a huge disconnect here DELL. I wonder how many customers are out major hardware for this SNAFU. This is obviously incompatible FW for the BCM57711 but all your update methods, except for the Wx64 OS, tries to push it out as valid effectively "bricking" the HW.

4 Beryllium
5057

Wonderful....... I do in fact have 1 blade (slot 15) out of my current 10 in this particular chassis that does have version 8.7.25 but I have not deployed this server to try NIC workability. (no clue how this ONE blade has this newer firmware though) Slot 15 and 16 were purchased as a matching pair....

Firmware-wise on other blades, I have been using the SUU iso and connecting via Virtual Media Wizard and using UEFI and the NIC updates on SOME blades still will not upgrade their firmware....

Slot15 and 16

Fabric A Gb Ethernet Embedded LOM

Fabric B Gb Ethernet Broadcom M5709t

4 Beryllium
5057

Here is the screenshot of another (Slot 2) that I need to try updating yet again.... Look at the version it wants to update the LOM to....

4 Beryllium
5057

Wow.... I let the above run, and it updated the LOM's to 8.7.25 as listed, but the Mezz NICs did NOT update. I ran the SAME SUU update and now we have: really? same SUU updated backwards to 7.12.19???

4 Beryllium
5057

and here we go again, blade 4, SAME blade hardware using SUU from UEFI:

Latest Solutions
Top Contributor