PowerEdge: 14G - BIOS update fails on the first attempt, second attempt works

For servers with greater than 24 days of power on time since the last AC power cycle, the first BIOS update fails because the Intel Management Engine (ME) fails to enter recovery mode for the BIOS update. Upon failure, the server stops at the F1 prompt and requires intervention to continue. A subsequent BIOS update is successful because the BIOS recovery issues a global reset.

In the Lifecycle Controller (LC) log, the error show up like this:

UEFI0305: Unable to update BIOS firmware in the previous boot because of some issues

Dell Update Package (DUP) BIOS updates (whether using the Operating System (OS) or the iDRAC UI) or even using the .efi file fails the first time if the server's power on time is greater than approximately 24 days (without global reset, AC power cycle). This affects all 14G BIOS versions 1.3.6 and below.

Intel (HSD125807) has confirmed that this issue is an industry-wide for all vendors who flash the BIOS in a similar way to our process.

Workaround

BIOS versions newer than 1.3.6 (1.1.5 for C4140) contains the Intel ME code that resolves this issue. However, since the code running in BIOS is used to perform updates, customers have to use a workaround until the BIOS with the Intel ME fix is installed.

There are multiple workarounds available. If the server AC power on time is greater than approx. 24 days, the key to ensure that the BIOS update is successful the first time is to perform a complete AC power cycle (more about this later) prior to attempting to update the BIOS. This is the case until the BIOS with the Intel ME fix is installed (version 1.3.7 or newer for all 14G servers except for the C4140 which is 1.1.6 or newer).

Prior to installing the BIOS DUP package, apply one of the following solutions:

1. Scripted steps

1. Graceful OS shutdown
2. After the server shutdown is complete, run a racadm script that performs the following:
   1. Perform racadm powerup  (racadm -r [IP address] -u [username] -p [password] servaction powerup)
   2. Pause for 5 s (sleep 5 for Linux or timer 5 for Windows)
   3. Perform racadm powercycle (or virtual reseat for module blades: racadm -r [IP address] -u [username] -p [password] servaction powercycle)
3. Install the BIOS DUP

2. Manual steps

1. Graceful OS shutdown
2. After the server is shut down, perform a complete AC power cycle (Remove power cables, press and hold power button for 10 s, reinstall power cables, turn the machine on)
3. Install the BIOS DUP

3. Double reboot:

1. Install the BIOS DUP package twice (the first time may fail. Human interaction is required to bypass the F1 prompt during the POST)
2. Install the BIOS DUP again