Start a Conversation

Unsolved

This post is more than 5 years old

8458

October 25th, 2017 10:00

R610 Lifecycle Controller/IDRAC Unable to Update

I've got a hard one for you guys here. I've been fighting with this thing for the past two weeks to get the IDRAC/LCC firmware updated to no avail. I'd really rather not have to replace the thing if at all possible (I got it for a deal, used, but everything was 'working' when purchased - just outdated), but I need a definitive answer.

TL;DR:
iDRAC6 version 1.10.00 (Build 13), LCC version 0.0.0.0
Server 2008 R2 LCC/iDRAC DUP fails
Server 2012 R2 LCC/iDRAC DUP fails
CentOS 6 LCC/iDRAC DUP fails
Dell Repository Manager bootable linux ISO fails
Uploading Firmware through iDRAC GUI times out/logs me out of web session
POST has "A System Services Update is Required" (can't press F10)
BIOS, NIC, SAS, backplane, etc all update fine

Full version:

Microsoft recently decomed a bunch of poweredge R610s they had used in their datacenters, and I was able to pick one up for $100 (two procs, 12GB, iDrac enterprise, who could say no?) Thing boots just fine, F10 error not present, has all original, non-updated firmware from 2009. Obviously this means I can't use the remove KVM/console, due to recent security updates in Java. I figure "well, we have PE1950s, PE2950s, a PER810, and I've updated all those - how hard could an R610 be?" So, I hit up Dell's support website and download some firmware. I throw Server 2012 R2 on it at first, and run some updates. Everything goes fine, until the iDRAC/LCC. I believe the error on S2012R2 was "BMCFWUW.exe has stopped working" and the iDrac update tab showed "a firmware update is in progress, please try again later". I said OK, and came back hours later to see it had cleared itself. Try again, same problem. I do some reading, and move to stage 2.

I'd never used the LCC on a server before, but I heard it can do some cool things so I figured I'd try it that way. I booted the server into the LCC menu (F10), and it sat there for a solid 45 minutes with the Dell logo on the screen. Eventually the server rebooted itself, booted back into the F10 menu (for far less time), then rebooted with the "System Services Update is Required" message. That didn't sound good, so I started searching and found just how far the rabbit hole can go. It was a learning experience (we don't use dell servers that often, this was purchased for personal use), so I got to use the SBUU, SUU, Repository manger, and bootable linux for the first time, but each was unable to solve the problem.

I've tried:
Running DUPs from Server 2012 R2
- This resulted in BMCFWUW.exe has stopped working, server reboot

Running DUPs from Server 2008 R2
- This resulted in "Failed to access Virtual USB Device", NO "Firmware update is in progress" message

Running racadm racreset soft
Running ipmitool raw 0x06 0x05 0x64 0x52 0x45 0x4D 0x24 0x73 0x74 0x38 and ipmitool raw 0x30 0xac 0x00 0x00 0x00 0x00 (as seen here)
- Result of first command was nothing (blank line), result of second was 0x00 0x00

Running the update that comes immediately after 1.10.13 on 2012 R2, 2008 R2, and CentOS 6 (believe it is 1.3x, as seen here)
Running updates with BIOS 1.3.6 from 2012 R2 (as seen here)
Draining flea power between attempts (holding power for 20 sec, unplugged)
Resetting the NVRAM via onboard jumper
Reseting BIOS and iDRAC (num/caps/scroll lock, CTRL+E, F, B)
Running linux live ISO from USB (had to use the re-mount trick, ALT+F2, mount /dev/sdx /opt/dell/toolkit/systems, run applybundles.sh manually)
- This resulted in some packages missing errors, later addressed by doing a full install of CentOS to install the dependencies

Running DSU from CentOS with required packages
- This resulted in, again, "Unable to access virtual USB Device", even though I see the "scrtch" device mounted (though its contents are empty)

Uploading the firmimg.d6 file to the controller through the web interface
- This results in a 10 minute+ upload session in IE (tried on Server 2012 R2, 2008 R2, and even 2003 R2), and a quick session in Chrome/Firefox, but all end with by browser headed back to the login screen, and none of them actually complete the upload (progress bar gets about halfway in IE, and Chrome/FF between 13%-89% depending on the file)

Uploading BDF_1.5.2_BIN-7.usc / USC_1.2.0_Rep_Pack_A00.usc / USC_1.7.5, and another, newer BDF
- This has the same outcome as the firmimg.d6 files

I've tried downloading the Dell Support Live Image Version 2.2
- This results in 404, file not found using both the browser download and download manager download, link here

Probably a few other solutions as well that I can't remember - I think the only one I haven't gotten to work is the SBUU / SUU combo, as I didn't have a CD large enough to store the images until now

Honestly I don't know what else to do. The server obviously has no warranty on it anymore, and I'm not sure which part needs replacing if it comes to that. I'm currently setup with a triple boot system, so I can get into 2012 R2, 2008 R2 and centOS without having to re-install, and try any fixes you guys might know of. I have probably close to every iDRAC6 firmware version and 11G BIOS version, and can flash/attempt flash through any OS to get some results for you guys. If you know of anything else that I can try, please tell me. I feel like of dump going through all this just to get remote console working on the newest version of Java, but I'm an IT guy, and it's my job to fix things. I'd like to fix this.

4 Posts

October 25th, 2017 13:00

Hi Daniel,

The firmware update proved gave me the error "Device: iDRAC6, Application: iDRAC6 Failed to access Virtual USB Device"

The volume "scrtch" was mounted during the update, on letter F:

Additionally, the following event log error was logged:

Dell - System Update, ID 6049

Update failed
Package: \ESM_Firmware_9GJYW_WN32_2.90_A00.EXE - DENEB1A\uts
Description: Dell iDRAC Monolithic Release 2.90, A00
Log file: C:\ProgramData\Dell\UpdatePackage\log\\ESM_Firmware_9GJYW_WN32_2.90_A00.txt
Exit code = 1 (Failure)

The log in the log file section says the following:

===============> Update Package application started <===============

Command: C:\Users\uts\Desktop\ESM_Firmware_9GJYW_WN32_2.90_A00.EXE

Date: 2017-10-25 19:55:24

=========================================================================


Release ID: 9GJYWA00

Update Package version: 14.11.201 (BLD_248)

User: DENEB1A\uts

Collecting inventory...

Running validation...

The version of this Update Package is newer than the currently installed version.
Software application name: iDRAC6
Package version: 2.90
Installed version: 1.10

Executing update...

Device does not impact TPM measurements.

Device: iDRAC6, Application: iDRAC6
Failed to access Virtual USB Device


==============================> Update Result <==========================

Update was not applied

=========================================================================

Exit code = 1 (Failure)

2017-10-25 19:58:39

===============> Update Package application started <===============

Command: C:\Users\uts\Desktop\ESM_Firmware_9GJYW_WN32_2.90_A00.EXE

Date: 2017-10-25 19:58:39

=========================================================================


Update Package finished. Exit code = 1


2017-10-25 19:58:39

Moderator

 • 

6.2K Posts

October 25th, 2017 13:00

Hello

If you are having any issues with the iDRAC you should resolve those issues before troubleshooting the LCC. The LCC requires the iDRAC to function.

Attempt this update from Windows 2008R2:

www.dell.com/support/home/drivers/driversdetails?driverId=9GJYW

Thanks

Moderator

 • 

6.2K Posts

October 25th, 2017 14:00

Is the d6 file you used previously the 2.9 version on the download page I linked?

I would shut down and disconnect power from the system. Open the chassis and look for a rectangular black plastic item about 2 inches long on the system board. That is the iDRAC Express module. You should be able to carefully remove that item and reseat it.

If the issue persists after reseating the iDRAC express module then remove the optional enterprise module. If the issue still persists then start taking the system to minimum-to-POST. Any USB or PCIe device could be causing communication issues with the iDRAC. Disconnect unnecessary devices and swap items as best as you can.

Each time you re-attempt the update I suggest testing both update methods, DUP from Win2008 and .d6 from the web interface.

Thanks

Moderator

 • 

6.2K Posts

October 25th, 2017 16:00

Did you attempt to reseat the iDRAC express module per my previous suggestion? If you have another R610 you can also try swapping the modules. According to the pictures I have the iDRAC express module should be next to the iDRAC enterprise module.

The LCC requires the iDRAC to function, so the LCC not functioning correctly when having issues with the iDRAC is to be expected.

If the issue is resolved by swapping the modules then you may be able to find the iDRAC express module available for sell somewhere. We did not stock the modules by themselves, we would have had to replace the entire system board to replace the iDRAC express module since it was stocked as a system board component. If the issue persists after swapping the iDRAC express module then you will likely need to replace the system board.

Thanks

4 Posts

October 25th, 2017 16:00

Hi Daniel,

The firmware was indeed the one you linked, the DUP and the .d6 file. The server is still pretty bare bones, so there isn't much to remove. The minimum to post lets me take out only the enterprise iDRAC module, after that nothing else.

Uploading through the web page still times out, and the DUP results in the "Virtual USB Device" error.

Additionally, I took a look at another R610 we have here that doesn't have this problem (also freshly purchased) and when entering the iDRAC menu (CTRL+E), it displays both the iDRAC firmware and the lifecycle controller firmware. The server we're troubleshooting does NOT display any lifecycle firmware on the iDRAC page, or on the Server Administrator application.

4 Posts

October 25th, 2017 18:00

Hi Daniel, 

I did try reseating the module, but it didn't change anything. After reading your post, I went and got a new server from the same vendor, and this one has an active lifecycle controller and shouldn't present an issue. Can you provide any insight as to why this happens, or how this can happen?

No Events found!

Top