LifeCycle Controller updates

Question

I have wasted a bunch of time recently attempting to update the firmware on some servers that have been running in production (without a reboot) for almost 3 years. The Lifecycle controller can find the Catalog, but then fails to download any of the update packages. I see there are several previous threads regarding this, and it seems to be caused by the downloads.dell.com switching to using HTTPS instead of HTTP, or via ftp.dell.com.

As the LCC firmware had not been updated, it didn't have HTTPS support. Nice chicken and egg problem.

My first attempt was to route the LCC via squid, where I could see the requests, and that's where I saw that it was resolving downloads.dell.com to a random IP each time, presumably a CDN, and then appending /downloads.dell.com/ on the path.. This of course doesn't work, and trying this manually gives you a "no host" error, which makes sense, since the CDN server has no idea what host you are trying to retrieve from when you are connecting to an IP address, instead of a host name.

Replacing the IP with downloads.dell.com gets slightly closer, but of course there is no path of http://downloads.dell.com/downloads.dell.com/some_file, so you get a 400 error, instead of a 404 (?)

I found someone else had written a URL rewrite script in perl, which would replace the IP with downloads.dell.com, and strip the path, and I hacked this in and added more IP addresses, but this only worked so far as to download the Catalog. I was then back to it putting the invalid double path in, and I couldn't work out how to get the regex to match and strip this.

In desperation, I wondered if I could just setup my own update server, with valid paths, so I installed apache on a host on the same subnet as the server, and instead of pointing the LCC to downloads.dell.com, I pointed it to the IP of the apache server, where it attempted to download Catalog.xml.gz. I wget that manually into the webroot of the apache server, and retried on the LCC, which worked.. The server now showed a list of available updates.

Attempting to download/install any failed of course, however the apache access log showed me what it was trying to download, eg downloads.dell.com/FOLDER05590166M/1/iDRAC-with-Lifecycle-Controller_Firmware_40T1C_WN64_2.63.60.61_A00.EXE I would then use "wget -x" to have wget download and the -x switch makes it recreate the directory structure, eg downloads.dell.com/FOLDER05590166M/1/, and then was able to go back to the LCC complaining that the download failed verification (or similar), which is a nice confusing error since the actual problem is that it got a 404.. I could then tell it to retry, and it would successfully download the file from my local webserver, ie [webserverip]/downloads.dell.com/FOLDER05590166M/1/iDRAC-with-Lifecycle-Controller_Firmware_40T1C_WN64_2.63.60.61_A00.EXE and then move on to the next one, where I would repeat the wget process to get the missing file.

After all this I had the 7 update files on my local web server, and the server has now installed them all.

Dell, what an atrocious mess you have made of something that used to work very well.

Daniel My · Answer

Hello

A couple of easier workarounds:

Update the iDRAC/LCC using a Dell Update Package. You can go to the update/rollback section of the iDRAC web server and upload a Win32 or Win64 DUP from your computer through the browser to update the iDRAC/LCC to add the HTTPS feature. You can also use DUPs through system services(F10).This is the best method if you are only managing a few systems.
http://www.dell.com/idracmanuals/
Use Dell EMC Repository Manager(DRM) to create a local repository. Use that as the update location instead of the online catalog. This is the best method if you are managing several systems.
http://www.dell.com/openmanagemanuals/

Thanks

squigley · Answer

Oh, I should probably specify that this is via the Update/Rollback function in the iDRAC web UI, uploading a local file.

squigley · Answer

I managed to step upgrade the iDRAC/LCC in an R620 some of the way, to 2.30.30.30, but when I try to install the latest version, using either WIN32 or the WIN64 DUP, I keep getting:

Start Time: Not Applicable
Expiration Time: Not Applicable
Message: RED007: Unable to verify Update Package signature.

I have downloaded the EXEs multiple times with no difference.

Daniel My · Answer

The latest packages use hash algorithms older LCC firmware cannot process. You need to update to iDRAC/LCC version 2.40.xx or later to use the latest update packages.

https://www.dell.com/support/article/sln316137/

squigley · Answer

I tried to followup on this last week but keep getting an error that I do not have permission to access
www.dell.com/community/forums/replypage.editorform.form.form.form" on this server.. I don't know if this is a forum or user issue or what..

squigley · Answer

This is what I attempted to post.. OK, the old version of the LCC being unable to handle the hash would explain why the packages fail to verify.

It's still a pretty nasty situation, that:

you can't update via FTP, which is all the old LCC has support for,
you can't update via HTTP after you update to a version of the LCC with HTTP support, since the files that are referred to in the catalog are either missing, the path is wrong, or they are unable to be verified,
you can't update from a local HTTP server since the files can't be verified,
you can't update via loading a local update file through the iDRAC UI since the file can't be verified.

It requires either finding some unknown in between version which will be able to be validated by the running version, and then implement support for the newer hash types..

Or, what I ended up working out/doing, was to unzip the EXE file of the DUP, get the firmimg.d7 file out, put it on a Tee eFf Tee Pee (if I write that protocol as an acronym then this form breaks and I am unable to post, ?..) server, and then SSHing into the iDRAC and telling it to "fwupdate -g -u -a (IP address)". I now see that this can also be done via the iDRAC web UI.

As I said in my first post, this is a pretty atrocious mess, considering I used to be able to just boot into the LCC, go to updates, point it at ftp.dell.com, and it would connect, show me all the updates, and download and install them, without any of this nonsense.

At least now I have a manual workaround.. Use the iDRAC webUI to update the iDRAC using the firmimg directly, which avoids the hash issue, and then gives me HTTPS support to be able to use the normal upload server downloads.dell.com.

Systems Management General

LifeCycle Controller updates

Was this post helpful?