Firmware & OpenManage will not update, BIOS settings will not stick

Hello,

I have a very unique situation in which I am looking for some guidance. I come from a background in HP’s and years of VMware and high level IT experience, but I seem to be a roadblock with the Dell setup I have. I have spent weeks and weeks trying to get things to work right and they just don’t seem too. I want to present my configuration to the tech world, before I contact Dell directly to return my equipment and software. I encourage anyone to please respond to me and provide any insight or suggestions you have.

Here is an overview of my environment

Dell PowerEdge 2950’s III (2)

-          Dual Xeon E5440 Quad Core @ 2.83 GHz

-          32GB RAM FB-ECC DDR2 PC5300

-          DRAC 5

-          SAS Controller Daughter Card

-          Perc 6/I SATA Controller

-          750 watt redundant power supply’s

-          6 NIC’s

• 2 on-board NIC’s (BCM5708C NetXtreme II GigE)
• 2 additional add-in cards each with 2 100/1000 NIC slots (82571EB Gigabit Ethernet Controller)

-          VMWare ESXi 5.5 U1 – Dell Branded

Here are the issues I have been experiencing with my environment:

Since these servers were acquired I have been attempting to update the firmware. Painstaking is not even the correct word to describe this tumultuous task. Following the directions on Dell’s web site exactly, I downloaded the System Update Utility (SUU) and the Systems Build & Update Utility (SBUU). I can boot my servers off of both of these DVD’s, but they do not contain any firmware updates, but contain the option to upgrade the firmware.

I did some research and downloaded the Dell Repository Manager on my workstation, so that I could download the firmware updates. After 3 installs, the only “option” I could get to work was Dell Repository Manager Datacenter Version (1.9). Following these directions exactly, I was able to get a “bundle” created and downloaded for my Power Edge 2950’s.   I downloaded the Linux bundles, because according to Dell’s instructions, that is necessary when installing on a system with ESXi.

To clarify, the name of the bundle I am using is: “System Bundle (Linux) PE2950 v502 – Version B87298.502

Following the documentation for the Repository Manager there are several options for “exporting” the firmware updates, here is what I did and the results:

-          Export to light weight deployment scripts

• This option was useless to me, because you can’t write batch files for Linux/hypervisor, so this did nothing

-          Bootable ISO (Using Linux Bundle)

• This option burned the ISO to a DVD for me, but when booted from it, it had a black “DOS” like screen and took about 3 hours to run through. It didn’t apply any firmware updates as it said that none of them were applicable to my system. This DVD is now in the trash.

-          Export as SUU

• Export as SUU to Directory
  • Again, this is useless, this did nothing. Booting from a USB drive containing this folder and Ubuntu Linux did not work, I wasn’t able to bring up the GUI to scan and deploy firmware.
  • In addition, I copied just the root directory of the exported repository to a USB (FAT32) as well as a DVD, without using any other files. Booted from SUU and inserted each device separately and SUU said it could not find the repository. This is the error I got when I used the USB flash drive: “USB key does not contain SUU repository in the correct format. Unable to proceed.”

• Export as SUU to ISO
  • Again, useless, when I booted from SUU and SBUU as instructed by Dell used those CD’s to get to the firmware update option, it them prompted me for a CD\DVD. I made a 32-bit and 64-bit DVD and inserted both of them, they did not work as the SUU told me it could not find the repository.
  • Tried to make this ISO bootable, using UBUNTU Linux, but again was able to run the update program.

As I have mentioned, I have not been able to successfully update any of the firmware on my Dell servers. Also, when I do reboot them (for VMware updates...etc.) the BIOS settings always reset. After doing a health check and extensive research, this seems to be an issue with this model and the recommendation is to update the firmware on the systems.

In addition, and giving Dell the benefit of the doubt, I did some extensive research and reading through forums...etc. Long story short, I purchased Dell OpenManage for VMware and VSphere. I paid $500 for the 5 host 3-year license model. From what I read and all of the documentation this would integrate with my VCenter and also my Dell branded hypervisor installs to provide hardware monitoring, system info and health, firmware updates, and interface with my servers. Come to find out, none of this is working once I installed the platform, as well as, it said my iDRAC5 cards were non-existent/incompatible because they are not LifeCycle.

Here is the situation with the Dell OpenManage software:

-          Installed the OVA with license as instructed by Dell

-          Created a connection Wizard.

-          Both hosts are registering “Non- Compliant”

• Clicked: “Fix vSphere Hosts for Compliance”
• Vmhost01 says OSMA Status – Update required, have fixed and updated this host 7 times (7 reboots) and its still displaying that. CSIOR status is unknown
• Vmhost02 says OSMA Status – Compliant but CSIOR status is unknown.

I cannot find anywhere how to turn on the CSIOR at boot, I don’t believe it’s available for my model. Also, as I had mentioned, my other host keeps saying updates required, even after several installs and updates.

-          iDRAC License tab is blank, says I do not have any compatible iDRAC cards for licenses. I do have an iDRAC5 (dell Branded) in each system, with the latest firmware.

-          When I go to the “OpenManage Integration” for each host, it displays minimal hardware info, (very generic) and the update IDRAC and update Firmware option are grayed out.

• After more research, it looks like this program is not designed for my model servers as they are too old and firmware updates won’t work because it is not a Generation 11 or 12 system.
• It is displaying IP’s and DRAC info, but it won’t let me monitor my power supplies, and under “Host Actions: Run Firmware Update Wizard and iDRAC Reset are grayed out.

When I click on my cluster in vCenter and then click “OpenManage Integration, I can see both my hosts, with the service IP, DRAC IP, OS version…etc. There is an option at the top, which is not grayed out, that says: Update Firmware. When I click that, it lets me start the wizard. I get to the second page and it says:

-          Last Inventory Run: July 11, 2014 at 8:31:59 (GMT-4), however nothing displays. Also using the vSphere web interface, the Firmware Update option is available, but doesn’t work.

My conclusion is that I have wasted a ton of time and money on these systems and software, they prove to be very cumbersome. I run a small business and cannot afford to spend $10-$12k on a new PowerEdge servers. I have always worked with HP’s in the past and never had an issue like this, especially with older hardware not being supported. I can’t believe I have had this much trouble, and can’t update firmware on my servers as well as the BIOS settings resetting.

I was to thank anyone who has taken the time to read this, I apologize that it’s quite lengthy, but I feel it’s necessary to be able to provide as much information as possible. Anyone with any suggestions and insight, please post/comment back to me.

Here is my plan of action for the next couple of days, blow out and reinstall the hypervisors and VCenter to see if that can fix any issues. If I don’t get a response on here or any other assistance from dell, I am going to return my software and sell these servers online/eBay and purchase some HP ProLiant’s. Any suggestions, please let me know.

Thanks,

Greg