nextech
1 Copper

PowerEdge T110 Continuously Reboots

Jump to solution

Our PowerEdge T110 has been running flawlessly and everything was fine.

Earlier today, I was searching the Dell website and noticed that there was a security notice, and there was an "urgent" PowerEdge T110 BMC 1.92 Firmware update.  (Critical Security Update BMC firmware 1.92, A00)

https://www.dropbox.com/s/g60sowuf1m1yb2v/Screenshot%202014-01-29%2019.30.60.png

https://www.dropbox.com/s/g60sowuf1m1yb2v/Screenshot%202014-01-29%2019.30.60.png

We downloaded the files from the Dell website here: 

http://www.dell.com/support/drivers/us/en/19/GetDriver/OtherFileFormats/Product/poweredge-t110?driverId=G03FR&FileId=3076002098&osCode=W12R2

We selected the driver files we would like to download here:

 https://www.dropbox.com/s/i1ycxr7k30mpv6z/Screenshot%202014-01-29%2019.31.23.png

The installation instructions seemed pretty simple:

Installation Instructions

1. Click Download Now to download the file.
2. When the File Download window appears, click Save to save the file to your hard drive.

Installation
1. Browse to the location where you downloaded the file and double-click the new file.
2. Read over the release information presented in the dialog window.
3. Download and install any prerequisites identified in the dialog window before proceeding.
4. Click the Install button.
5. Follow the remaining prompts to perform the update.

We followed the instructions, and then after the download completed, we ran the program/update package (on Windows Server 2012 R2), and it said that it completed successfully and the system needed to reboot.

After the system rebooted, the Dell splash screen (with BIOS version) comes back up then it shows the BMC version as 1.30.05 (not the 1.92 updated BMC version which supposedly was the update) and now the computer continuously reboots.

It seems that something went wrong with the BMC firmware update, the BMC firmware was not properly updated to 1.92, and instead our Dell PowerEdge T110 now just continuously reboots now.  

Is this a bad firmware/security update?  Any ideas as to how this can be fixed?

Is there a way to rollback the BMC firmware update back to the previous version?  Or a way to fix/get the system to boot normally?

0 Kudos
1 Solution

Accepted Solutions
nextech
1 Copper

RE: PowerEdge T110 Continuously Reboots

Jump to solution

Geoff,

You won't even believe this...

I was 100% convinced that the motherboard was DESTROYED.  I had already placed an order for a new motherboard and I was doing a search on the internet, looking for OTHERS that seemed to have this same exact problem (system gets to the point of "Press <CTRL-E> for Remote Access Setup within 5 sec" but then continuously reboots)

I can't enter the System BIOS, but I can press CTRL-E and enter into the Remote Access Setup (BMC).

It sounds completely crazy, but I found this thread here where "Mike Fink" was having the EXACT same problem:

http://en.community.dell.com/support-forums/servers/f/956/t/19445447.aspx

I've read THOUSANDS of users having this EXACT same problem, but there didn't seem to be ANY solution.  Dell seems to say the exact same thing (trying clearing the NVRAM, etc.) but I tried the SOLUTION that the USER figured out (he disconnected ALL USB devices including the KEYBOARD).

Powering off the system and disconnecting the USB Keyboard and USB Mouse is what solved the problem.

I tried the EXACT same thing, and discovered that YES, it was the USB keyboard that was causing the problem!!

The HOW/WHAT/WHY, I'm not sure of, but when I powered down the system and disconnected the USB keyboard & mouse and then rebooted the system WITHOUT any keyboard and mouse... a Keyboard error popped up, but then the system continued booting and it booted just fine! (without the keyboard and mouse)

I couldn't believe it!!

A USB Keyboard was causing the problem!??  I have no idea how/what/why this is the SOLUTION to the problem, but sure enough it actually worked!!

I would suspect that some device is the issue and taking it to the minimum configuration (removing all cards, drives, memory but one, USB and all peripherals (including mouse, keyboard - or try different ones)) will isolate that device.

I was reading the thread found here:  http://en.community.dell.com/support-forums/servers/f/956/t/19445447.aspx

After I unplugged the USB keyboard, it finished booting (past the "Press <CTRL-E> for Remote Access Setup within 5 sec..." where it normally just reboots the system).  Sure enough, after shutting the system completely off and unplugging the USB Keyboard, and USB Mouse, and then turning the system back on without any USB keyboard/mouse, it actually worked!!  It actually booted up!

The system booted up (normally) without the keyboard/mouse.  It displayed a keyboard error, but then it finished posting.  It got past the "Press <CTRL-E> for Remote Access Setup within 5 sec..." problem/error (where it normally rebooted the server), and it finished booting up and then after booting it finally came to a screen that finally said:

Strike F1 to retry boot, F2 for system setup, F11 for BIOS boot manager

I was dancing for joy, and couldn't believe the system actually finished booting up (just because I removed the USB keyboard/mouse).

I plugged the USB keyboard back in and then I pressed "F2" for system setup, and went into the bios, and I set the system time and reconfigured the system (everything was blank after previously resetting the NVRAM).  I also setup the boot priority in the boot manager.  (Set it back to booting from the USB device).  Then everything seemed to work (as normal) and the system now works and boots normally!!

I found the SOLUTION by doing a Google Search, and stumbled upon a thread by Mike Fink.

Mike Fink seemed to have this 100% EXACT same error/problem here:  http://en.community.dell.com/support-forums/servers/f/956/t/19445447.aspx

I just got really lucky and stumbled upon his thread, while doing a Google search on " Press <CTRL-E> for Remote Access Setup within 5 sec "

Mike Fink's thread came up, it described the EXACT same problem that I was having and I read his thread and read the solution and realized that he was having the EXACT same problem that I was having, and he said that disconnecting his keyboard fixed the problem.  So I gave it a shot and disconnected my keyboard and mouse, and sure enough it actually worked!!

I can't believe it worked!!

I don't really know WHY on earth the USB keyboard would have been causing the problem (it's a brand new Dell keyboard that I've been using with this system for months now) and the USB keyboard works (because I can press "CTRL-E" to enter into the Remote Access Setup) but for some reason by just shutting the system completely down and removing the USB keyboard/mouse and booting the system without ANY keyboard/mouse it gave the keyboard error and then for some reason it allowed the system to continue to boot normally (and fixed the error/problem of not being able to get into the BIOS and the continuous rebooting).

I was pretty shocked, but the SOLUTION to the problem is to just power the system down and remove the USB Keyboard/Mouse.  Power the system back up (without any keyboard/mouse or ANY USB devices connected to the system) and just let the system boot without ANY USB devices attached to the system and for some reason it ends up booting up just fine/normally.

Once this happens, you can reconnect the USB keyboard and USB mouse, and press "F2" to enter setup, and then go into the BIOS and configure your startup/boot preferences, and everything seems to work normally afterwards.  The system no longer reboots continuously!

After unplugging the USB keyboard/mouse and rebooting the system and then after the system finished booting, pressing "F2" to enter the BIOS and then reconfiguring the system BIOS and rebooting, everything worked perfectly!

I hope that this helps, and hope that someone (having the same problem) stumbles upon this thread in the future, and hopefully they will give it a shot and hopefully this will help someone in the future.

Thanks DELL-Geoff for all of your help, and thank-you Mike Fink and TheFlash1932 for your suggestions, posts and the solution!

I already ordered a new motherboard yesterday (it hasn't arrived yet), but thanks again for the SOLUTION!

 

0 Kudos
13 Replies
Moderator
Moderator

RE: PowerEdge T110 Continuously Reboots

Jump to solution

Nextech,

You will probably have to reflash back to v.1.30 and then stair step the update to version 1.92.    Using our 32 bit diagnostics package to create a bootable USB key,  you can download each one of these updates, rename the file to a 8 character file and copy to the root of the USB key.

Download our diagnostic package and install on your PC:  

http://ftp.dell.com/diags/DELL_32-BIT-DIAGNOSTICS_5154-1_R302859.exe

 

 

Boot the server with the USB key to the 32 bit diagnostic menu

Choose option 4 to exit the menu

You will be at a DIAG>D: prompt

Change to DIAG>C: 

Remember what you have renamed each update file and run.  Then restart the server and run the next update till you have version 1.92

 

Here are the links for each update:

 

BMC Firmware v.1.30: http://www.dell.com/support/drivers/us/en/19/DriverDetails/Product/poweredge-t110?driverId=3NMWR&fil...#

BMC Firmware v 1.50: http://www.dell.com/support/drivers/us/en/19/DriverDetails/Product/poweredge-t110?driverId=NY0XJ&fil...#

BMC Firmware v 1.59: http://www.dell.com/support/drivers/us/en/19/DriverDetails/Product/poweredge-t110?driverId=15PT7&fil...#

BMC Firmware v 1.92: http://www.dell.com/support/drivers/us/en/19/DriverDetails/Product/poweredge-t110?driverId=G03FR&fil...#

 

Let us know how it goes.

 

Regards,

 

Geoff P
Dell | Social Outreach Services - Enterprise


Download the Dell Quick Resource Locator app today to access PowerEdge support content on your mobile device!
(iOS, Android, Windows)

0 Kudos
nextech
1 Copper

RE: PowerEdge T110 Continuously Reboots

Jump to solution

Geoff P,

Thank-you for your help!!  We've run into a few problems...

We ran out and bought a 2GB USB flash drive (because the Dell System Diagnostics software won't install/work on anything larger than a 2GB USB Flash drive (and nobody in the office had anything that small).  So we purchased a 2GB Flash drive, we created a bootable USB stick (using the Dell Diagnostic software) and then we moved/copied over the expanded/extracted BMC firmware files/folders.  We renamed everything properly (seven characters or less), just like we normally would when doing a firmware update for one of our other Dell servers.

The problem seems to be that this PowerEdge T110 will NOT boot from a USB stick anymore.  It was ALWAYS booting directly from USB (because we had ESXi 5.5.0 installed on a 8GB Kingston USB stick which was inserted inside the PowerEdge T110 and was installed directly in port 1 of the two onboard internal USB ports.  

The PowerEdge T110 was ALWAYS configured to boot directly from this USB stick.

Unfortunately now, after the BMC flash, the system just continuously reboots.  We did create the new 2GB USB firmware flash drive and tried to boot from it, but we've been unable to get the system to boot from a USB stick.

The Dell splash screen comes up (when it's powered on) a few lights on the front of the system blink (1,3,4) and I'm guessing those are some type of Dell Error Codes?  (based on the numbers that are flashing and turning on, staying on, changing, turning off, staying on, etc.)

Then it does a POST-UP (for the most part) and gets to the point of where it checks the BMC and says something like:

"Press <CTRL-E> for Remote Access Setup within 5 sec..."

Then it counts down, and after 5 seconds it simply shuts the display off and reboots the system.  This happens continuously.

We pressed "CTRL-E" and did go into the Remote Access Setup, and tried to select "Reset to Default" and also set "IPMI Over LAN" to OFF, and we did everything we could to disable the BMC.  Hoping that maybe this would help, and we could actually boot the system up again.

But as of right now (and during the past 20+ hours) the system will do nothing but continuously reboot.

Out of maybe 2,000+ reboots, we did somehow manage to get the system to boot from a USB stick ONE SINGLE TIME.  (We don't know how/what/why)  It just somehow appeared with a dos prompt, and that is how we managed to type "bmcfwud.exe" at the prompt, and it said that it completed successfully, and said that we needed to type "bmcfwud.exe" again after the reboot, but when the system rebooted it went directly back to the same exact thing of the continuous never-ending reboot.

It posts up, gets to the "Press <CTRL-E> for Remote Access Setup within 5 sec..." prompt, it counts down to zero, then the power cuts off (display shuts off) and it reboots the system again.

This has been going on for over 20+ hours now, and we're just not sure what to do next, or how we can even get the system to stop rebooting and to actually BOOT from a USB flash drive.

We did go out and buy a 2GB flash drive, we did install the Dell 32-bit diagnostic software on a separate PC, we did use the software to create a new formatted 2GB bootable USB flash drive (with the dell/dos commands).  We also copied the firmware files (that we had already downloaded and extracted, and renamed).  We used 7 character (DOS compliant) folder/file names.  

We tested the stick to make sure that it is in fact bootable (yes it is bootable and yes it works).  The problem is that we can't get the Dell PowerEdge T110 to boot from the USB stick!

We stick the USB stick into the Dell PowerEdge T110, and then it doesn't seem to do much of anything other than Post-up, and then it gets to  the "Press <CTRL-E> for Remote Access Setup within 5 sec..." prompt and then shuts the display/monitor off and then power cycles (reboots) the machine.

Same thing over and over and over and over again.  Any ideas as to how we can get this to stop, and actually boot from the USB stick?

If we press F2, or try to enter setup, or press F11 (to try and set the boot menu), it says that it will go into the menu, but it never does, it just finishes with the post-up and then gets to the "Press <CTRL-E> for Remote Access Setup within 5 sec..." and then just reboots the machine.

Any ideas as to what to do/try next?  😞

Thanks for all your help,

0 Kudos
Moderator
Moderator

RE: PowerEdge T110 Continuously Reboots

Jump to solution

Make sure any external devices are removed prior to doing the following steps:

Here is a link to the online manual: ftp://ftp.dell.com/Manuals/all-products/esuprt_ser_stor_net/esuprt_poweredge/poweredge-t110_Owner%27...  page 22,   Clear the NVRAM on the board;   pull the power cord prior to moving the jumper.  Move the jumper, put the cord on. Then remove the power again, move the NVRAM jumper to its previous position.    Then power up the server. 

Let us know the outcome.

 

Geoff P
Dell | Social Outreach Services - Enterprise


Download the Dell Quick Resource Locator app today to access PowerEdge support content on your mobile device!
(iOS, Android, Windows)

0 Kudos
nextech
1 Copper

RE: PowerEdge T110 Continuously Reboots

Jump to solution

Geoff,

Thanks for the manual!!  We figured out that the system lights that are coming on, mean that there is a "Possible system board resource and/or system board hardware failure" (caused by the bad BMC flash).

Lights 1, 3, and 4 turn on when the system is first powered on.

after a few seconds, the lights 1, 3, 4 shut off, and then light number 3 turns on for about seven seconds, and then lights 1,3,4 turn on for about two seconds. (Which again, is possible system board hardware failure).   Then lights 2, 4 turn on (Possible video failure?) for about 3 seconds, then all four lights flash for one second, and then lights 2,3,4 turn back on for about 7 seconds as the system begins to boot up (and the Dell Splash screen appears with the yellow indicator bar showing that the Dell system is loading up something (and it shows the PowerEdge T110 system Bios version as 1.10.0).  It has been continuously rebooting for about 28+ hours straight now.

Here is a video to show you EXACTLY what it is doing:  https://www.dropbox.com/s/h4odlf5hl435uza/2014-01-29%2020.01.02.3gp

After about fourteen seconds, the Dell Splash Screen finishes "loading" (the yellow loading bar indicator) and then it begins to post-up, and shows the processor, Broadcom eXtreme Ethernet adapter, the Dell Serial ATA AHCI BIOS version (1.1.0), and then shows that no devices (Ports A : D) and only the TSSTcorp DVD drive is found (Port E:) and then says "AHCI BIOS Installed" and then the system continues to post up, and show the BMC Configuration Utility (1.61) and BMC Firmware Revision version 1.30.05, and then it lists the IPv4 settings, and then says:  "Press <CTRL-E> for Remote Access Setup within 5 sec..."

Then after 5 seconds (it counts down with about 6 dots), and then the monitor/display shuts off, and it appears that the system begins to power-cycle (reboot).  The whole exact same process starts over again (and again and again and again).

I will download the manual, and follow the steps that you have outlined above (clear the NVRAM, pull the power cord prior to moving the jumper, move the jumper and then wait a few seconds and re-insert the power cord).  Then wait a few seconds and remove the power again, and then move the NVRAM jumper back to its previous position, and then power up the server.

Hopefully this will get it to boot up.  🙂  🙂  🙂

I'll post more details later tonight after we see what happens.

 

0 Kudos
nextech
1 Copper

RE: PowerEdge T110 Continuously Reboots

Jump to solution

Geoff,

After following your instructions (about clearing the NVRAM), the system actually booted/rebooted when it was powered back on, and surprisingly it actually booted from the flash drive.

I watched the screen, and it turned blue and it looked like it was running some diagnostic tests (probably some form of Dell diagnostics that were on the USB flash drive), then it looked like it was checking/testing the system RAM, and then checking/testing the serial ports, and it seemed like it was checking/testing everything in the system, and after a few minutes/hour, the screen started flashing different colors (it went black at first, and then white lines started appearing/flashing, and then colors such as RED, BLUE, GREEN started flashing, and it seemed to do it's thing for a few more minutes/hours, and then when all of that was done, it looks like it rebooted again, and did a few more things, and then when it was all done, a black blank screen with white text appeared that said "Displaying (the end) of test result log: auto.txt" and just below that text it said "Strike a key when ready..."

After we pressed the space bar, the screen changed and all kinds of text appeared on the screen (VESA, yada, yada, yada) and at the very bottom it was red and it said "Test Results: FAIL".

After skimming through the log (that was on the screen) and sifting through everything (about 657 lines of text), it seems that EVERYTHING had a "PASS" except for the following:

** SATA DVD+RW 0 - Confidence Test **
No additional sense information

Test Results : Fail
Error Code 0F00:133C

I've attached a copy of the auto.txt file.  

In the log file it said "Test Results: Fail, Error Code 0f00:133C, Msg: DISK - No suitable disk media is present"

I at least had a DOS prompt after the test had completed, and I had also copied the BMC firmware to the same USB flash drive, but unfortunately I thought maybe that everything was ok (after seeing the system test information say "PASS" on everything), so I rebooted the machine and it's back to the never-ending endless rebooting.

It's been rebooting for three more days now (continuously) and I can't seem to get it to boot from the USB Flash drive again.  I'll give it a few more days of never-ending endless reboots, and hope that maybe it decides to boot off of the USB drive (at least once) and possibly run the Dell Diagnostic tests again (for a second time) and hopefully once it's done, I'll get a DOS prompt and then maybe I can try to reflash the BMC from the DOS commandline.

But as of right now, it's just continuously rebooting again (for the past few days).  Haven't had any luck trying to get it to boot from a USB flash drive.

 

0 Kudos
nextech
1 Copper

RE: PowerEdge T110 Continuously Reboots

Jump to solution

Geoff,

We've watched it reboot probably close to 7,000+ times during the past few days, and we've only managed to get it to boot from the USB flash drive ONE single time.

I just happened to walk into the room and I saw the screen flashing different colors (which means it must have booted from the USB flash drive and started running diagnostics).

I did copy the auto.txt file to a local computer, and did post the results of the auto.txt file in the message/post above.

The test results were "FAIL" (in red letters).

Boot the server with the USB key to the 32 bit diagnostic menu

Choose option 4 to exit the menu

You will be at a DIAG>D: prompt

Change to DIAG>C: 

Remember what you have renamed each update file and run.  Then restart the server and run the next update till you have version 1.92

Yes, I should have actually typed "C:" and then typed the BMC firmware update command (BMCFWUD.EXE) and let the BMC update run, but unfortunately we rebooted the machine after copying the auto.txt file test results (before trying to run the BMC firmware update), because we were assuming that everything was ok with the system, but then after we rebooted the server we realized that it was stuck in the never-ending endless reboot cycle again.

We're hoping that during the next day or two, it will/may boot off of the USB stick again, and hopefully that will give us an opportunity to run the BMCFWUD.EXE command to update the firmware, but if I remember correctly it says to run the BMCFWUD.EXE command (and then reboot the machine and then run it a second time within 20 minutes?)  What happens if we run the BMCFWUD.EXE command and then it starts rebooting continuously again?  (and we're unable to run the BMCFWUD.EXE command a second time within the 20 minutes due to it continuously rebooting?)

Is there any way to break the continuous rebooting cycle?

Thank-you for all of your help,

 

 

0 Kudos
nextech
1 Copper

RE: PowerEdge T110 Continuously Reboots

Jump to solution

Geoff,

Here is a copy of the auto.txt file (the results of the Diagnostic tests running).

Here:  https://www.dropbox.com/s/bodbx3iphvoltg9/AUTO.BAK.TXT

Clearing the NVRAM didn't seem to do much.  It's still rebooting continuously.  😞

Let me know if there is anything else that you can think of doing/trying to get it to stop rebooting continuously.  

Thank-you for all of your help,

0 Kudos
Moderator
Moderator

RE: PowerEdge T110 Continuously Reboots

Jump to solution

Thanks for the log file;  I reviewed it and the only error is from the DVD not having media in the tray when the test was ran.  Otherwise, everything else passes. It appears the BMC flash most likely tanked the board.  Not knowing what the previous version was prior to updating, I can only say that it may have been too great of gap between versions.  Some of the BMC updates required intermediate updates or this could happen.  I have seen bios flashes do the same thing for the same reason.  We use the general rule of thumb if its more than 2 updates out date, then its best to stair-step the updates.

Regards,

Geoff P
Dell | Social Outreach Services - Enterprise


Download the Dell Quick Resource Locator app today to access PowerEdge support content on your mobile device!
(iOS, Android, Windows)

0 Kudos
nextech
1 Copper

RE: PowerEdge T110 Continuously Reboots

Jump to solution

It appears the BMC flash most likely tanked the board.  Not knowing what the previous version was prior to updating, I can only say that it may have been too great of gap between versions.  Some of the BMC updates required intermediate updates or this could happen.  I have seen bios flashes do the same thing for the same reason.  We use the general rule of thumb if its more than 2 updates out date, then its best to stair-step the updates.

The BMC version on the machine was/is 1.30.05 and we attempted to reflash the system to the latest 1.92 BMC version, which supposedly was the "critical security" update.  

So the mainboard is pretty much toast?  No way to fix it?  

0 Kudos