PowerEdge T110 Continuously Reboots

Nextech,

You will probably have to reflash back to v.1.30 and then stair step the update to version 1.92.    Using our 32 bit diagnostics package to create a bootable USB key,  you can download each one of these updates, rename the file to a 8 character file and copy to the root of the USB key.

Download our diagnostic package and install on your PC:  

http://ftp.dell.com/diags/DELL_32-BIT-DIAGNOSTICS_5154-1_R302859.exe

Boot the server with the USB key to the 32 bit diagnostic menu

Choose option 4 to exit the menu

You will be at a DIAG>D: prompt

Change to DIAG>C: 

Remember what you have renamed each update file and run.  Then restart the server and run the next update till you have version 1.92

Here are the links for each update:

BMC Firmware v.1.30: http://www.dell.com/support/drivers/us/en/19/DriverDetails/Product/poweredge-t110?driverId=3NMWR&fileId=3076002098&osCode=WS8R2#

BMC Firmware v 1.50: http://www.dell.com/support/drivers/us/en/19/DriverDetails/Product/poweredge-t110?driverId=NY0XJ&fileId=2731091954&osCode=WS8R2#

BMC Firmware v 1.59: http://www.dell.com/support/drivers/us/en/19/DriverDetails/Product/poweredge-t110?driverId=15PT7&fileId=2731094580&osCode=WS8R2#

BMC Firmware v 1.92: http://www.dell.com/support/drivers/us/en/19/DriverDetails/Product/poweredge-t110?driverId=G03FR&fileId=2731098720&osCode=WS8R2#

Let us know how it goes.

Regards,

Make sure any external devices are removed prior to doing the following steps:

Here is a link to the online manual: ftp://ftp.dell.com/Manuals/all-products/esuprt_ser_stor_net/esuprt_poweredge/poweredge-t110_Owner%27s%20Manual_en-us.pdf  page 22,   Clear the NVRAM on the board;   pull the power cord prior to moving the jumper.  Move the jumper, put the cord on. Then remove the power again, move the NVRAM jumper to its previous position.    Then power up the server. 

Let us know the outcome.

Geoff P,

Thank-you for your help!!  We've run into a few problems...

We ran out and bought a 2GB USB flash drive (because the Dell System Diagnostics software won't install/work on anything larger than a 2GB USB Flash drive (and nobody in the office had anything that small).  So we purchased a 2GB Flash drive, we created a bootable USB stick (using the Dell Diagnostic software) and then we moved/copied over the expanded/extracted BMC firmware files/folders.  We renamed everything properly (seven characters or less), just like we normally would when doing a firmware update for one of our other Dell servers.

The problem seems to be that this PowerEdge T110 will NOT boot from a USB stick anymore.  It was ALWAYS booting directly from USB (because we had ESXi 5.5.0 installed on a 8GB Kingston USB stick which was inserted inside the PowerEdge T110 and was installed directly in port 1 of the two onboard internal USB ports.  

The PowerEdge T110 was ALWAYS configured to boot directly from this USB stick.

Unfortunately now, after the BMC flash, the system just continuously reboots.  We did create the new 2GB USB firmware flash drive and tried to boot from it, but we've been unable to get the system to boot from a USB stick.

The Dell splash screen comes up (when it's powered on) a few lights on the front of the system blink (1,3,4) and I'm guessing those are some type of Dell Error Codes?  (based on the numbers that are flashing and turning on, staying on, changing, turning off, staying on, etc.)

Then it does a POST-UP (for the most part) and gets to the point of where it checks the BMC and says something like:

"Press for Remote Access Setup within 5 sec... "

Then it counts down, and after 5 seconds it simply shuts the display off and reboots the system.  This happens continuously.

We pressed "CTRL-E" and did go into the Remote Access Setup, and tried to select "Reset to Default" and also set "IPMI Over LAN" to OFF, and we did everything we could to disable the BMC.  Hoping that maybe this would help, and we could actually boot the system up again.

But as of right now (and during the past 20+ hours) the system will do nothing but continuously reboot.

Out of maybe 2,000+ reboots, we did somehow manage to get the system to boot from a USB stick ONE SINGLE TIME.  (We don't know how/what/why)  It just somehow appeared with a dos prompt, and that is how we managed to type "bmcfwud.exe" at the prompt, and it said that it completed successfully, and said that we needed to type "bmcfwud.exe" again after the reboot, but when the system rebooted it went directly back to the same exact thing of the continuous never-ending reboot.

It posts up, gets to the "Press for Remote Access Setup within 5 sec... " prompt, it counts down to zero, then the power cuts off (display shuts off) and it reboots the system again.

This has been going on for over 20+ hours now, and we're just not sure what to do next, or how we can even get the system to stop rebooting and to actually BOOT from a USB flash drive.

We did go out and buy a 2GB flash drive, we did install the Dell 32-bit diagnostic software on a separate PC, we did use the software to create a new formatted 2GB bootable USB flash drive (with the dell/dos commands).  We also copied the firmware files (that we had already downloaded and extracted, and renamed).  We used 7 character (DOS compliant) folder/file names.  

We tested the stick to make sure that it is in fact bootable (yes it is bootable and yes it works).  The problem is that we can't get the Dell PowerEdge T110 to boot from the USB stick!

We stick the USB stick into the Dell PowerEdge T110, and then it doesn't seem to do much of anything other than Post-up, and then it gets to  the "Press for Remote Access Setup within 5 sec... " prompt and then shuts the display/monitor off and then power cycles (reboots) the machine.

Same thing over and over and over and over again.  Any ideas as to how we can get this to stop, and actually boot from the USB stick?

If we press F2, or try to enter setup, or press F11 (to try and set the boot menu), it says that it will go into the menu, but it never does, it just finishes with the post-up and then gets to the "Press for Remote Access Setup within 5 sec... " and then just reboots the machine.

Any ideas as to what to do/try next?  :(

Thanks for all your help,

Geoff,

Thanks for the manual!!  We figured out that the system lights that are coming on, mean that there is a "Possible system board resource and/or system board hardware failure" (caused by the bad BMC flash).

Lights 1, 3, and 4 turn on when the system is first powered on.

after a few seconds, the lights 1, 3, 4 shut off, and then light number 3 turns on for about seven seconds, and then lights 1,3,4 turn on for about two seconds. (Which again, is possible system board hardware failure).   Then lights 2, 4 turn on (Possible video failure?) for about 3 seconds, then all four lights flash for one second, and then lights 2,3,4 turn back on for about 7 seconds as the system begins to boot up (and the Dell Splash screen appears with the yellow indicator bar showing that the Dell system is loading up something (and it shows the PowerEdge T110 system Bios version as 1.10.0).  It has been continuously rebooting for about 28+ hours straight now.

Here is a video to show you EXACTLY what it is doing:  https://www.dropbox.com/s/h4odlf5hl435uza/2014-01-29%2020.01.02.3gp

After about fourteen seconds, the Dell Splash Screen finishes "loading" (the yellow loading bar indicator) and then it begins to post-up, and shows the processor, Broadcom eXtreme Ethernet adapter, the Dell Serial ATA AHCI BIOS version (1.1.0), and then shows that no devices (Ports A : D) and only the TSSTcorp DVD drive is found (Port E:) and then says "AHCI BIOS Installed" and then the system continues to post up, and show the BMC Configuration Utility (1.61) and BMC Firmware Revision version 1.30.05, and then it lists the IPv4 settings, and then says:  "Press for Remote Access Setup within 5 sec... "

Then after 5 seconds (it counts down with about 6 dots), and then the monitor/display shuts off, and it appears that the system begins to power-cycle (reboot).  The whole exact same process starts over again (and again and again and again).

I will download the manual, and follow the steps that you have outlined above (clear the NVRAM, pull the power cord prior to moving the jumper, move the jumper and then wait a few seconds and re-insert the power cord).  Then wait a few seconds and remove the power again, and then move the NVRAM jumper back to its previous position, and then power up the server.

Hopefully this will get it to boot up.  :)  :)  :)

I'll post more details later tonight after we see what happens.

Geoff,

After following your instructions (about clearing the NVRAM), the system actually booted/rebooted when it was powered back on, and surprisingly it actually booted from the flash drive.

I watched the screen, and it turned blue and it looked like it was running some diagnostic tests (probably some form of Dell diagnostics that were on the USB flash drive), then it looked like it was checking/testing the system RAM, and then checking/testing the serial ports, and it seemed like it was checking/testing everything in the system, and after a few minutes/hour, the screen started flashing different colors (it went black at first, and then white lines started appearing/flashing, and then colors such as RED, BLUE, GREEN started flashing, and it seemed to do it's thing for a few more minutes/hours, and then when all of that was done, it looks like it rebooted again, and did a few more things, and then when it was all done, a black blank screen with white text appeared that said "Displaying (the end) of test result log: auto.txt" and just below that text it said "Strike a key when ready..."

After we pressed the space bar, the screen changed and all kinds of text appeared on the screen (VESA, yada, yada, yada) and at the very bottom it was red and it said "Test Results: FAIL".

After skimming through the log (that was on the screen) and sifting through everything (about 657 lines of text), it seems that EVERYTHING had a "PASS" except for the following:

** SATA DVD+RW 0 - Confidence Test **
No additional sense information

Test Results : Fail
Error Code 0F00:133C

I've attached a copy of the auto.txt file.  

In the log file it said "Test Results: Fail, Error Code 0f00:133C, Msg: DISK - No suitable disk media is present"

I at least had a DOS prompt after the test had completed, and I had also copied the BMC firmware to the same USB flash drive, but unfortunately I thought maybe that everything was ok (after seeing the system test information say "PASS" on everything), so I rebooted the machine and it's back to the never-ending endless rebooting.

It's been rebooting for three more days now (continuously) and I can't seem to get it to boot from the USB Flash drive again.  I'll give it a few more days of never-ending endless reboots, and hope that maybe it decides to boot off of the USB drive (at least once) and possibly run the Dell Diagnostic tests again (for a second time) and hopefully once it's done, I'll get a DOS prompt and then maybe I can try to reflash the BMC from the DOS commandline.

But as of right now, it's just continuously rebooting again (for the past few days).  Haven't had any luck trying to get it to boot from a USB flash drive.

Geoff,

We've watched it reboot probably close to 7,000+ times during the past few days, and we've only managed to get it to boot from the USB flash drive ONE single time.

I just happened to walk into the room and I saw the screen flashing different colors (which means it must have booted from the USB flash drive and started running diagnostics).

I did copy the auto.txt file to a local computer, and did post the results of the auto.txt file in the message/post above.

The test results were "FAIL" (in red letters).

Boot the server with the USB key to the 32 bit diagnostic menu

Choose option 4 to exit the menu

You will be at a DIAG>D: prompt

Change to DIAG>C: 

Remember what you have renamed each update file and run.  Then restart the server and run the next update till you have version 1.92

Yes, I should have actually typed "C:" and then typed the BMC firmware update command (BMCFWUD.EXE) and let the BMC update run, but unfortunately we rebooted the machine after copying the auto.txt file test results (before trying to run the BMC firmware update), because we were assuming that everything was ok with the system, but then after we rebooted the server we realized that it was stuck in the never-ending endless reboot cycle again.

We're hoping that during the next day or two, it will/may boot off of the USB stick again, and hopefully that will give us an opportunity to run the BMCFWUD.EXE command to update the firmware, but if I remember correctly it says to run the BMCFWUD.EXE command (and then reboot the machine and then run it a second time within 20 minutes?)  What happens if we run the BMCFWUD.EXE command and then it starts rebooting continuously again?  (and we're unable to run the BMCFWUD.EXE command a second time within the 20 minutes due to it continuously rebooting?)

Is there any way to break the continuous rebooting cycle?

Thank-you for all of your help,

Thanks for the log file;  I reviewed it and the only error is from the DVD not having media in the tray when the test was ran.  Otherwise, everything else passes. It appears the BMC flash most likely tanked the board.  Not knowing what the previous version was prior to updating, I can only say that it may have been too great of gap between versions.  Some of the BMC updates required intermediate updates or this could happen.  I have seen bios flashes do the same thing for the same reason.  We use the general rule of thumb if its more than 2 updates out date, then its best to stair-step the updates.

Regards,

It appears the BMC flash most likely tanked the board.  Not knowing what the previous version was prior to updating, I can only say that it may have been too great of gap between versions.  Some of the BMC updates required intermediate updates or this could happen.  I have seen bios flashes do the same thing for the same reason.  We use the general rule of thumb if its more than 2 updates out date, then its best to stair-step the updates.

The BMC version on the machine was/is 1.30.05 and we attempted to reflash the system to the latest 1.92 BMC version, which supposedly was the "critical security" update.  

So the mainboard is pretty much toast?  No way to fix it?  

Geoff,

Here is a copy of the auto.txt file (the results of the Diagnostic tests running).

Here:  https://www.dropbox.com/s/bodbx3iphvoltg9/AUTO.BAK.TXT

Clearing the NVRAM didn't seem to do much.  It's still rebooting continuously.  :(

Let me know if there is anything else that you can think of doing/trying to get it to stop rebooting continuously.  

Thank-you for all of your help,

The SOLUTION to the problem was to power the system off.  Unplug the power.  Unplug the USB Keyboard, unplug the USB Mouse, and Unplug ALL USB Devices from the system.  Then turn plug the power back in and power the system on (without ANY USB devices connected to the system).

You'll get a keyboard error (because no keyboard is connected) upon bootup, but the system will finish booting and get past the "Press for Remote Access Setup within 5 sec... " problem/error (where the system normally reboots).

After the system finishes booting, you'll probably have a "Strike F1 to retry boot, F2 for system setup, F11 for BIOS boot manager".  Then just re-plug the USB Keyboard into the server and press "F2" to enter the BIOS setup, and reconfigure your BIOS.  Save the settings, let the system reboot and it should work!

Hope this helps!

I found the solution here:

http://en.community.dell.com/support-forums/servers/f/956/p/19445447/20514397.aspx

(Mike Fink had this same exact problem and unplugging the USB Keyboard is what solved his problem and this same solution worked for me as well)

This is why Dell and many other are a Substitute character removed as per TOU>. They are Profane word removed as per TOU>. Worse we are Profane word removed as per TOU> for purchashing such Substitute character removed as per TOU>.

Substitute character removed as per TOU> is Dell attempt to edit the word  Substitute character removed as per TOU> what Profane word removed as per TOU>