As happens more I have inherited a project of a client with limited IT resources and knowledge in house.
My client wanted to bring his R730xd back to newer fw and BIOS but thought he could jump from a very old BIOS into the latest one and hope for the best.
Result: an R730xd which hangs on POST with BIOS Version 2.12.1 on screen with iDRAC IP reporting as ....
However the iDRAC had been configured so it we can still connect to it and try to resolve everything from there.
In the iDRAC the system still reports 1.2.10 (weird, no?)
Tests done:
reset NVRam, remove BIOS battery
update to older BIOS from iDRAC => remains in the scheduled stated, planned reboot is done manually (because we cant get past the first screen in the post)
rollback to older BIOS from iDRAC => BIOS is not visible in the rollback list
Tried disconnecting everything but PerC, kept CPUs and memory in system
Currently the iDRAC reports the following when going into update or rollback:
SUP0108: A firmware update operation is already in progress. Wait for the update operation to conclude and then re-try.
After a few minutes, it goes away and we can access those details again (but the error returns some time later)
iDRAC has been updated to 2.75.100.75 through iDRAC itself.
My idea is that somehow the BIOS update to 2.12.1 is in the backend still running, not showing in the queued jobs, but failing and preventing any other BIOS updates or rollback to be executed.
Maybe racadm can help, but the problem is that a BIOS update gets pushed into scheduled on next reboot and then remains there forever (even after several reboots, complete disconnects from power, etc )
I am now stuck with a problem which i didnt create but want to solve.
The unit is at the end of its 5 years warranty, I don't know if it can be extended or that we can solve this matter in another way
in the end we replaced the whole mobo and the issue was resolved. Probably mobo stuck on BIOS update and didnt find a way to get it out of the BIOS update loop
Can you check POST Code(Overview -> Server -> Trouble Shooting-> Post Code) and Life Controller Logs(Overview -> Server -> Logs-> Lifecycle Logs) on iDRAC and check whether that have any information on failure?
Post code is stuck at 0x6 Multiprocessor initialization
Lifecycle logs:
2021-04-27T14:36:15+0200
PSU0800
Power Supply 2: Status = 0x00, IOUT = 0x0, VOUT= 0x0, TEMP= 0x0, FAN = 0x0, INPUT= 0x0.
2021-04-27T14:36:15+0200
PSU0800
Power Supply 1: Status = 0x00, IOUT = 0x0, VOUT= 0x0, TEMP= 0x0, FAN = 0x0, INPUT= 0x0.
2021-04-27T14:25:29+0200
SYS1003
System CPU Resetting.
2021-04-27T14:25:29+0200
SYS1001
System is turning off.
2021-04-27T14:19:27+0200
RAC1195
User root via IP 192.168.0.186 requested state / configuration change to ServiceModule using GUI.
2021-04-27T14:17:50+0200
RAC1195
User root via IP 192.168.0.186 requested state / configuration change to iDRAC Information using GUI.
2021-04-27T14:14:45+0200
USR0030
Successfully logged in using root, from 192.168.0.186 and GUI.
2021-04-27T14:14:08+0200
USR0032
The session for root from 192.168.0.186 using GUI is logged off.
2021-04-27T14:02:55+0200
USR0030
Successfully logged in using root, from 192.168.0.84 and GUI.
2021-04-27T13:42:35+0200
SYS1000
System is turning on.
2021-04-27T13:42:27+0200
RAC0702
Requested system powercycle.
2021-04-27T13:42:26+0200
SYS1003
System CPU Resetting.
2021-04-27T13:42:26+0200
SYS1001
System is turning off.
2021-04-27T13:42:21+0200
RAC1195
User root via IP 192.168.0.186 requested state / configuration change to Power Control using GUI.
2021-04-27T13:37:41+0200
RED030
Reboot is complete.
2021-04-27T13:37:38+0200
SYS1000
System is turning on.
2021-04-27T13:37:32+0200
RAC0701
Requested system powerup.
2021-04-27T13:37:30+0200
RAC0704
Requested system powerdown.
2021-04-27T13:37:29+0200
SYS1003
System CPU Resetting.
2021-04-27T13:37:29+0200
SYS1001
System is turning off.
2021-04-27T13:37:23+0200
RAC1195
User root via IP 192.168.0.186 requested state / configuration change to Power Control using GUI.
2021-04-27T13:30:35+0200
SEL0004
Log cleared.
2021-04-27T13:30:35+0200
SEL0014
The System Event Log (SEL) was cleared by root from 192.168.0.186.
2021-04-27T13:30:35+0200
RAC1195
User root via IP 192.168.0.186 requested state / configuration change to SEL using GUI.
When connecting through SSH, and I try a racadm fwupdate -s to interupt it says no updates are in progress, but a few minutes later it says a firmware update is in progress. I feel like the BIOS update is somehow starting and not starting at the same time, but I dont know how to interrupt it properly (and probably get the server to boot again normally)
If it isn't allowing you to continue through POST at this point, I'd consider performing a power drain of the system, first. This would be accomplished by shutting the server off, unplugging it from the supply, then holding the power button down for ~10 seconds. You can then attempt to bring it back up as normal.
I'm not familiar with the error code you're seeing, though an image may provide the context needed to provide any further troubleshooting recommendations.
The messages about the failed virtual disks wouldn't have anything to do with an inability to POST, though they could certainly prevent you from booting the OS. I'm not seeing anything in those entries that would stand out as a problem, aside from the virtual disk entries.
I had already shut down and disconnected the system several times during previous testings (and to see if an actual cold boot makes a difference)
I just did it again, including 20 seconds of keeping the power button pressed, just to be sure.
the screen that comes up almost immediately on a warm boot shows the flashed BIOS, the rollback screen doesnt show any BIOS, the overview screen shows the old BIOS.
I put an intermediate BIOS update (to 1.5.6) through IDRAC on the system yesterday and did a warm boot, but it is still sitting in the queue as scheduled, the reboot job is showing as completed.
I will now complete the very cold boot and hope the actual BIOS upgrade job will start now!
after the cold boot, the scheduled BIOS upgrade job to 1.5.6 remains in the queue doing nothing.
But i did notice another thing, in my roll back overview the BIOS is back visible but the backplane 2.20 seems to have gone missing.
I now cleared the jobqueue and started a rollback job on 1.2.10 and hope the rollback actually gets initiated, afterwards I would attempt to go to idrac version that is available below and slowly upgrade through yearly BIOS version differences. (but that will remain a plan until we can solve this)
After the cold boot the BIOS rollback job remains scheduled. I do see my upload of 1.5.6 sitting available now but any BIOS update or rollback job is stuck in the queue, probably because it is stuck on that POST screen?
dendob
1 Rookie
•
20 Posts
3452
0
Posted January 9th, 2022 05:00
in the end we replaced the whole mobo and the issue was resolved. Probably mobo stuck on BIOS update and didnt find a way to get it out of the BIOS update loop