Dell T620 Shutsdown randomly

Hi Chris,

Thanks for chiming in - I got so caught up in testing every bit of hardware and upgrading firmware, that I didn't even think about doing a minimum post.

I should mention, when I first bought the server, I ran the server for a couple of weeks straight with no issues. When I figured out how I wanted to use the server (Promxmox w/ various VMs hosting containers and TrueNAS), I figured it was time to update all Firmware: 
BIOS
IDRAC
Lifecycle controller 
PSU

My last firmware to update was the PSUs. I did 1 at a time and it would take roughly 2 hours before the server powered itself back on at the tail end of the upgrade. iDRAC confirms the firmware upgrades were successful + Firmware version #s match.
After the PSU firmware upgrade, I installed my second CPU and filled the remaining empty 22  x 8 GB DIMM slots. All 192 GB of RAM were detected and memtested. I did a light CPU burn and some other tests to see if that was fine. . Then - a week after the PSU firmware upgrades/hardware upgrades - I've been having this issue.  Last week, I ran through the gambit of tests I mentioned above, all passed, everything green in iDrac etc... 

Both PSUs have a solid GREEN light. In the iDRAC controller, everything is detected and green (when the server is powered on) - with temperatures in normal ranges under 60C. When PSUs are plugged in and powered, the motherboard has a green "+12V AUX" light that is solid on. However, the server is stone-cold dead. When the server does power on, everything POSTs relatively quickly, and all my VMs run just fine (Media server works fine, steam/game server runs great etc etc). No bugs or weird errors. iDRAC is very responsive. 

Last night the server shut down without warning after running for 8 days with no issue. This morning, I got it to run for about 10 mins. I've been trying all day today to get it to turn on. As soon as it does come on, I can screenshot the BIOS and iDRAC versions as well as provide the exact error messages I am getting. 

I am also beginning to suspect that the PSU firmware upgrades did not go as smoothly as they should have. 2 hours is a long time for the upgrade to complete. I have "upgraded" the PSU twice, therefore I don't see an option to roll back to the other PSU firmware either. 

Looks like I should try the minimum requirements for a post... the painful part about this process is that it would take 10 minutes to a week before the server cuts off (this is the 5-6th time this has happened in total). So what may seem to work for a couple of days might be another red herring. Granted - I already had a month's worth in downtime with this server - I should have spent it ensuring a minimum post wouldn't flake out on me.
 
I also order 2 PSUs online as well as a replacement - just in case. 
+ I forgot to mention the replacement power distribution board came with cables, so those cables were replaced. 

Update - 
New PSUs came in. Didn't make any difference. 

Pulled all the RAM out except A1 and pulled CPU2 out. Unplugged both SAS connectors. 
The server won't even post. 
Tried swapping CPU1 and CPU2. Still nothing. Tried a different stick of RAM. Nothing. 

This is my 2nd motherboard and 2nd power distributor board. I have replaced everything EXCEPT the SAS backplane. This is a nightmare. 

Hi Randy,

So basically you have already done checking the OS and iDRAC logs for any signs of shutdown initiation, there is none stated any hints, right? I read you mentioned it shuts down in BIOS, hence that should be hardware issue. 

I've previous done troubleshooting for such cases before, years ago, and the engineer found out it's the chassis that had made some kind of short circuit issue to the mainboard. I wouldn't recommend doing this, but previous for that case, the engineer built the server out of the chassis to check the issue. 

You have unplugged the SAS backplane cables, but the issue persist, so it's not going to be the backplane. How about the chassis control panel where the power button is, could it be the button issue? 

Hey Joey- thanks for chiming in.

Thanks for bringing up shorts and the power button.
I took apart the front bezel and everything *looks* fine with respect to the power button. I haven't examined it as closely yet since the iDRAC is not powering on - and lets me believe a different issue. However, if the chassis control panel is bad - then I can see how the system may try not to power itself on for safety reasons. 

Once upon a time, I tried to install a supermicro motherboard in a server case we had laying around. One bad standoff caused a short between 2 pins for a fan connector. Powering up that server, I smelled, and learned - quickly. Spare screws or anything metal can easily cause a short. With repsect to this T620 server, I have removed the motherboard and power distribution card twice now and both times I've flipped the chassis upside down and gave it a gentle shake in case I missed anything. 

I think I found a solution - 
The first board had a voltage problem - during shipping one of the memory voltage regulator chips popped off. The second board is a mystery for it's failure. I put the first board back in - I get iDRAC and sometimes full boot into OS. Because of that voltage chip, the system is unreliable for obvious reasons - BSOD and kernel panics. But at least I got 'consistent' power~ So, I bought another motherboard from Ebay just in case.  This afternoon, I installed the new-used board. I did the minimum POST. Everything looks good. I loaded up the server with all my RAM and CPU, ran health tests. Still looks good.
Currently, I am flashing BIOS and then iDRAC7 firmware to get me up to date.  
Two bad boards later, I hope this time I am all set. 

Maybe is optimism or a placebo, but I swear navigating through the Lifecycle controller and through iDrac is much much faster with this board. 

I will keep you guys posted if anything else happens. I will also check back in about a month to share a status update. 

1 month later - new motherboard is working great.  

Proxmox, plex/jellyfin, truenas, pihole, nginx, and all the other containers / VMs I've spun up on this hardware is running great.