Hello,
I'm sorry to hear that. If the result is the same, especially when you try it with the spare PERC, there may be hardware failure. You can try a few things that come to my mind. At first, I see the latest version of the BIOS as 2.9.0, if it is not this version, you can try to update it. https://dell.to/3hHMMaB
As the easiest step, you can first power drain the server.
Power the server down
Disconnect server from all power cables, Network cables.
Hold down the power button continuously for at least 10 seconds.
Insert power cables and network cables back to the system.
Wait about 2 minutes before powering on server to give the iDRAC time to initialize.
Power the system on.

Apart from that, you can try to reset the server to default settings from the retire and repurpose section in the LCC. Provides NVRAM clear convenience. https://dell.to/2T8PI63
If this problem is caused by the current BIOS FW, the downgrade may be attempted. However, risky BIOS FW changes can be risky. There is no direct workaround for this error, but can you try these methods that come to my mind.

Let us know how it goes!

Ill give it a shot, nothing to lose at this point.. But I can not get into the LCC as the system halts so I am guessing there is a way to accomplish this via the iDRAC?  I'll take a look..

Hello,

Yes, you're correct - firmware updates can be done through the iDRAC web interface. If you click on iDRAC Settings, you should see an option to Update and Rollback below. You can upload the BIOS update .exe file there and once it uploads, click install and reboot to bein the update. The link below goes into detail on the subject.

https://dell.to/2QIsFhA

Ok, removed all power, drained power.. removed H710 mini, disconnected battery.. and I let it sit overnight..

Fired it up this morning and ran diags and got this..  Failing on Systems Management section

Error Code: 2000-0251

Validation Code: 90639

My guess would be there is an issue with BIOS.. How do you "force" the server to erase and re-apply BIOS.  It was upgraded to 2.9.0 from a much older version (2.1.2 I think?) and I am thinking maybe back-level it to 2.5.4, then run 2.9.0 again..

Thoughts?

Well.. Unless there is some unknown Dell magic, I am at a loss..

When I got the server up and running, everything was fine. Did the upgrade package and then I started receiving the PCIe training Error.  Since this point, I downgraded BIOS to 2.5.4, the same issue, upgraded to 2.9.0, same issue.  So it WAS once working, but now it seems It is lost.. I have tried every disabling, resetting, NVRAM clearing procedure out there and to no avail, nothing.

Am I missing something or did a BIOS upgrade really kill the mobo?

Hello, could you try updating idrac as well? 

How does one FORCE bios to write over previous BIOS of same version? or I should say, write over iDRAC & LCC which I did upg to 2.65.65.65, but I can do it again..

Good morning Steve,

There are command arguments to force a BIOS update through. In Windows, it's /forcetype. I think the easiest thing to do may be to use a FreeDOS USB to run the executable with the command argument, since the RAID error is going to give boot issues. You should be able to run a similar command in a live Linux environment, but our live support image is a MUCH larger download than a FreeDOS image. The command would probably be something different, like -f or --forcetype, as well.

What I'm suggesting would involve downloading the BIOS .exe file, then using a tool like Rufus to create a FreeDOS image. To my knowledge, DRM won't offer this.

I do have the USB image created out of DRM, but how does one add a command-line variable to that image?

If you could point the way, I have no issue editing whatever is needed to do so