Precision Rack 7910 won't accept 512GB RAM

I'm trying to do a memory upgrade on this system.  It has two 2680v3 CPUs, All DIMMs are 32GB DDR4-2400R dual rank (yes they only work at 2133 with these CPUs).  When installing all 16 DIMMs (8 per CPU, 2 per memory channel) it always stops right after Configuring Memory and I get an error in the iDRAC log that BIOS has halted. 

I tried swapping DIMMs around in different orders and kept getting the same result.  Then I started the process of removing a single DIMM and trying to boot.  Then all of a sudden it's working fine with 480GB of RAM.  So I take a DIMM from a different slot and put it in the original one I removed and it still works.  Place all 512GB and it doesn't work again.  Remove a totally different DIMM than the first two and it works with 480GB again.  

This was all on CPU2.  I then put 256GB on CPU2 and remove a DIMM from CPU1 and it still boots with 480GB!  I try putting the "extra" DIMM on CPU1 and leave one channel with only one DIMM on CPU2 (back to 512GB) and it doesn't boot again.

So it doesn't seem to matter at all which DIMM I remove, 480GB memory boots and 512GB fails after memory training.  

Since all I can see is a splashscreen, don't get any useful console messages, and don't have a way to enable BIOS debug output I have no clue if this is an issue with the MRC, handoff to the BIOS or something else.  I have a few other 32GB DIMMs available that I will try to debug with soon, but as far as I can tell from the manual and DIMM config guide this should be working.