Hello,

I would troubleshoot this step by step.

First, I would go back to the original memory configuration that was booting correctly before the upgrade. Then confirm that the server starts without memory errors.

On our side there is also a fatal PCIe error on the Tesla T4 in slot 1, I would also try the same test without the GPU installed, just to isolate the issue properly.

Once the server is stable, I would perform a full power drain / flea drain following Dell’s procedure:

https://www.dell.com/support/kbdoc/en-us/000175625/how-do-i-reset-and-drain-power-of-my-dell-poweredge-server

Then I would test the new DIMMs by replacing existing known-good DIMMs in the same slots, without changing the overall memory population at first. This should confirm whether the new memory modules are working correctly.

We actually had a similar case at a customer site yesterday. We followed this approach — removing the GPU, performing a proper power drain, then reinstalling and testing the new DIMMs progressively — and the server came back up without memory errors.

If the new DIMMs work fine as replacements, then I would retry the complete upgrade while strictly following the Dell / VxRail memory population rules.

The goal is to avoid combining too many variables at once: new DIMMs, new slots, GPU/PCIe error, firmware state, and previous POST/MRC memory errors.

Best regards,

Thanks for your reply

Hello Victor-Bar,

I'm sorry to see the errors on the memory.

Can you post a picture of the DIMM showing the label with Dell part number? I'd like to confirm it is validated for the R650xs.

Have you checked the DIMM slots for any pin damage?

What memory mode are you in?

Boot press and navigate to System Setup Main Menu > System BIOS > Memory Settings

Memory Operating Mode

Are you getting 768GB if you populate all sixteen 64G DIMMs?

Try running the diagnostics:

Boot to LifeCycle controller > Hardware Diagnostics > Run Hardware diagnostics

Verify you are following Memory guidelines from Owner Manual

Memory guidelines Page 58-59 :  https://dell.to/3RaVugp

an you post a picture - yes, add it, but it is not the dell memory, its crutial/micron PartNumber 36ASF8G72PZ-3G2F1

Have you checked the DIMM slots for any pin damage? - everything is OK

What memory mode are you in? - optimizer mode

Are you getting 768GB if you populate all sixteen 64G DIMMs? - yes

if I install 8 modules- 384GB

Try running the diagnostics: Boot to Life Cycle controller > Hardware Diagnostics > Run Hardware diagnostics - will show later

Verify you are following Memory guidelines from Owner Manual

- yes, do like read: RDIMM/LRDIMM configurations are 1, 2, 4, 6, 8 DIMMs per processor the same modules.

Hello Victor-Bar,

Unfortunately I can't validate the Crutial/Micron memory.

These are the 64G DIMMs we have validated for the 650xs:

    the results of Hardware diagnostics- all memory shown correctly.

Can we like engineers say, why in tests all Ok, but in bios is not?

I install 1 CPU and 8 dimms

Hi, In fact, there may be some problems when using non-Dell-certified parts. That doesn't mean those parts won't work, but they may not work with full functionality or cause other problems. If I would have done the necessary checks according to the Memory guidelines and I am getting an error. If Dell-certified DIMM was used here, if the BIOS and iDRAC are up to date, I would check with Memory cross test to see if the problem is in the slot or in the memory itself. I would check to see if different DIMMs on the same slot give the same error. If necessary, I would do the minimum and check the new processor with a DIMM that I'm sure is working. I would be the last to suspect the motherboard and processor, in case there is a problem with the memory channel.

Also, I want to share a useful article "DDR4 memory: managing Correctable error threshold events" https://dell.to/3KY5tDt

Hi Victor

Is it solved ? Please advice with the solution, 

I have the same problem in the same Server R650xs 2X4316, adding extra 6X32 GB 3200 RAM (Micron and supposed to be DELL Original) to the standard 2X32 GB  3200, total is 8X32;  I have the same errors.

Soli.

Hi @midors84,

Could you let us know if the BIOS and iDRAC/LCC have been updated. What are the slots that you have installed the memories in? How many processor configurations is in the server? May I know if the memories are RDIMM? How many ranks are they? Have you tried to install just the new memories, does the error prompts?