PowerEdge: DDR4 Self-healing on Dell PowerEdge Servers with AMD Rome and Milan processors

What is DDR4 "self-healing" on AMD Rome and Milan processor based PowerEdge Servers (R65xx, R75xx, and C65xx)?

Do the previous generation AMD based PowerEdge servers with AMD EPYC processors (R64xx and R74xx) support these same "self-healing" capabilities?

How do these DDR4 "self-healing" capabilities (BIOS enhancements) change recommended customer and Technical Support actions when encountering memory errors on a server?

There are ongoing improvements and enhancements to the Dell Technologies PowerEdge BIOS to improve Memory error event messaging, error handling, and "self-healing" upon a server reboot, that prevents the need for a scheduled maintenance window and onsite presence to replace a DDR4 memory DIMM that was logging error events.

There are two main memory-related "self-healing" BIOS enhancements that are included with AMD processor based PowerEdge Servers (65xx and 75xx) with DDR4 memory available at product launch. These enhancements do change the recommended steps and actions to take if memory errors occur and are logged to the Lifecycle log.
 

1. Memory retraining enhancements - Memory retraining, which happens during boot, optimizes the signal timing or margining for each DIMM and slot for best access. Timing characteristics of a DIMM may change for several different reasons:

• Changes in Server memory configuration
• BIOS changes
• Different operating temperatures of the Server or DIMM
• The general age of the DIMM

Current AMD Rome and Milan based PowerEdge servers (65xx and 75xx) perform Memory retraining upon every boot. This differs from the current Intel based PowerEdge server implementation.

If any of the following errors are logged to in the SEL or Lifecycle logs, the Dell Technologies engineering's recommendation is to reboot the server to allow for memory retraining to occur.

Warning - MEM0701- "Correctable memory error rate exceeded for DIMM_XX."
Critical - MEM0702 - "Correctable memory error rate exceeded for DIMM_XX."
Critical - MEM0005 - "Persistent correctable memory error limit reached for a memory device at location(s) XX."
Critical - MEM0001 - "Multi-bit memory errors detected on memory device at location(s) DIMM_XX.

With any of these correctable or uncorrectable (multibit) memory errors, the resulting memory retraining on reboot or restart may "self-heal" the failing DIMM by optimizing the signal timing and margining for each DIMM and slot. A DIMM replacement for these errors is not necessary unless memory retraining fails (UEFI0106) during boot or these same errors continue to occur.
 

2. Post Package Repair (PPR) - The second "self-healing' memory enhancement, results in repairing a failing memory location on a DIMM by disabling the location or address at the hardware layer enabling a spare memory row to be used instead. The exact number of spare memory rows available depends on the DRAM device and DIMM size.
 

Previously, this functionality was limited to the manufacturing process. Like with the memory retraining enhancements mentioned earlier, in there are certain correctable and uncorrectable memory errors that result in PPR being scheduled on a specific DIMM slot for the next reboot (warm or cold). BIOS automatically forces a cold reboot regardless of what is initiated. Since the PPR operation is scheduled on a specific DIMM slot, DO NOT change DIMM slot locations until the PPR operation has been run. Examples of the errors are:

Warning - MEM0701- "Correctable memory error rate exceeded for DIMM_XX."
Critical - MEM0702 - "Correctable memory error rate exceeded for DIMM_XX."
Critical - MEM0005 - "Persistent correctable memory error limit reached for a memory device at location(s) XX."
Critical - MEM9072 - "The system memory has faced an uncorrectable multi-bit memory errors in the non-execution path of a memory device at the location arg1."

Any of these errors being logged in the SEL/Lifecycle log results in PPR being scheduled for the next reboot (warm or cold).

An example of a successful PPR operation is similar to:

• Message ID MEM9060 - "The PostPackage Repair operation is successfully completed on the Dual In-line Memory Module (DIMM) device that was failing earlier."

A DIMM replacement for these correctable memory errors is not necessary unless the PPR operation fails after the reboot. An example of a failing PPR message is:

• Critical - Message ID UEFI0278 - "Unable to complete the Post Package Repair (PPR) operation because of an issue in the DIMM memory slot X."

Updated April 24, 2020

Dell Technologies is continuing to enhance and expand our "self-healing" capabilities. The following section documents the updates/enhancements and what BIOS version the changes were implemented in.

BIOS 1.0.x - Initial article publication of the "self-healing" capabilities available starting with BIOS 1.0.x and higher, including example error messages and recommended actions.

BIOS 1.1.x and newer changes (December 2019)

• MEM0702 (Correctable error rate exceeded […]) - Message updated from a critical to warning event and the recommended actions updated to reboot the server to allow "self-healing"(Post Package Repair (PPR)) to occur.
  • Requires December 2019 or newer iDRAC to also be installed to get the updated message
  • Recommended Action: Reboot the server to allow PPR to run
• MEM9060 - Message description updated to indicate "self-healing" was successfully completed

BIOS 1.2.x and newer changes (February 2020)

• A "Correctable Error Logging" BIOS option was added to allow customers to disable all Lifecycle and SEL logging related to correctable errors. All the "self-healing"(PPR) features still function and memory retraining is still scheduled and run during the next reboot.
• Addition of MEM08xx errors for RDIMMs and LRDIMMs replacing existing error messages and actions. Existing error messages are still used for platforms that do not support the "self-healing" capabilities.
  • Requires February 2020 or newer iDRAC for messages to get logged 

• MEM0802 - Replaced MEM0702  - correctable error rate exceeded
  • Recommended Action: Reboot the server to allow PPR to run
• MEM0804 - Replaced MEM9060 indicating PPR was successful. Now includes DIMM slot locations that ran PPR
  • Recommended Action: None, it indicates "self-healing" occurred, no DIMM replacement is needed.
• MEM0805 - Replaced UEFI0278 indicating PPR failed
  • Recommended Action: Replacing failing DIMM

Updated January 25, 2021

BIOS 1.7.x and newer changes (December 2020)
 

• MEM8000 (Correctable error logging disabled) - Early in BIOS, Dell Technologies Engineering made a BIOS change to enhance the rate of correctable error detection that may impact performance. This change resulted in an uptick in MEM8000 events that was not substantiated by results from memory component failure analysis. Starting with BIOS1.7.x there are two changes related to MEM8000. The first is that signaling of the MEM8000 event has been modified. Second, BIOS schedules self-healing (PPR) for the next reboot. iDRAC messages are not yet updated to reflect the new actions
  • Recommended Action: Reboot the server to allow self-healing/PPR to run. Confirm that PPR was successful (MEM0804).

There are additional RAS feature enhancements being evaluated for inclusion in future BIOS updates.

A white paper is planned that describes Dell Technologies PowerEdge server (AMD Rome and Milan based processors) Memory-related Reliability, Availability, and Serviceability (RAS) features.

This article is updated as new information becomes available.