Dell EMC VxRail: VXR500MEM8000/MEM8000 (Correctable error logging disabled)

This article explains the actions that are required when VXR500MEM8000 occurs for VxRail code 4.5.400-4.5.420 and 4.7.210-4.7.420 which equates to BIOS 2.1-2.5.

VXR500MEM8000/MEM8000 (Correctable error logging disabled)

Starting with BIOS ~2.0.x, Dell Engineering made a BIOS change to enhance the rate of correctable error detection that may impact performance. This change resulted in an uptick in MEM8000 events that was not substantiated by results from memory component failure analysis. Starting with BIOS 2.7.x, there are two changes that are related to MEM8000. The first is signaling of the MEM8000 event has been modified. The second change is BIOS schedules self-healing (PPR) for the next reboot. iDRAC messages are not yet updated to reflect the new actions.

VxRail 4.5.462+ and 4.7.530+ include BIOS 2.10.2 which contain multiple memory-related fixes and customers should upgrade to resolve this issue permanently. In the short-term, to check if the DIMM is serviceable and resolve any correctable issues should they exist we need the host to be rebooted. At the time, of the host reboot if correctable issues exist the event VXR500UEFI0278/UEFI0278 are posted in VC or iDRAC. 

The reboot has two impacts.

• Logging which has been disabled for DIMM is restarted.
• Memory self-healing is also run if required. Correctable errors previously were the trigger for MEM8000. In the fixed BIOS, we have returned to that behavior which resolves the false positives.

Below are the steps to place a Member of vSAN Cluster in Maintenance Mode ready for reboot.
Before you shut down, reboot, or disconnect a host that is a member of a vSAN cluster, you must place the host in maintenance mode. When you place a host in maintenance mode, you must select a data evacuation mode, such as Ensure accessibility or Full data migration.

When any member host of a vSAN cluster enters maintenance mode, the cluster capacity is automatically reduced, because the member host no longer contributes capacity to the cluster.

Any vSAN iSCSI targets that are served by this host are transferred to other hosts in the cluster, and the iSCSI initiator are redirected to the new target owner.

Procedure:
You place a host in maintenance mode when you must service it such as to install more memory. A host enters or leaves maintenance mode only as the result of a user request.

Virtual machines that are running on a host entering maintenance mode must be migrated to another host (either manually or automatically by DRS) or shut down. The host is in a state of Entering Maintenance Mode until all running virtual machines are powered down or migrated to different hosts. You cannot turn on virtual machines or migrate virtual machines to a host entering maintenance mode.

When no more running virtual machines are on the host, the host s icon changes to include under maintenance and the host s Summary panel indicates the new state. While in maintenance mode, the host does not allow you to deploy or turn on a virtual machine.

Note: DRS does not recommend (or perform, in fully automated mode) any virtual machine migrations off a host entering maintenance or standby mode if the vSphere HA failover level would be violated after the host enters the requested mode.

Procedure

1. Browse to the host in the vSphere Client.
2. Right-click the host and select Maintenance Mode > Enter Maintenance Mode.
   • If the host is part of a partially automated or manual DRS cluster, browse to Cluster > Monitor > DRS > Recommendations and click Apply Recommendations.
   • If the host is part of an automated DRS cluster, virtual machines are migrated to different hosts when the host enters maintenance mode.
3. If applicable, click Yes.

The host is in maintenance mode until you select Maintenance Mode > Exit Maintenance Mode.

Once the host has been rebooted, memory self-healing has either run or was not required. We must look for VXR500UEFI0278/UEFI0278 in either VC or IDRAC respectively. If this message appears, it means that memory self-healing was required and it failed.

This requires a DIMM replacement, and a case should be opened with support for replacement. All other instances are considered a false positive.