PowerFlex: How to Detect if Persistent Memory (NVDIMM) Has Bad Blocks
Summary: PowerFlex’s Fine Granularity Storage Pools use Persistent Memory (PMEM) also known as Non-Volatile DIMM (NVDIMM) to accelerate and compress data. These DIMMs may have bad blocks just like regular RDIMMs and must be monitored periodically for faults like any other type of disk. In an earlier release, the SDS device will only detect the bad block upon accessing the relevant section and failing the read by failing the SDS. For the earlier release, the following steps can be followed proactively to detect NVDIMMs with bad blocks and replace them. Note: since we have inherent redundancy in the product the data of these blocks are backed up on another node and in case of failure the SDS will fail and the other SDSs will service the IO. ...
Instructions
This procedure is only applicable to Linux-based OSs.
Note: VMware based implementation on ESXi maintains the capability to check if the device has faults or not and cannot be queried from the PowerFlex Storage VM, in case of NVDIMM failure the NVRAM datastore will be offline and the Storage VM will power off and not be able to boot until the NVDIMM is replaced.
How to Detect the Bad blocks on Linux OS:
1. Run the following command using the shell on each Host that has an NVDIMM:
NOTE: The frequency recommendation is once a day.
ndctl list --media-errors
Example of the command output for a DAX device that has failures.
In the below part of the output, namespace 1.0 with dax 1.0 has 13 bad blocks and must be replaced.
[
{
"dev":"namespace1.0",
"mode":"devdax",
"map":"dev",
"size":62914560,
"uuid":"47072fee-1447-4e04-b4ca-f28e2c03af80",
"chardev":"dax1.0",
"align":4096,
"badblock_count":13,
"badblocks":
2. Correlating the DAX device that is in use in a PowerFlex Storage Pool:
a. In order to retrieve a list of all storage devices configured to each Storage Pool in the cluster, first, we must log in:
scli --login --username <NAME> --password <PASSWORD>
b. View the list of devices configured to each Storage Pool:
for SNAME in $(scli --query_all_sds | grep ID: | awk '{print $5}'); do echo $SNAME ; scli --query_sds --sds_name $SNAME | grep -E "Path|Pool" ; done
Output similar to the following should appear for each SDS in the cluster:
SDS_Node1 Storage Pool information (total of 2 Storage Pools): 1: Storage Pool: default inflight requests factor: 115, inflight bandwidth factor 115 2: Storage Pool: fg_1 inflight requests factor: 115, inflight bandwidth factor 115 1: Name: N/A Path: /dev/sdv Original-path: /dev/sdv ID: ddfeb59d00030000 Storage Pool: default, Capacity: 1786 GB, State: Normal 2: Name: N/A Path: /dev/sdk Original-path: /dev/sdk ID: ddfedc3100030005 Storage Pool: fg_1, Capacity: 894 GB, State: Normal 3: Name: N/A Path: /dev/sdl Original-path: /dev/sdl ID: ddfedc3200030006 Storage Pool: fg_1, Capacity: 894 GB, State: Normal 4: Name: N/A Path: /dev/sdm Original-path: /dev/sdm ID: ddfedc3300030007 Storage Pool: fg_1, Capacity: 894 GB, State: Normal 5: Name: N/A Path: /dev/sdn Original-path: /dev/sdn ID: ddfedc3400030008 Storage Pool: fg_1, Capacity: 894 GB, State: Normal 6: Name: N/A Path: /dev/svm_sdo Original-path: /dev/svm_sdo ID: ddfedc3500030009 Storage Pool: fg_1, Capacity: 894 GB, State: Normal 7: Name: N/A Path: /dev/svm_sdp Original-path: /dev/svm_sdp ID: ddfedc360003000a Storage Pool: fg_1, Capacity: 894 GB, State: Normal 1: Name: N/A Path: /dev/dax0.0 Original-path: /dev/dax0.0 ID: ddfedc2100030001 Acceleration Pool: ac_1, Capacity: 15.7 GB (16052 MB), Used: 1.3 GB (1372 MB), State: Normal 2: Name: N/A Path: /dev/dax1.0 Original-path: /dev/dax1.0 ID: ddfedc2200030002 Acceleration Pool: ac_1, Capacity: 15.7 GB (16052 MB), Used: 1.3 GB (1372 MB), State: Normal 3: Name: N/A Path: /dev/dax2.0 Original-path: /dev/dax2.0 ID: ddfedc2300030003 Acceleration Pool: ac_1, Capacity: 15.7 GB (16052 MB), Used: 686.0 MB (702464 KB), State: Normal 4: Name: N/A Path: /dev/dax3.0 Original-path: /dev/dax3.0 ID: ddfedc2400030004 Acceleration Pool: ac_1, Capacity: 15.7 GB (16052 MB), Used: 686.0 MB (702464 KB), State: Normal
This means that this node has a DAX device /dev/dax1.0 used by acceleration pool ac_1 that has bad blocks and must be replaced.
PowerFlex Support Guidance
- If such a case is found only on one host proceed with the removal of the DAX device from the PowerFlex system and open a service call to FRU (Field replace unit) the faulty PMEM\NVDIMM device.
- Dell Technologies recommends replacing the PMEM\ NVDIMM device as soon as possible in a scheduled maintenance window.
- The process to replace a faulty NVDIMM is documented in the R640\R740xd\R840 Field Replaceable Unit Guide, relevant chapter is: "Replacing an NVDIMM in a Linux-based VxFlex Ready Node R640\R740xd\R840 server."
- If you have more than one device on different nodes in the same Fine Granularity Storage Pool with bad blocks, then open a service call to PowerFlex support to verify data stability.