PowerFlex: Single Slow Device May Cause Data Unavailability (DU)

The device may or may not report its own degradation. In some cases, it generates SMART, iDRAC, and MDM events. In other cases no errors or alerts are generated, and health checks continue to report the device as healthy, while its I/O response time slows by orders of magnitude. In both cases the device is slow enough to affect client applications, but not slow enough for PowerFlex to fail it, so it remains in service. Under heavy write load, I/O can queue past the SDS internal timeout and the SDS service restarts.

Scenario 1: The device reports its own degradation

• The system is in an endless Rebuild state.
• An SSD device's health state is deteriorating:
  • MDM events show the device has not yet failed, as the MDM logic has several thresholds to meet before it fails the device at the SW level.
    MDM events - might be several events:
    

    2025-02-06 11:18:24.544000:0327876:SDS_DEV_FAILURE_STATE_CROSSED_THRESHOLD ERROR    The device failure state threshold was crossed on SDS: SDS2, Device: /dev/sda
    ...
    2025-02-06 11:39:38.780000:0328162:SDS_DEV_FAILURE_STATE_CROSSED_THRESHOLD ERROR    The device failure state threshold was crossed on SDS: SDS2, Device: /dev/sda
    ...
    2025-02-06 12:00:53.541000:0328221:SDS_DEV_FAILURE_STATE_CROSSED_THRESHOLD ERROR    The device failure state threshold was crossed on SDS: SDS2, Device: /dev/sda

  • SDS OS smartctl events
    SDS  - /var/log/messages:
    

    Feb  6 10:58:25 s-pflex02-sto602-krw1a smartd[3727]: Device: /dev/sda, SMART Failure: CONTROLLER IMPENDING FAILURE THROUGHPUT PERFORMANCE

  • SDS iDRAC predictive failures
    iDRAC - SEL logs:
    

    2025-02-06 08:51:11    575    PDR16    Predictive failure reported for Disk 0 in Backplane 1 of RAID Controller in SL 3. Part Number = PH-0HCTYM-TBPKK-42G-029G-A00
    2025-02-06 08:51:17    576    PDR1002  A predictive failure detected on drive 0 in disk drive bay 1.
    2025-02-06 08:54:58    577    PDR16    Predictive failure reported for Disk 0 in Backplane 1 of RAID Controller in SL 3. Part Number = PH-0HCTYM-TBPKK-42G-029G-A00

  • SDC OS I/O retries and I/O ERRORs
    SDC - /var/log/messages:
    

    Feb  6 10:55:58 s-dcn01-com601-krw1a kernel: [17520951.198034] PowerFlex mapVolIO_Do_CK:1481 :Mit: 000000000a48de18. Retrying IO Type WRITE. Failed comb: 245a000b829a. offsetInComb 12075736. SizeInLB 32. SDS_ID 3aee030900000003. Comb Gen  1. Head Gen d8.
    Feb  6 10:55:58 s-dcn01-com601-krw1a kernel: [17520951.198831] PowerFlex mapVolIO_Do_CK:1497 :Mit: 000000000a48de18. Vol ID 0xe3872b8100000368. Last fault Status IO_HARD_ERROR(20).Last error Status SUCCESS(65) Retry count (1) chan (2)
    Feb  6 10:55:58 s-dcn01-com601-krw1a kernel: [17520951.199659] PowerFlex mapVolIO_Do_CK:1481 :Mit: 0000000066d001ac. Retrying IO Type WRITE. Failed comb: 245a000b80c2. offsetInComb 11965032. SizeInLB 32. SDS_ID 3aee030800000005. Comb Gen  1. Head Gen d8.
    Feb  6 10:55:58 s-dcn01-com601-krw1a kernel: [17520951.200316] PowerFlex mapVolIO_Do_CK:1497 :Mit: 0000000066d001ac. Vol ID 0xe3872b8100000368. Last fault Status IO_HARD_ERROR(20).Last error Status SUCCESS(65) Retry count (1) chan (3)
    Feb  6 10:56:01 s-dcn01-com601-krw1a kernel: [17520954.206068] PowerFlex mapVolIO_Do_CK:1481 :Mit: 00000000be0da9e4. Retrying IO Type WRITE. Failed comb: 245f800f8365. offsetInComb 9690128. SizeInLB 8. SDS_ID 3aee2a1a00000004. Comb Gen 1 . Head Gen f8.

Scenario 2: The device reports nothing 

The device serves I/O continuously and returns no errors, but its response time rises from well under a millisecond to hundreds of milliseconds. Every error based check comes back clean:

• No SDS_DEV_* MDM events. The device stays State: NORMAL, DevErrorState: NONE.
• No smartd or SMART events. SMART reads healthy, typically Critical Warning 0x00, Available Spare 100%, Percentage Used 0%, Media and Data Integrity Errors 0.
• No iDRAC predictive failure, and no OS or kernel NVMe errors.
• The condition can be intermittent, and its onset does not necessarily correlate with client workload.

What is observed instead:

• SDC OS I/O retries and I/O ERRORs, the same mapVolIO signature shown in Scenario 1. These name the SDS holding the primary copy, which is usually not the node with the slow device.
• Client application latency and, on ESXi clients, datastore access loss.
• SDS service restarts, most often during a rebuild or rebalance, or when another SDS enters Protected Maintenance Mode. The SDS that restarts is not necessarily the node holding the slow device.

SDS - sds/logs/trc.0:

2026/07/17 04:33:26.050124 [CHOKE_POINT] Panic in file /data/build/workspace/ScaleIO-Common-Job/src/tgt/ioh/ioh.c, line 360, function iohIo_TimerExpired, PID 5940.Panic Expression ALWAYS_ASSERT IO takes too long - Restarting.

Impact

Depending on the state of the scenario, it may cause:

• Client-side DU
• Overall system slowness
• System DU
• SDS panic and self-restart, and brief data unavailability where a second SDS is affected while already degraded

As a first action, the problematic device needs to be removed from the system as soon as possible, using the known methods of removing the SDS Device from the PowerFlex Cluster.

All the methods below will rebuild the device's data across the remaining devices.

• Primary MDM SCLI method

scli --remove_sds_device --sds_id xxxx --device_id xxxxx

• PowerFlex Manager (PFxM) or PowerFlex Management Platform (PFMP) UI method

PFxM 3.X

1. Navigate to Resources → Storage
2. Select the target SDS node
3. Select the storage device you want to remove
4. Click Actions → Remove Device (or Remove SDS Device, depending on the exact 3.x version)
5. Confirm the removal. The device will be removed and the rebalance process will start (if the system deems necessary)

PFMP 4.X

1. Navigate to Block → SDSs
2. Select the target SDS node
3. In the right-side topology panel, click X Devices (where X is the number of devices attached to the SDS)
4. Select the storage device you want to remove
5. Click More Actions → Remove
6. Confirm the removal. The device will be removed and the rebalance process will start (if the system deems necessary)

• Manual OS-level method

To turn the device offline at the OS level, follow the OS's vendor' documented procedure for the drive type (SATA/SAS or NVMe).

If the OS-level offline method is not viable and drive removal is blocked, for example, because another SDS is already in maintenance mode, open an SR ticket for PowerFlex Support to use an internal debug action to mark the device as being in an error state and force trigger a rebuild.

If the device cannot be removed using any of the methods above, open an SR ticket for Dell EMC Support, who will raise an engineering escalation to have the device removed safely.

In order to avoid such incidents in the future, it is recommended to upgrade to one of the fixed version, mentioned in the "Fixed In Version" section below.

If an upgrade to the a PowerFlex version that contains the improvements is not feasible, open an SR ticket for PowerFlex Support to use an internal procedure to improve the MDM response time to slow SSD devices, by adjusting the thresholds that would cause the MDM to move the device into an error state sooner.

Impacted Versions

PowerFlex 3.x

PowerFlex 4.x

Fixed In Versions

PowerFlex 3.6.6

PowerFlex 4.5.5