PowerFlex 3.X: vTree Migration May Cause SDS Panic or Hang

In PowerFlex version 3.6.7, during a scheduled vTree migration, multiple SDSs panic, causing the system to enter a DU state.

As a result of the migration delays, Storage Pool (SP) capacity consumption increases rapidly and can reach full utilization within a short period of time.

MDM events:

2026-06-30 07:27:19.340 SDS_DECOUPLED ERROR SDS: SDS11 (id: 64cb920800000006) decoupled.
2026-06-30 07:27:19.465 MDM_DATA_DEGRADED ERROR The system is now in DEGRADED state
....
2026-06-30 07:32:20.638 SDS_DECOUPLED ERROR SDS: SDS9 (id: 64cb920b00000009) decoupled.
2026-06-30 07:32:20.638 SDS_DECOUPLED ERROR SDS: SDS12 (id: 64cb920a00000008) decoupled.
2026-06-30 07:32:21.842 MDM_DATA_FAILED CRITICAL The system is now in DATA FAILURE state. Some data is unavailable
...
2026-06-30 07:32:23.538 DEV_CAPACITY_USAGE_CRITICAL ERROR Capacity usage on Protection Domain PD1, Storage Pool FGSP_01 is CRITICAL.
2026-06-30 07:32:23.605 DEV_CAPACITY_USAGE_FULL ERROR Capacity usage on Protection Domain PD1, Storage Pool FGSP_02 is FULL.

Trace logs on all the affected SDS nodes produce the same panic stack trace:

2026/07/01 17:45:33.506630 Panic in file /data/build/workspace/ScaleIO-Common-Job/src/mos/mos_timer.c, line 437, function mosTimerReq_AddByQ, PID 1291427.Panic Expression pReq->state == MOS_TIMER_REQ_STATE__IDLE .
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(mosDbg_PanicPrepare+0xe5) [0x932ce5]
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(mosTimerReq_AddByQ+0x143) [0x917043]
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(vaeCleaner_CleanVae+0x64) [0x668904]
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(head_removeVae+0x170) [0x545e00]
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(head_Update+0x373) [0x546363]
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(contHead_UpdateFull+0x69) [0x4db689]
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(contCmd_AddHead+0x18a) [0x60d85a]
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(contCmd_AddHeadGroup+0x100) [0x61b630]
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(contCmd_NewRequest+0x15fd) [0x62b96d]
/opt/emc/scaleio/sds/bin/sds-3.6.7000.123(contNet_RecvRequest+0xd0) [0x4ce880]

In PowerFlex versions earlier than 3.6.7, during a scheduled vTree migration, the migration progress hangs and never completes.

In PFxM UI > Running Storage Jobs, it shows Loading endlessly:

query_vtree_migration output shows the Progress percentage never changing:

scli --query_vtree_migration --volume_id 53b80e00000000d1
VTree ID: ba46bf57000000d1 Name: PDC-GC-LNX-N-01-PFSP01-01-V043
    Storage Pool 5f2f4acc00000000 Name: PDC-FLEX-LNX-SP01
    Protection Domain 908ff43700000000 Name: PDC-FLEX-LNX-PD01
    Data layout: Medium granularity
    Provisioning: Thin
    Total capacity in use: 504.7 GB (516787 MB)
    Total user data: 504.7 GB (516787 MB)
            Total base user data: 504.7 GB (516787 MB)
            Total snapshots user data: 0 Bytes
    VTree migration info:
            Migration status: Migrating
            Source:      Storage Pool: PDC-FLEX-LNX-SP01 ID: 5f2f4acc00000000 Protection Domain: PDC-FLEX-LNX-PD01 ID: 908ff43700000000
            Destination: Storage Pool: PDC-FLEX-LNX-SP03 ID: 5f2fe70d00000004 Protection Domain: PDC-FLEX-LNX-PD03 ID: 9090907800000003
            Conversion type: No conversion
            Queue position: 1
            Progress percentage : 34%

Impact

On versions earlier than 3.6.7, volume migration never ends, SP capacity utilization is critical or higher, and applications may experience I/O errors.

On version 3.6.7, DU.

This issue is caused by a multi-threaded race condition involving a shared timer mechanism within the Volume Allocation Element (VAE) cleaner component during volume migration operations.

In PowerFlex versions prior to 3.6.7, an asynchronous cleanup process was used to delete data from old SP locations after a migration. Under rare conditions, such as concurrent volume management operations, simultaneous volume deletions, high I/O load, or network instability, this process could experience a race condition that caused migrations to appear indefinitely stuck.

To address this, an enhanced monitoring mechanism was introduced in version 3.6.7. This safety mechanism featured a timer designed to detect a stuck VAE cleaner thread and proactively panic the SDS service to prevent data corruption or prolonged stalls, similar to how the system handles stuck I/O or MDM commands.

However, in the initial implementation of this fix, the monitoring timer was initialized only once during migration data construction and was shared globally across all subsequent VAE cleaner calls. When multiple background threads attempt to access this single shared timer simultaneously during heavy migration workloads, a new race condition is triggered. This concurrent access causes the tracking mechanism to incorrectly flag the VAE cleaner as stuck, resulting in unintended SDS service crashes and localized disruption to data availability, resulting in DU.

Permanent fix (recommended)

Upgrade the cluster to PowerFlex 3.6.7.1 or later. The fix for both the VAE‑cleaner and timer race conditions are included.

Workaround (if upgrade cannot be performed immediately)

On versions earlier than 3.6.7

1. On the Primary MDM, navigate to the trace logs directory locates at /opt/emc/scaleio/mdm/logs/

To find the current trace log , run ls -ltr , so the latest trc.z.* log will be listed at the bottom of the list.

2. Since logs are compressed, use the trace_decompress utility located at /opt/emc/scaleio/mdm/bin/

tail -f <current_trace_log> | /opt/emc/scaleio/mdm/bin/trace_decompress

3. Filter for SDS ID (i.e., TGT) entries. Use the grep command to find relevant entries related to the SDS the operation is run in:

tail -f <current_trace_log> | /opt/emc/scaleio/mdm/bin/trace_decompress | grep volumeBlock_HandleQueryCleanVaeResponse

4. Identify the TGT. Look for lines similar to the following in the filtered output:

TGT 41a903f100000036 not done yet

If the TGT ID remains consistent over time, it likely corresponds to the SDS node involved. Rerun step 3 to confirm that is stays consistent.

5. Run the following command to list all SDS nodes and match the TGT ID with the SDS name/IP address:

scli --query_all_sds

6. Once you have the needed information, you can then SSH to that SDS and restart the SDS service by running:

pkill sds

On version 3.6.7

• Serialize volume management operations – run only one delete, resize, or migration at a time.
• Reduce I/O load on the affected Protection Domain(s) while migrations run.
• Avoid bulk vTree migrations; batch them in small groups.
• Do not pause or cancel a migration when the network shows instability.

These actions lower the probability of concurrent timer use and thus reduce panic occurrences.

Impacted Versions

PowerFlex Core 3.x.x.x

Fixed In Version

PowerFlex Core 3.6.7.1