PowerFlex: Client IO Errors When Replication Is Being used

Summary: Client/servers are experiencing IO errors against PowerFlex-backed devices. The overall backend (MDM/SDSes) appears to be healthy. PowerFlex replication is being used and there are some RPO errors against 1 or more of the RCGs. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

  • No degraded or failed capacity
  • No SDSs decoupled or SDS devices reporting errors.
  • No disconnected MDMs

The replication feature is being used.
One or more alerts in the UI reporting the following errors:

Major - The RCG consistent image is too large to be consumed by the destination in one piece.

Minor - Remote Consistency Group RPO Exceeded
MDM event logs may report the following: 
2024-06-11 15:55:56.592000:0001566:RPL_PD_CAP_UTILIZATION_MINOR     WARNING  Protection Domain ID <pd_id> Replication journal capacity is at MINOR utilization level
2024-06-11 16:20:12.848000:0001567:RPL_PD_CAP_UTILIZATION_MAJOR     ERROR    Protection Domain ID <pd_id> Replication journal capacity is at MAJOR utilization level
2024-06-11 17:19:57.272000:0001584:RPL_PD_CAP_UTILIZATION_CRITICAL  CRITICAL Protection Domain ID <pd_id> Replication journal capacity is at VERY_HIGH utilization level
2024-06-11 17:52:26.352000:0001585:RPL_PD_CAP_UTILIZATION_CRITICAL  CRITICAL Protection Domain ID <pd_id> Replication journal capacity is at CRITICAL utilization level
...
2024-06-11 16:25:14.381000:0001576:RPL_CG_MOVED_TO_SLIM_MODE        INFO     Replication Consistency Group ID <rcg_id> entered slim mode
2024-06-11 18:27:29.738000:0001586:SDR_CRITICAL_CAP_CHANGE          ERROR    SDR ID <sdr_id>) handling user data changed discarded old user data and stopped to accumulate new user data due critical capacity
 

Impact 

Clients are unable to access volumes that are intercepted for replication.


Cause

A rare software defect may occur where the MDM and the SDR component disagree on the internal counters related to journal capacity. This discrepancy may cause the MDM to unallocated additional journal capacity when the SDR's capacity is full, potentially leading to IO errors on client/servers using devices backed by PowerFlex.

Resolution

A rolling restart of all SDR components is required, and the MDM ownership on the source system must be switched to resolve the issue.

Restart SDR components:

1) Identify all the SDRs:

scli --query_all_sdr

2) Enter maintenance mode on the SDR:

scli --enter_sdr_maintenance_mode --sdr_name <name>

3) Validate that the SDR is fully in maintenance mode by running the command in step 1.

4) Restart the SDR component.

pkill sdr

5) Repeat this for all SDRs on the source site.
 

Once all SDRs are restarted, switch the MDM ownership:

#3.x
scli --switch_mdm_ownership --new_master_mdm_name <name>
 
#4.x
scli --switch_mdm_ownership --new_primary_mdm_name <name>

*If wanted, the ownership can be transferred back to the original MDM server.

Validate that the IO errors on the client/servers are no longer are reported. If the client has entered a read-only file system, the client/server may require a reboot.

Until the cause can be determined, it will also be recommended that all RCGs on the source system be terminated.

If the IO errors continue after applying the above steps, engage PowerFlex Engineering.

Additional Information

Impacted Versions

PowerFlex 3.x

PowerFlex 4.x

Fixed In Version

PFMP 4.5.3

Affected Products

PowerFlex rack, PowerFlex Appliance, PowerFlex Software
Article Properties
Article Number: 000227849
Article Type: Solution
Last Modified: 17 Nov 2025
Version:  6
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.