PowerFlex: High Volume Latency Encountered Exiting PMM

• Fault sets are in use, and multiple SDSs are exiting PMM at the same time.

ESXi logs displaying latency against a PowerFlex volume:

2025-03-26T19:20:58.764Z Wa(180) vmkwarning: cpu33:2098795)WARNING: ScsiDeviceIO: 1772: Device eui.##### performance has deteriorated. I/O latency increased from average value of 350 microseconds to 28802 microseconds.

SDS trace logs showing the PMM migration for an example comb - it takes roughly 27 milliseconds to complete the internal migration for comb 746b009a00b6.

2025/03/26 14:20:58.730516 MED:7f19cf3cedb0:contCmd_EndMigration:06164: CombId 746b009a00b6 - Start rc = SUCCESS  
2025/03/26 14:20:58.730757 MED:7f19cf3cedb0:raidComb_EndExitPmmInter:02999: Comb 746b009a00b6: PMM becomes Rebuild (PMM remote 41a8b5e300000023),  (Old rebuild remote 41a8b5df00000012). 
2025/03/26 14:20:58.757184 MED:7f19cf3cedb0:contCmd_EndMigration:06197: CombId 746b009a00b6 - Done rc = SUCCESS

Impact

Temporary volume latency observed by SDC clients while exiting PMM

During the exit from PMM, the combs are quiesced, preventing I/O operations. Latency is primarily introduced in the final phase of the raidComb_EndExitPmmInter function, where the system hardens bitmaps.
While this operation is typically fast, in this case, the delay is more pronounced due to the large number of combs involved, likely caused by a full fault set exiting PMM simultaneously. This increases the likelihood of encountering multiple combs in the hardening stage, amplifying the impact on latency observed by SDC clients.

Possible solutions to prevent this issue:

• Use Instant Maintenance mode (IMM).
• Only exit PMM on a single SDS at a time.

Impacted Versions

PowerFlex 3.x

PowerFlex 4.x

Fixed In Version

PowerFlex 4.5.5