PowerStore: Overlapping I/O on a Metro volume may lead to unexpected reboot or increased latency
Summary: Overlapping I/O on a Metro volume can lead to an unexpected node reboot or higher latency.
Symptoms
A node may reboot unexpectedly, or increased latency may be observed if all these conditions are present:
- PowerStoreOS 3.6.1.0
- Metro volumes
- ESXi hosts configured with access set as co-located with both systems
- The issue can also manifest during NDU from 3.6.1, if the workaround described below is implemented.
Cause
Due to the software issue, any overlapping I/O - I/O arriving to both appliances on the same volume on the same Logical Block Address (LBA) - may lead to an unexpected module restart or node reboot. The unexpected software module restart or node reboot may repeat until the fix or workaround is applied.
Resolution
Pre-Upgrade Health Check (PUHC)
A new PUHC check has been released to identify appliance clusters running PowerStoreOS 3.6.1.0 with a Metro configuration. The PUHC test will fail and recommend pausing the Metro sessions:
Fix
This issue is fixed in PowerStoreOS 3.6.1.1 (3.6 Service Pack 1 Patch 1) and above.
Note:
When performing an NDU, pause all Metro sessions before upgrading from 3.6.1 (only) to any future release and resume replication when the NDU is complete.
- Pause all Metro sessions before upgrading from 3.6.1 to any future release.
Alternatively:
- Avoid configuring a metro volume as "Metro_Optimized_Local" on a Non-Preferred appliance until NDU completes.
- If the local appliance is the non-preferred for all metro sessions on this appliance (Protection -> Metro -> Local Preferred Role"), make sure any host attached to a metro volume is configured as Metro_Optimized_Remote until NDU completes.
- If the local appliance has a mix of metro volumes with some preferred and some non-preferred, it is best to review each host configured as Metro_Optimized_Local and make sure all its associated metro volumes are configured as preferred on the local appliance or modify the host access to Metro_Optimized_Remote.
Workaround
Since this issue only affects PowerStoreOS 3.6.1.0 with hosts configured as Co-located with both systems, the following workaround is suggested:
- This procedure can be performed non-disruptively.
- On the preferred PowerStore, configure host-access to: Host is co-located with this system.
- On the non-preferred PowerStore, configure host-access to: Host is co-located with remote system.
- Example configuring on the preferred appliance.
Contact Dell Technical Support for additional guidance if needed.
Additional Information
Additional Information
This issue can also be seen during NDU with ESXi hosts configured as Metro_Optimize_Local on the Non-Preferred.
If the customer is performing an NDU from 3.6.1.0, it is recommended to pause metro volume, complete the NDU, then re-enable metro
Host connectivity can be viewed in the PowerStore Manager user interface under Compute > Host Information > HOSTS & HOST GROUPS.
For easier verification, add the Host Connectivity column to the default view:
