PowerFlex: Rebuild and Rebalance impacting frontend IO workload performance
Summary: Rebuild and Rebalance impacting frontend IO workload.
Symptoms
When a rebuild or rebalance begins, the frontend application IOPS can drop drastically.
- An SDS node or disk has failed, causing a rebuild to begin.
- An SDS node or disk has been added to the cluster or storage pool, causing a rebalance to begin.
- An SDS node or disks are removed from the cluster or storage pool, causing a rebalance to begin.
- Frontend IOPS drop-down drastically (up to 50% or more of the expected IOPS) after the rebuild/rebalance starts.
Impact
If increases to the speed and resources of a rebuild or rebalance are performed, it reduces the time needed to create the two full copies of each block that ScaleIO needs. However, SDCs' throughput and IOPS can be affected and end user may notice application slowness or possibly even IO errors. The effect these changes can have on frontend IO should be understood before making the change.
Cause
Rebuild and rebalance use the same disk and possibly network resources that the SDCs are using, thus taking bandwidth in both disk and network. Rebuild and rebalance also use two SDS threads by default, defined as the parallelism parameter.
The SDS created these threads and takes extra CPU and Memory if bumped above the default of two threads. This causes the SDS to spend more CPU time and resources on the rebuild/rebalance than it would, causing less CPU time and resources to be spent on serving frontend IOs.
Resolution
There are a few things to check here. The rebuild and rebalance can be affected by the following parameters:
1- SDS Network Throttling This limit how I/O each SDS uses for the various categories listed. The default in all categories is "Unlimited."
A-From scli:
scli --set_sds_network_limits (--protection_domain_id | --protection_domain_name ) [--rebuild_limit ] [--rebalance_limit ][--overall_limit ] [--i_am_sure] Example, setting the rebuild limit to 100 MB/s: scli --set_sds_network_limits --protection_domain_name pd1 --rebuild_limit 100
B-From UI:
Go to the protection domain list, select the protection domain in question and, choose "Network Throttling" from the Modify list.
2- Storage Pool I/O Priority This setting is per storage pool and will change the IO priority for rebuild or rebalance and does affect system performance. The default is "Limit Concurrent IO" set to 1 for both rebuild and rebalance.
A-From scli:
scli --set_rebuild_policy ((( protection_domain_id | --protection_domain_name ) --storage_pool_name ) | --storage_pool_id ) --policy [Options][--i_am_sure] scli --set_rebalance_policy ((( protection_domain_id | --protection_domain_name ) --storage_pool_name ) | --storage_pool_id ) --policy [Options][--i_am_sure]
The POLICY can be one of: no_limit, limit_concurrent_io, favor_application_io, dynamic_bandwidth_throttling
If using anything but "no_limit" for the policy, the use of command OPTIONS becomes required. See PowerFlex documentation for more references on these settings.
B-From UI: Go to the Storage Pool list, select the Storage Pool in question and, choose "I/O Priority" from the modify list.
3- Concurrent Rebuild and Rebalance Activities. The default setting here is 2.
A- This parameter can only be changed from scli:
scli --set_rebuild_rebalance_parallelism (((--protection_domain_id | --protection_domain_name ) --storage_pool_name ) | --storage_pool_id ) (--limit ) Example, setting the limit to 5: scli --set_rebuild_rebalance_parallelism --protection_domain_name pd1 --storage_pool_name HDD_POOL --limit 5
NOTE: If you use the storage_pool_id parameter in the command, you do not need to also specify the protection domain ID or name, and the storage pool name. Unless specified by ScaleIO support, leave this parameter at 2. Increasing this setting will adversely affect the frontend IO quicker than the network throttling and I/O priority settings.