PowerScale: Scaling Partition Performance to Track 5K Top workloads

Partitioned Performance (PP) in PowerScale OneFS provides per-dataset Top workload tracking, allowing administrators to identify the most active workloads (by CPU usage, bandwidth, or protocol/disk operation counts) on the cluster.

Customers looking to track up to 5,000 active Top workloads per dataset can do so by applying the configuration changes described in this article.

Important considerations:

• This is an opt-in configuration. Default cluster behavior is unchanged.
• The configuration applies cluster-wide and affects all configured datasets.
• Workload queries at higher scales take longer to respond. Using the --numeric flag with CLI queries is strongly recommended to avoid timeouts.

Prerequisites

• OneFS 9.15 or later
• CLI access with root privileges
• This configuration has been validated on all-flash clusters with up to 40 nodes.
• Understanding of your cluster's dataset configuration and active initiator count

Step 1: Apply PAPI Settings

Run the following command from any node in the cluster. The ?force=true parameter is required. Replace <VALUE> with the appropriate max_stat_size from the sizing guidance below.

curl -sk -u root -H 'Content-Type: application/json' -X PUT \ 'https://<node_IP>:8080/platform/25/performance/settings?force=true' \ -d '{"top_n_collection_count": 5000, "max_top_n_collection_count": 5000, "max_stat_size": <VALUE>}'

Set the following parameters:

• max_top_n_collection_count - Upper bound for Top workload collection. Default is 1,024. Recommended: 5,000.
• top_n_collection_count - Active Top workload collection count. Default is 1,024. Recommended: 5,000.
• max_stat_size - Maximum stat sample size in bytes. Default is 524,288 (512 KiB). The recommended value varies by cluster configuration - see the sizing guidance below.

Determining the correct max_stat_size

The max_stat_size parameter controls the maximum size of a single stat sample. This value must be set based on your cluster's number of active initiator nodes and configured datasets. Setting it too high can cause the statistics service to exceed memory limits during workload queries.

The recommended value is derived using the following formula:

max_stat_size = min(2700000, 68000000 / (active_initiator_nodes × configured_datasets))

Some examples are provided in the table below, to select the appropriate max_stat_size for your deployment:

• For clusters with 1 to 6 initiator nodes and 1 to 4 datasets, use 2,700,000.
• For clusters with 7 to 25 initiator nodes and 1 dataset, use 2,700,000.
• For clusters with 7 to 12 initiator nodes and 2 datasets, use 2,700,000.
• For clusters with 7 to 8 initiator nodes and 3 datasets, use 2,700,000.
• For 7 initiator nodes with 4 datasets, use 2,428,571.
• For 20 initiator nodes with 2 datasets, use 1,700,000.
• For 20 initiator nodes with 3 datasets, use 1,133,333.
• For 20 initiator nodes with 4 datasets, use 850,000.
• For 40 initiator nodes with 1 dataset, use 1,700,000.
• For 40 initiator nodes with 2 datasets, use 850,000.
• For 40 initiator nodes with 3 datasets, use 566,666.
• For 40 initiator nodes with 4 datasets, use 425,000.

Note: For large clusters or datasets with multiple metrics it may not support the full 5,000 workloads.

Step 2: Apply Kernel Sysctls

Run the following commands to apply the settings across all nodes in the cluster:

isi_sysctl_cluster "efs.acct.options.max_clients=13000"
isi_sysctl_cluster "efs.acct.options.max_db_actors=16384"
isi_sysctl_cluster "efs.acct.options.max_actor_records=10000"

• efs.acct.options.max_clients - Sets the maximum tracked clients per CPU in the hot-path pipeline stage. Default is 1,000. Recommended: 13,000.
• efs.acct.options.max_db_actors - Sets the maximum database actors per node in the collation pipeline stage. Default is 8,192. Recommended: 16,384.
• efs.acct.options.max_actor_records - Sets the maximum actor records per node in the frontend pipeline stage. Default is 1,024. Recommended: 10,000.

Step 3: Verify Configuration

After applying the settings, verify the PAPI configuration:

curl -sk -u root -X GET 'https://<node_IP>:8080/platform/25/performance/settings'

Confirm that top_n_collection_count and max_top_n_collection_count both show 5000 and max_stat_size matches the value selected from the sizing guidance above.

Verify sysctl values:

isi_sysctl_cluster "efs.acct.options.max_clients"
isi_sysctl_cluster "efs.acct.options.max_db_actors"
isi_sysctl_cluster "efs.acct.options.max_actor_records"

Step 4: Query Workloads

When querying workloads at the 5K scale, use the --numeric flag to avoid timeouts caused by path resolution and lookup overhead:

isi statistics workloads --numeric

Important Notes:

Setting max_stat_size too high

If max_stat_size is set higher than recommended for your cluster configuration, it can cause:

• The active workload query to fail.
• Temporary loss of collected statistics.
• Impact on services that depend on statistics, including SmartQoS.

Always use the value from the sizing table that matches your cluster's initiator node count and dataset count.

Cluster or dataset configuration changes

If you add nodes to the cluster or create additional datasets after enabling 5,000 workloads, the memory required for workload queries increases. After making such changes:

• Re-check the sizing table to confirm your max_stat_size is still appropriate for the new configuration.
• If the new configuration requires a lower max_stat_size, update the setting using the PAPI command in Step 1.