Resource-intensive workloads are the lifeblood of the digital economy. Traditional applications running in virtual machines (VMs) and modern cloud native applications running in containerized ecosystems drive underlying virtualization clusters to their limits. Many IT professionals support these business-critical services with antiquated systems that cannot deliver the end-user response times that are required to meet service-level agreements (SLAs). Even if the infrastructure is meeting expectations, inflexible configurations often prevent easily scaling-up or scaling-out to meet projected performance and capacity demands in the future.

Incorrect and suboptimal infrastructure designs and configurations also lead to performance degradation over the life of IT systems. Failure to test and validate components properly with the correct combination of BIOS, firmware, and operating system driver revisions can significantly impact IOPS, throughput, latency, CPU utilization, and more. IT administrators must also perform maintenance by routinely applying hardware and software updates to keep their systems secure and running at peak performance and availability. However, this task is often neglected due to risky and time-consuming manual update processes.

Once precisely tuned and optimized, highly performant systems must be monitored carefully to ensure that they are meeting SLAs. Some IT operational tooling is complex and lacks the features and functionality that is required to address modernized cloud native applications, such as those running in Kubernetes clusters. These tools may not be updated frequently enough to keep up with the constant changes in enterprise IT environments.

Dell Integrated System for Azure Stack HCI (Azure Stack HCI) is an ideal solution for organizations that are refreshing and modernizing their aging virtualization environments to support high-value, highly performant workloads. This all-in-one validated hyperconverged infrastructure (HCI) system includes full-stack life cycle management, native integration into Microsoft Azure, flexible consumption models, and solution-level enterprise support and services expertise.

Azure Stack HCI offers a broad portfolio of intelligently designed AX nodes that are workload-focused and validated with deliberately selected hardware components and BIOS, firmware, and driver revisions. High-performance architectures leverage technologies such as 25/100 GbE RDMA networking and all-NVMe storage configurations. Our engineering organization also validates scalable and switchless storage networking topologies using Dell PowerSwitch network switches. We also provide prescriptive guidance on expanding clusters to accommodate future workload demand.

Dell OpenManage Integration with Microsoft Windows Admin Center enables automatic cluster creation with the Dell HCI Configuration Profile and one-click full-stack life cycle management using Cluster-Aware Updating. This ensures that Azure Stack HCI is deployed in a guided and consistent way and helps maintain the latest quality, security, and performance enhancements. Microsoft and Dell Technologies provide extensions and snap-ins that reduce risk, eliminate guesswork, and liberate IT administrators to focus on performance tuning and other high-value activities.

Azure Stack HCI's native integration with Microsoft Azure allows IT administrators to leverage feature-rich management and governance services such as Azure Monitor, Azure Security Center, and Azure Policy. These services are consistent with and complementary to Microsoft Windows Admin Center, which serves as the edge or local, always-available management console. Microsoft continually adds new innovative functionality to Microsoft Azure services to ensure that even the most complex and demanding VM-based and containerized applications are properly monitored and performing at the levels that are required to meet SLAs.

In this white paper, we demonstrate how the architectural components of the Dell Integrated System work together to deliver extraordinary value to high-performing workloads. We performed synthetic testing on an Azure Stack HCI cluster running four AX-7525 nodes, each populated with two AMD EPYC 7742 64-core processors, 24 NVMe drives (PCIe Gen 4), and 100 GbE RDMA networking. This configuration delivered the following results:

• 5,727,985 IOPS at 1.3-millisecond latency using a 4 KB block size and 100% random-read I/O profile.
• 700,256 IOPS at 9-millisecond latency using a 4 KB block size and 100% random-write I/O profile.
• 105 GB/s and 8 GB/s throughput for 100% sequential-read and 100% sequential-write I/O profile, respectively, using 512k block sizes.
• Even with two nodes failed, the cluster delivered 3,842,997 IOPS at under 1-millisecond latency using a 4 KB block size and 100% random-read I/O profile.

Dell Technologies and the authors of this document welcome your feedback on the solution and the solution documentation. Contact Dell Solutions Team with your comments.

Authors: Michael Lamia

Contributors: Tom Smaistrla, Rizwan Ali, Jim Norton

Before exploring the lab environment and performance testing results, we will first review the process of selecting and sizing an optimized cluster configuration. Dell Integrated System for Microsoft Azure Stack HCI offers a broad range of validated AX node and network topologies to meet the requirements of nearly any use case and workload. Example scenarios include the following:

• For small Remote Branch Offices (ROBO), we may recommend a cluster with two AX nodes using lower core count processors, modest amounts of RAM, all-flash storage, and a switchless storage network topology.
• For applications with significant storage capacity and expansion requirements, we may suggest a two-tier, hybrid storage configuration, and scalable networking topology.
• For the most demanding workloads in the industry, we often select processors with the highest number of cores and fastest clock speeds, large memory footprints, NVMe drives, and 100 GbE RDMA networking.

The following figure illustrates our specification sheet at a glance. It depicts the component combinations in our four AX node models.

Dell Technologies follows a consultative and systematic approach to helping IT professionals modernize their infrastructure. Before recommending any configurations like those mentioned above, we begin with a thorough analysis of currently running workloads. Dell Live Optics is an industry-standard method of impartially documenting server and storage configuration and performance. By recording and exploring the current state of IT environments, we can help accelerate informed decisions and reduce the risk of under or overprovisioning the infrastructure.

An in-depth discussion of Live Optics is beyond the scope of this white paper. From a high-level, customers first engage with their Dell Technologies account team to have an account created on the Live Optics portal. Then they download and run the Optical Prime data collector software to capture inventory and performance insights from current hosts and VMs. Collectors can run from four hours to seven days, in offline or online mode, to create a local file of the results or stream the data securely to the associated online viewer profile. When data collection is complete, Live Optics visualizes the data using various rich reports available in the portal, such as the examples depicted in the following figures.

The Dell Technologies account team and customers use the data that Live Optics collects in order to influence selections in our exclusive Azure Stack HCI Sizer Tool. This tool integrates all the Azure Stack HCI design constraints and best practices from extensive engineering validation efforts. It provides peace of mind knowing that each potential configuration that the tool produces is an optimized and viable option. We work with our customers to generate multiple configurations for comparison. We then select the configuration that meets current demand and future growth projections within the allocated budget.

In this section, we share the results of synthetic testing that was performed on Dell Integrated System for Microsoft Azure Stack HCI running four AX-7525 nodes. We used VMFleet 1.0, as many Microsoft customers and partners rely on this tool to help them stress-test their hyperconverged infrastructures. VMFleet consists of PowerShell scripts that deploy VMs to a Hyper-V cluster and run Microsoft's DISKSPD within those VMs to generate I/O. DISKSPD is helpful for creating synthetic workloads to test an application's resource utilization before deployment.

Figure 4 shows the cluster's physical networking as built in the lab. We set up a scalable, fully converged network topology with management, VM, and RDMA storage traffic traversing two 100 GbE Mellanox CX-6 adapter ports. With this topology, Data Center Bridging (DCB) configuration was required on the S5232F-ON ToR switches. We also required Quality of Service (QoS) configuration in the host operating system. Figure 5 shows the virtual network configuration on the host operating system using Switch Embedded Teaming (SET).

Ancillary services that are required for cluster operations such as Active Directory, DNS, and a file server or cloud-based witness are outside the scope of this white paper. For information about all the steps that are required to deploy this cluster, see our Deployment Guide and Network Integration and Host Network Configuration Options Guide.

The following two tables provide additional detail about the cluster configuration and AX-7525 specifications as built in the lab. We chose the three-way mirror resiliency type for the volumes we created with VMFleet because of its superior performance compared to erasure coding options in Storage Spaces Direct. The raw storage capacity of the cluster was a little over 150 TB. This resulted in just under 50 TB of usable storage capacity due to three-way mirroring's capacity efficiency of 33 percent. Microsoft and Dell Technologies also recommend leaving reserve capacity equivalent to the capacity of one drive per server.

We ran the VMFleet tests under the following conditions:

• Healthy cluster running 64 VMs per node
• Healthy cluster running 32 VMs per node
• Degraded cluster with one node failed
• Degraded cluster with two nodes failed

Healthy cluster running 32 VMs per node

To prepare for the other three test scenarios, we redeployed VMFleet with 32 VMs running per node. We then stressed the storage subsystem using I/O profiles more reflective of the types of demanding workloads that are found in the modern enterprise. These applications had smaller block sizes, random I/O patterns, and various read/write ratios.

The 8 KB block size and 30 percent random write I/O profile is typical of an Online Transactional Processing (OLTP) workload. We observed VMFleet driving over 1.6 million IOPS at slightly over 1-millisecond average latency with this I/O profile, indicating that the cluster has the potential to accelerate OLTP workloads and improve end-user response time to database applications. Even increasing the block size to 32 KB and increasing the write I/O to 50 percent produces over 400,000 IOPS at under 7-millisecond latency.

The following figure depicts the results of this workload testing.

Online Analytical Processing (OLAP) workloads focus on retrieval and analysis of large datasets. We selected larger block sizes and sequential I/O patterns to test workloads in this category. Throughput became the key performance indicator to analyze. The results that are shown in the following figure indicate an impressive sustained throughput that can greatly benefit this category of OLAP workloads. Other types of IT services that could benefit include file and application backups and streaming video.

Testing results on a degraded cluster

We simulated two different failure scenarios on the healthy cluster running 32 VMs per node. In the first scenario, which is depicted in the following figure, we shut down one node, which had the following impact:

• The cluster moved the CSV owned by Node 4 to Node 2. Node 2 became the new owner of that CSV.
• We live-migrated the failed VMs to Node 2 to ensure they were running on the same host that owned the CSV containing their virtual disks.

In the second scenario, which is depicted in the following figure, we shut down two nodes, which had the following impact:

• The cluster moved the CSVs owned by Nodes 3 and 4 to Nodes 1 and 2, respectively. Nodes 1 and 2 became the new owners of these volumes.
• We live-migrated the failed VMs to Nodes 1 and 2, ensuring that all VMs were running on the host that owned the CSV containing their virtual disks.

According to the VMFleet test results, we continued to see outstanding performance even with the cluster in a degraded state. This allows system administrators extra time to recover from node failures without negatively impacting the end-user response times of their most critical applications. The following table compares the testing scenarios using the I/O profiles that are aimed at identifying the maximum thresholds.

 Table 5.  Performance comparisons 

I/O profiles	Healthy cluster	One-node failure	Two-node failure
B4-T2-O32-W0-PR	IOPS: 4,856,796 Read latency: 378 microseconds	IOPS: 4,390,717 Read latency: 380 microseconds	IOPS: 3,842,997 Read latency: 262 microseconds
B4-T2-O16-W100-PR	IOPS: 753,886 Write latency: 3.2 milliseconds	IOPS: 482,715 Write latency: 5.7 milliseconds	IOPS: 330,176 Write latency: 11.4 milliseconds
B512-T1-O8-W0-PSI	Throughput: 91 GB/s	Throughput: 113 GB/s	Throughput: 77 GB/s
B512-T1-O1-W100-PSI	Throughput: 8 GB/s	Throughput: 6 GB/s	Throughput: 10 GB/s

The following figure shows the IOPS and latency results from the VMFleet tests. These tests used I/O profiles that are more representative of real-world applications with smaller block sizes, random I/O patterns, and various read/write ratios. Notice that IOPS decrease and latency increases for the 4k and 8k block sizes, as one would expect. With the 32k block size, we observed the following behavior:

• Latency is less variable across the failure scenarios. We found an 8 percent decrease in latency after the single-node failure and a 14 percent decrease in latency after the two-node failure. This was primarily because write I/O did not need to be committed across as many nodes in the cluster. After the single-node failure, we observed that the network traffic between nodes decreased by more than 3 Gbps per node.
• IOPS increased by 20 to 30 percent after the two-node failure, depending on the percentage of writes in the I/O profile. There were two reasons for this behavior:
  • The 3-way mirrored volumes running on the healthy cluster became 2-way mirrored volumes running on the two remaining online nodes. With 33 percent fewer backend drive write IOPS, the overall drive write latency decreased, which drove higher read and write IOPS. However, this is only in cases when CPU was not the bottleneck.
  • Each of the two nodes remaining online were running double the VMs for a total of 64 VMs per node. The more VMs running on a virtualized host, the greater the potential for higher IOPS.

Note: After shutting down the nodes in the failure scenarios, we live-migrated the VMs to the same nodes that owned the CSVs hosting their virtual disks to avoid any I/O redirection, which would negatively affect performance.

The following figure shows the throughput results from the VMFleet tests using I/O profiles with larger block sizes and sequential I/O patterns. Throughput decreased an average of 17 percent after the single-node failure. However, throughput increased by an average of 52 percent after the two-node failure for the same reasons as applied to Figure 10.

• The 3-way mirrored volumes running on the healthy cluster became 2-way mirrored volumes running on the two remaining online nodes. With 33 percent less backend drive write throughput, the overall drive write latency decreased, which drove higher read and write throughput. However, this is only in cases when CPU was not the bottleneck.
• Each of the two nodes remaining online were running double the VMs for a total of 64 VMs per node. The more VMs running on a virtualized host, the greater the potential for higher throughput.

The performance results described in this document are specific to the components, component configuration, and test tools outlined in Tables 1 - 4. Similar results should not be expected with any deviations, including, but not limited to, the following:

• Different number of physical cluster nodes or VMs
• Different Windows Operating Systems (for example, HCI OS/WS19/WS22)
• Different cluster node or VM hardware configurations processors, memory capacity, storage subsystem, and network adapters
• Different versions of test tools such as DiskSPD or VMFleet
• Different network topologies, switch models, switch operating system, or switch configuration

To keep Dell Integrated System for Microsoft Azure Stack HCI running at peak performance and reliability, IT administrators must take the appropriate steps to monitor and maintain it correctly. Dell Technologies provides the expertise to make Azure Stack HCI operations efficient and worry-free. Administrators should first familiarize themselves with the Operations Guide – Managing and Monitoring the Solution Infrastructure Life Cycle. The following are some of the topics that are covered in this guide:

• Expanding clusters by scaling up and scaling out
• Recovering from AX node hardware and operating system failures
• Monitoring the hardware and software using Microsoft Windows Admin Center and Dell OpenManage Integration for Microsoft Windows Admin Center
• Updating the operating system, BIOS, firmware, and drivers with a simple consolidated workflow in Windows Admin Center

For any Azure Stack HCI cluster expected to deliver high performance, administrators must rely on specially trained experts who can help accelerate time to value and decrease mean time to resolve (MTTR) failures. Dell Technologies makes services simple, flexible, and worry-free - from installation and configuration to comprehensive single-source support. Certified deployment engineers ensure accuracy and speed, reduce risk and downtime, and free IT staff to work on higher value priorities. One-stop cluster level support covers the hardware, the operating system, the hypervisor, and Storage Spaces Direct.

When it comes to expanding clusters, customers can choose to follow the prescriptive guidance in the Operations Guide or enlist the help of our support and services. In addition to improving the storage performance of the cluster, adding AX nodes increases storage capacity and provides more compute resources to the VMs. Administrators must verify hardware symmetry before attempting any expansion operations. This means that the processor model, HBA, and network adapter configuration in the new AX node must be identical to the existing nodes.

For the full list of requirements for scaling out the cluster to include additional nodes and scaling up existing nodes, see the Operations Guide.

Life Cycle Management (LCM) is a priority for every IT operations team. LCM is especially critical for Azure Stack HCI clusters running an organization's most resource-intensive, mission-critical workloads. This practice ensures that systems run at peak performance and availability, maintain a sound security posture, and benefit from newly released features and functionality.

Dell OpenManage Integration with Microsoft Windows Admin Center makes the update process automated, repeatable, and predictable. The OpenManage Integration extension includes a specialized snap-in to Microsoft's Cluster-Aware Updating extension. This innovative collaboration between Microsoft and Dell Technologies creates a single-click full stack update experience. A simple Windows Admin Center workflow orchestrates Azure Stack HCI operating system and AX node BIOS, firmware, and driver updates, requiring only a single reboot per node and resulting in no impact to running workloads in the VMs. The following figure depicts the initial steps in the updates workflow that check for AX node authenticity, correct licensing, and hardware symmetry.

For a detailed technical walkthrough of the full stack life cycle management with the Cluster-Aware Updating workflow in Windows Admin Center, see the OpenManage Integration Technical Walkthrough blog.

It is also critical to monitor a high-performing cluster to ensure it is meeting SLAs. Windows Admin Center with the Dell OpenManage Integration extension provides local hardware and software monitoring in-band and agent-free. Since Azure Stack HCI is a service in Microsoft Azure, administrators can also take advantage of native integration with a host of Azure management and governance services such as Azure Monitor, Azure Security Center, and Azure Policy. This native integration is made possible in large part by Azure Arc.

Azure Arc enables administrators to simplify the management and governance of their on-premises workloads by projecting existing resources into Azure Resource Manager. Once onboarded to Azure Arc, VMs, Kubernetes clusters, and databases can be monitored and managed with the same familiar tools that are used for Azure resources. Tools include the Azure Portal and programmatic means such as PowerShell, Azure CLI, and the REST API.

Demanding applications require a modernized infrastructure that is intelligently designed and correctly sized to meet current performance SLAs and to scale easily to accommodate future growth.

In this white paper, we discussed the systematic and consultative approach Dell Technologies takes in the presales process using Live Optics and our exclusive Azure Stack HCI Sizer Tool. We also shared synthetic testing results using VMFleet 1.0 on a Dell Integrated System for Microsoft Azure Stack HCI cluster running four AX-7525 nodes and two S5232F-ON switches. These results indicated that the model cluster we set up in the lab was capable of impressive performance - even running in a degraded state with two nodes failed. Finally, we highlighted best practices surrounding support, maintenance, life cycle management, and monitoring for resource-intensive infrastructures.