Two validation tests were combined into one best practice for the PowerMax Storage Directors. The first test involved configuring the PowerMax Director to only use the interfaces used by the business. For the second-best practice, we increased the number of front-end connections to improve bandwidth between the PowerMax storage and PowerEdge servers. We combined these best practices as they both apply to configuring the PowerMax Directors. 

Overview of Optimizing Director Interfaces

PowerMax storage arrays use bricks that enable customers to add more power, performance, and capacity to an existing array. Each PowerMax brick has two engine directors. These directors support multiple functions, including front-end I/O modules. The front-end I/O modules provide connectivity to the array using various interfaces including Fibre Channel SCSI, Fibre Channel NVMe, iSCSI, and others. In the best practice validation tests, only the Fibre Channel interface was used, which provided the ability to disable unused interfaces like Fibre Channel NVMe and iSCSI. 

Unused interfaces on the directors consume processor resources even when running in an idle state. By disabling interfaces, the business does not plan to use the directors can direct more processing power to the interfaces that are being heavily used. The goal is to optimize the director interfaces to support only those used by the business to optimize performance. 

This best practice originated from Dell EMC support. It is highly recommended to work with the support team to validate whether this director interface optimization will work for your business.

Overview of Adding Director Front-end I/O Modules

There are four front-end I/O modules per director. In the baseline configuration, two of the four total directors were used. The following table shows all four front-end I/O modules were used on director 2 and two front-end modules were used on director 4. The two other front-end modules on director 4 were reserved for replication. Each front-end module was a 32 Gbps Fibre Channel connection and six total were used in the baseline connection. 

Table 1: Front-End Director Port Baseline Configuration

For this best practice two additional directors were added (1 and 3) with all the front-end I/O modules for a total of eight more connections to the server. The following table shows the two directors and eight ports that were added (marked in darker grey):

Table 2: Front-End Director Port Optimized Configuration

With the additional directors and front-end ports, this best practice showed the greatest improvement of all the storage recommendations. Doubling the connections to the database server provided substantial improvements. 

Recommendation

The combination of the two PowerMax Director best practices (removing unused interfaces and connecting additional front-end I/O modules) provided a substantial performance boost in the following metrics: 

• New Orders per Minute (NOPM) 
• Transactions per Minute (TPM) 
• Server CPU utilization
• PowerMax IOPS 
• PowerMax Average Read Response Time
• PowerMax Average Write Response Time

Both TPC-C metrics (NOPM and TPM) showed a substantial improvement – the number of orders completed and transactions processed increased significantly. Correspondingly, array IOPS also increased significantly. The PowerMax average read and write times dropped significantly, indicating faster responses from storage. 

Server CPU utilization went up which is a positive indication of more efficient use of the cores in the server. Until this best practice was implemented processors were likely waiting on read and write I/O from the storage array. Now that the average read and write response times are faster the processors are processing more data thus, are more efficiently used. 

Overall, the best practice of adding PowerMax directors and front-end I/O modules and disabling unused interfaces in the director can significantly improve performance. The recommendation is to implement this best practice as a Day 1 activity, as it is a highly recommended configuration that optimizes database performance.   

Implementation Steps

In this best practice we removed all the unused iSCSI and NvME emulations and utilized all of the ports of the FE director. In all other cases we tested using two of the FE director ports. 

We used FC connectivity from our servers to our storage. For case 6 part 1, we removed additional iSCSI and NvME emulations on the PowerMax. This was useful since our tests were conducted using FC as the backend connectivity between our servers and storage. Unisphere provided the capability to manage the emulations available on the PowerMax storage. These are the necessary steps:

1. To remove unused partitions, log into the Unisphere console with the appropriate credentials.

2. Select the storage section and navigate to the System panel. Navigate to Hardware and click on the Manage Emulation button.

3. Click Remove Director Emulation and then click NEXT. Select the emulation to be removed on the next pane. Review your changes and click FINISH.

 If there are no conflicts, the change should proceed. If you encounter any errors, contact PowerMax support.  

For case 6 part 2, we added one additional port to each director, effectively doubling the number of ports available. To make this change, first create the required FC connections from the storage ports to the switches and zone them appropriately, in other words ensure that the updated zoning maps the new ports to the host (server) ports. Contact Brocade support if there are any issues.

1. Once zoned, navigate to the Hosts pane in Unisphere and select the sub-option for Port Groups.

2. Select your port group and click the Modify button. Select the ports to add to the port group, click on the arrow button next to Add to Job List and then click Run Now.

Additional Resources

This best practice was referenced from the Deployment Best Practices for SQL Server Database with Dell EMC PowerMax. 

Round Robin path policy defines how to balance I/O load across active storage paths. In this best practice, the Round Robin path policy will be changed to optimize performance. 

Overview

At the hypervisor layer, ESXi controls the flow of data to and from the server. There are several configuration choices including Most Recently Used (MRU), Fixed, and Round Robin. In this best practice, we made changes to the Round Robin path policy to optimize data path usage between the PowerEdge servers and the PowerMax storage. 

Round Robin automatically rotates through all available paths to distribute the data flow. As PowerMax has all active paths to storage, Round Robin will rotate data across those pathways. By default, Round Robin sends 1,000 I/O operations across one path before using the next. In this best practice, we change the default of 1,000 I/O operations to 1. With a Round Robin IOPS limit of 1, the paths to storage rotate more frequently, improving overall performance, providing a positive impact upon storage performance. 

Recommendation

Changing the Round Robin Path Policy to 1 showed a significant performance improvement in these performance metrics:

• New Orders per Minute (NOPM) 
• Transactions per Minute (TPM) 
• Server CPU utilization
• PowerMax IOPS 
• PowerMax Average Read Response Time
• PowerMax Average Write Response Time

NOPM and TPM significantly increased when compared to the prior test, indicating the change to the Round Robin path policy improved performance. 

An increase in PowerMax IOPS indicates more efficient use of I/O paths to storage. Optimal path usage benefits the database as the response times for reads and writes decrease. 

Server processor utilization can be impacted by default settings that are not optimized. In this case, the default Round Robin setting of 1,000 I/O operations was impacting performance. Changing the Round Robin setting to 1 I/O before moving to the next path resulted in a more frequent path switch rate, which improved performance. An increase in server CPU use indicates less storage wait time and more efficient processor utilization.   

PowerMax IOPS significantly increased, indicating that optimizing Round Robin multipathing enabled storage to process more I/Os per second. Both the PowerMax average read and write response times decreased. The combined increase in IOPS and decreases in response times means databases can process a greater volume of data faster. 

Overall, the best practice of changing the Round Robin path policy from 1000 I/O operations to 1 is highly recommended and should be considered as a Day 1 practice as part of initial data provisioning. 

Implementation Steps

To configure Round Robin, I/O operations from 1,000 (the default value) to 1, login into the ESXi console and run the following command (with the .xxxx in naa.xxxx matching the first few characters of your NAA ID):

for i in `esxcfg-scsidevs -c |awk '{print $1}' | grep naa.xxxx`; do esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=$i; done

Verify the change using the following command:

esxcli storage nmp device list

Additional Resources

This best practice was referenced from VMware Multipathing policies in ESXi/ESX and Adjusting Round Robin IOPS limit from 1000 to 1.