Notice: Dell Technologies announced the availability of new PowerStore models in July 2022. The new PowerStore models offer greater performance, more features, and capacity compared to previous storage arrays. In this paper we optimize Oracle database performance on the PowerStore 1000T. All the recommendations and best practices in this paper also apply to the new PowerStore arrays.

As the number of applications and databases has increased for most businesses, so has the infrastructure that supports them. These increases have led to challenges, including data silos, increased management complexity, and rising costs—all of which can hinder an organization’s ability to achieve Oracle and IT modernization goals.  

Database consolidation has historically been viewed as an antidote for many of the challenges that come with database sprawl and data storage silos. Strategies for consolidation must ensure that the availability and performance of business-critical applications are prioritized, while also maintaining low latency with fewer resources. The introduction of faster, more powerful CPUs and new storage technologies have made it possible for businesses to achieve consolidation imperatives with greater confidence. 

However, IT staff may not always be able to perform a consolidation strategy as quickly as management requires. It can take months to plan, procure, deploy, integrate, test, and fine tune the new consolidated architecture before you can push applications live. In a 2022 ESG study, Accelerate IT Modernization with Dell Technologies Validated Designs, 90 percent of surveyed IT decision makers reported that their organizations need to move faster when deploying applications, infrastructure, and services compared to three years ago.  

In support of database consolidation and modernization strategies, a solution that is pre-designed and tested to avoid impacts on Oracle database performance and cost can reduce burdens on staff and budgets. This results in shorter deployment timelines, faster time to value, greater focus on innovation, and a faster timeline for services to be released to the market.

The purpose of this document is to outline the power of a Dell integrated infrastructure for Oracle Databases, including design concepts and validated configurations. The focus is on deploying an optimized environment through hardware performance and vetted best practices. This paper focuses not only on the database, but all aspects of the underlying infrastructure including server, VMware, storage, and network.

This document is intended for solution architects, data scientists, database administrators, VMware administrators, and storage administrators.

In today’s market, businesses are mandating greater consolidation of IT infrastructure to lower data center footprints and operational costs, decreasing the number of environments with large data centers with information silos. Some businesses are moving away from on-premises data center to the public cloud entirely. Those who are still operating on premises are looking for avenues to manage IT infrastructure costs. Lowering cost, however, is not always possible because businesses are growing daily, requiring more IT infrastructure to support the rapid growth of data.

To meet the business need for quicker time to market, IT teams must deploy new infrastructure solutions quickly and support it according to the service level agreement (SLA). This presents a challenge for IT teams in terms of speed of deployment.

Building an infrastructure solution for such a requirement is a time-consuming and complex process, especially for Oracle database solutions, as Oracle solutions demands high performance and robustness. Building this type of solution requires both hardware and software expertise.IT teams must spend time researching for the right hardware vendors, develop a professional relationship with these vendors, procure the equipment for a proof of concept, and perform in-house tests to ensure that all layers of the solution work as expected. 

The initial proof of concept phase ensures that the infrastructure is working without any hardware failure. Servers, switches, and disk storage fail from time to time. The key is how the solution recovers during a hardware failure, such as a failed disk, while maintaining performance and still providing data protection during the recovery.

The second phase ensures that performance is sustainable for the infrastructure solution.  This requires product knowledge and expertise to tune each layer properly. Even though all vendors provide their product best practices that can be referenced but not all best practices of each component will work when putting all pieces together.

Not only the initial setup is time-consuming and complex, but the infrastructure solution also requires life cycle management. This includes ongoing maintenance by the IT and database administration teams. To do so, a separate identical infrastructure is required for testing with newer firmware or software as they become available.

Dell Validated Designs for Oracle databases address these obstacles and barriers to success. These validated solutions undergo vigorous testing with best practices applied at all layers. This ensures confidence for IT, knowing that the work has been performed by vendor experts.

This document describes a validated design infrastructure for Oracle database workloads. It consists of servers, network switches, and storage. This validated design has been vigorously tested by Dell engineers to ensure it is optimized for Oracle databases. The entire design can be customized based on database workload requirements and provides businesses increased flexibility.

The server in this validated design is the Dell PowerEdge R750 which is a 2U dual-socket rack server. It is powered by 3rd Generation Intel Xeon scalable processors. The servers can be configured with different CPU types such as the number of cores per processor and clock speed best suited for the workload requirements.

The network switch is the Dell PowerSwitch S5224F-ON. It has 24 x SFP28 ports that can be configured with 10 GbE or 25 GbE for data network. It also has 4 x 100 GbE QSFP28 ports that can be used for uplinks.

The storage system is the Dell PowerStore 1000T. It is a 2U two-node dual socket Intel® Xeon® platform. It is a midrange, modern storage infrastructure that is engineered for data-centric workloads. It is designed for performance, scalability, and efficiency. The PowerStore platform includes multiple models that can be used for this solution design. Each model has different specification such as supported drive types, I/O modules, expansion enclosures, NVRAM drives, and more. These specifications can be tailored to any database workload requirement. The following figure shows an overview of the Dell optimized Oracle Database platform Validated Design.

Figure 1.   Optimized Oracle Database platform architecture

Dell customers are aware that running Oracle database workloads requires a robust infrastructure. The various infrastructure components must be carefully researched and tested once they are procured. If the components are not from the same vendor, the database becomes more difficult to manage because it requires support from multiple vendors to troubleshoot problems as they arise. Another potential issue is that each vendor deploys their own set of best practices, meaning a solution one component may conflict with another component and could disrupt optimal infrastructure function.

This solution solves those problems, as the entire system is designed for Oracle workloads. Dell engineers have expertise in Oracle databases, Linux, and VMware. This validated design was properly configured, tested, and tuned according to best practices, avoiding any potential failures or outages.

This infrastructure solution addresses both bare metal and virtualized environments for Oracle databases. It allows businesses to quickly stand-up Oracle database platforms to roll out a new business line or meet deadlines. This infrastructure also quickly provides a tested Oracle platform solution for application developers to test new codes and free up time so they can focus more on value-added tasks.

This optimized Oracle Database platform was built using Dell’s state of the art PowerEdge R750 servers, PowerSwitch S5224F-ON, and PowerStore 1000T storage system. These components are Dell’s flagship products. Dell uses these building blocks to create validated designs for Oracle database workloads. This validated design also includes best practice guidance in the form of toolkits and scripts which can be downloaded from GitHub.

Dell engineers used a HammerDB test tool to simulate an Oracle OLTP workload to ensure that the infrastructure is optimized for performance. During the testing, Dell applied best practices to each of the hardware and the software stacks. The software stack includes Oracle database 19c Grid Infrastructure, Oracle database 19c software, Red Hat Enterprise Linux 8.5 x64, and VMware vSphere 7.3.

Dell PowerEdge servers include multiple BIOS profiles. Each profile, when set, changes certain BIOS settings on the server. Dell engineers tested Oracle OLTP workloads with different BIOS settings to determine the best performance outcome. From these tests, they designed a customized profile to allow users to customize their configurations when deploying Oracle databases. The following list includes available BIOS profiles that come with the Dell PowerEdge servers.

• Virtualization Optimized Performance Profile
• Virtualization Optimized Performance Per Watt Profile
• Database Optimized Performance Profile
• Database Optimized Performance Per Watt Profile
• SDS Optimized Performance Profile

The recommended profile for Oracle OLTP workloads is Database Optimized Performance Profile. Select this profile when you are setting up the servers for Oracle database. When deploying Oracle database in a virtualized environment, Virtualization Technology and x2APIC must also be enabled in addition to this profile. These two additional settings are not required when deploying Oracle databases on bare metal servers. For security reasons, the following ports were also disabled in the BIOS.

• User accessible USB ports
• Internal USB ports
• iDRAC Direct USB port
• Serial communication

For detailed explanation of each BIOS profile, see CPU Best Practices.

The following table shows the PowerEdge R750 details used for this Validated Design.

Table 1.      PowerEdge server configuration details

The Dell PowerStore is a 2U two-node dual socket Intel® Xeon® platform. It is an active/active configuration where both nodes (controllers) can serve data to the volumes. However, only one node can own a volume at a time. For this validation, Dell engineers manually configured the volumes with node affinity to provide the best performance possible for running Oracle database workloads. PowerStore node affinity provides the load balancing of the volumes created on the array to maximize performance. In later versions of the PowerStore operating system, the node affinity is automated during the volume creation process.

PowerStore has four Fibre Channel front-end I/O ports per node for storage connectivity.  To maximize storage performance, all four front-end I/O ports on each PowerStore node are zoned to all PowerEdge HBA ports. The following table provides an overview of a PowerStore 1000T model.

Table 2.      PowerStore 1000T configuration details

PowerStore has many features such as volume performance policy, volume group, and snapshot. The volume performance policy consists of three priority levels: High, Medium, and Low. This policy controls the quality of service for the data path to the volume. The default priority setting for newly created volumes is medium.

The volume group feature provides users with the ability to snapshot more than one volume at the same time. Oracle requires all volumes belonging to the database to be snapshotted atomically for a functional snapshot of an Oracle database. Depending on the business requirements, Oracle snapshots can be used to create clones for dev/test or other purposes. 

This validated design includes four storage volume groups. This design was to allow granularity for the database administrators to only snapshot what is necessary, therefore saving capacity. Data in the TEMP tablespace is not required when snapshotting the database and resides in its own storage volume group. Even though, a single instance database was tested, a separate ASM disk group named “GRID” was created for the Oracle Restart option.  This ASM disk group is not required for a functional database snapshot; therefore, it has its own storage volume group. The following list describes the four storage volume groups:

• VM_OS volume group
• VM_GRID volume group
• VM_DB1 volume group
• VM_TEMP volume group

The VM_OS storage volume group has one volume configured with the PowerStore volume performance policy of medium (default). This volume group was for the Linux operating system.

The VM_GRID storage volume group contains three volumes configured with the PowerStore volume performance policy of medium (default).  Since Oracle Restart was configured, a separate ASM disk group was created for Grid Infrastructure and the ASM registry file.   

The VM_DB1 storage volume group contains twelve volumes. This volume group consists of the following ASM disk groups:

• DATA_DG
• REDO_DG
• FRA_DG

Each of these ASM disk groups has four volumes. The DATA_DG volumes were configured with the PowerStore volume performance policy of medium. The REDO_DG volumes were configured with the PowerStore volume performance policy of high. The FRA_DG volumes were configured with the PowerStore volume performance policy of low.

The VM_TEMP storage volume group has one volume and was configured with the PowerStore volume performance policy of medium (default).  Since the temp tablespace is not needed for database snapshots, a dedicated storage volume group was created.

The following table provides an overview of the volume groups created for this validation. 

Table 3.      
Optimized volume groups configuration for Oracle database

For detailed PowerStore best practices and features, see Storage Best Practices.

In addition, the PowerStore also includes Oracle Enterprise Manager (OEM) plugins to allow users to gain performance insights of the PowerStore from the OEM user interface.

The network stack consists of the Dell PowerSwitch S5224F-ON, which has 24 x SFP28 ports that can be configured with 10 GbE or 25 GbE and 4 x 100 GbE QSFP28 ports.  We configured this switch with 25GbE for the public network and the VMotion network during the tests. We also enabled Jumbo Frame for these two VLANs. When possible, users should consider Jumbo Frame for better performance. When using Jumbo Frame, it is extremely important to ensure that all devices on the same VLAN are configured with Jumbo Frames, or the system will not function properly. In an Oracle RAC deployment, the private interconnect between RAC nodes should be configured with Jumbo Frames. The following table provides an overview of the PowerSwitch.

Table 4.      Dell PowerSwitch S5224F-ON configuration details 

The hypervisor layer is the VMware vSphere. vSphere provides four different power policy settings that can be selected. They are:

• High Performance: Do not use any power management features.
• Balanced (Default): Reduce energy consumption with minimal performance compromise.
• Lower Power: Reduce energy consumption at the risk of lower performance.
• Custom: User-defined power management policy.  Advanced configuration becomes available.

The High Performance power policy was chosen during the test because it provides the best performance. Path Selection was set to “policy=rr, iops=1” when using the PowerStore storage system.  For more information on VMware ESXi host power management policies, see VMware Best Practices.

The operating system stack for the Oracle database virtual machine is Red Hat Enterprise Linux x64. In Linux, “tuned” is a profile-based system tuning tool that uses the udev device manager to monitor connected devices and enables both static and dynamic tuning of the system settings. It comes with several pre-defined profiles for common use cases, but the tuned service also enables you to define custom profiles.

Oracle software installation requires certain kernel settings, meaning Oracle provides a pre-packaged RPM called oracle-database-preinstall. However, these settings may not always be optimal for OLTP workloads. To determine optimal settings for these kernel parameters, Dell engineers performed various tests. The engineers used a tuned profile to accelerate the implementation of these settings. This profile included the recommended settings for Oracle OLTP workloads, which can be placed in a file and enabled with the tuned-adm command. In addition to the Oracle OLTP profile, HugePages parameter should also be enabled. Oracle provides a script called hugepages_settings.sh which can be used to get the recommended value for the Linux vm.nr_hugepages parameter. To use the HugePages setting, the transparent_hugepage parameter should be disabled by setting it to never. For more detail on the Linux operating system kernel parameters best practices, see Linux Best Practices.

The link to the Oracle script can be found here. 

Figure 2.   Oracle Database VM layout

ASM disk group settings for the Oracle ASM are displayed in the following table. Selecting multiple volumes for the GRID disk group was based on Oracle’s recommendation for high availability when deploying RAC. In this Validated Design, Dell followed this recommendation for a single instance database with Oracle Restart option.  This recommendation is not needed in a production environment. The DATA, REDO, and FRA disk groups also contain multiple volumes. This provides better performance because of multiple devices and higher I/O concurrency. The GRID disk group was configured with an ASM Normal Redundancy setting, as described in Oracle best practices, while other disk groups were configured with External Redundancy. Since the PowerStore has built-in redundancy, it is not necessary to apply a Normal Redundancy setting for other disk groups; which saves capacity on the storage.

Table 5.      Oracle ASM disk groups configuration

Figure 3.   Oracle ASM disk groups per VM

While testing OLTP workload using the HammerDB test tool, the Oracle online redo logs were configured to prevent frequent log switching. If database switches online redo log too often, it will affect database performance. We look at the AWR reports to determine the right size of the redo logs and adjust accordingly. In a production deployment, the number of redo logs and sizes also depend on the Recovery Time Objective (RTO) and Recovery Point Objective (RPO). Sizing incorrectly could affect database performance and business RTO and RPO requirements. The following table shows the online redo log layout during our tests.

Table 6.      Oracle online redo logs configuration 

It is important to size the SGA properly when deploying Oracle databases. If the SGA is too small, excessive physical I/O is unavoidable and could lead to poor database performance. To reduce the time that the database is waiting on I/O, the engineers increased the SGA from 128GB (default) to 256GB during testing and observed a significant increase in performance. To determine the optimal setting for the SGA requires examination of the Oracle AWR report. Changing it arbitrarily would not help the performance but could compromise it. It is important to thoroughly examine this report to assign the proper SGA size for you system because every Oracle workload is unique.

Another database setting to consider adjusting is the Oracle undocumented parameter _high_priority_processes. This parameter allows the LGWR process to have higher CPU priority. When running an Oracle database in a production environment, it is best to consider setting this hidden parameter to “LMS*|VKTM|LGWR” for better database performance. However, since this is an undocumented database parameter, consider consulting Oracle Support before using it. For more information on the database best practices, see Database Best Practices.

This solution architecture for Oracle brings together an optimized database infrastructure that offers flexibility in terms of hardware resources. It has been thoroughly tested by Dell engineers to that ensure all layers, hardware and software, work together. The best practices for each component have been tuned and validated to provide a robust infrastructure for Oracle database OLTP workloads deployments. This work is a time-consuming process due to its complexities and having it validated before procurement saves valuable time for the system administrators.

Database Solution Optimizer (DSO) is a Dell’s toolkit that allows System Administrators and DBAs to enable best practices at each layer in the solution stack. This toolkit allows the IT teams the ability to configure the solution with best practices during the initial stand-up of the infrastructure.  It operates using a module-based approach allowing the IT teams the granularity and flexibility.  The following modules are currently available with this Validated Design.  Please note that this toolkit is not a supported product.

• PowerEdge server BIOS module
• Red Hat Enterprise Linux Operating System module
• Storage System module
• Report module
• Oracle module

This toolkit is a work in progress and more modules will be available in future releases.  Those modules will include:

• Log module
• VMware ESXi module
• Network module

Figure 4.   Database Solution Optimizer modules

BIOS Module

This module interacts with iDRAC Redfish Rest API to allow System Administrators to select the proper profile setting without having to go into the BIOS mode and change various parameters which can be error prone.  The recommended profile for Oracle OLTP workload is Database Optimized Performance Profile.

Red Hat Enterprise Linux Operating System Module

This module allows System Administrators the ability to verify and set kernel parameters according to best practices such as using the correct tuned profile for Oracle database and the setting of HugePages. 

Storage Module

This module uses storage Rest API to interact with the storage system to gather information about the data path connectivity, volume groups, and LUNs details.  System Administrators can verify if the number of data paths, volume groups, and LUNs have been adequately configured based on best practices. In future releases, the same module can be used to applied recommended settings.

Report Module

This module provides the ability to compare the current settings with the recommended settings for all components in the stack.  This will allow System Administrators and DBAs to verify if all settings in the stack have been set according to best practices.   It will be available in HTML format in the future release so it can be viewed with any web browser.

Oracle Database Module

This module provides the DBAs insights into the database initialization parameters settings.  The DBAs can use this module to retrieve the current settings related to database and storage and change them according to best practices. This ensures that the database is operating optimally within the whole stack.

Log Module

The purpose of this module is to log all operations when this utility is invoked.  This allows an audit trail to the System Administrators and DBAs of what has been done to the whole solution.

VMware ESXi Module

This module provides the ability to gather ESXi related parameters and configuration for best practices.  Parameters such as SCSI controllers, path policy, and ESXi power policy are some examples that will be verified to ensure they are set properly.

Network Module

This module allows IT teams to verify if network settings have been configured based on best practices.  One such setting would be the MTU size for private and VMotion VLANs.

This optimized Oracle Database platform provides a validated solution for Oracle customers that focuses on maximizing flexibility in design and robustness. It has been vigorously tested with actual Oracle OLTP workloads to ensure all hardware and software are working according to best practices.

This validated design was built for Oracle databases, allowing customers to speed up the deployment of an Oracle database infrastructure in less time. This saves time, resources, and mitigates risk as customers are not required to procure the necessary components and perform their own tests to ensure that the layers all work together. This offers a significant advantage for IT teams and administrators, helping them to move more quickly and confidently on database consolidation strategies for Oracle, eliminating data silos and freeing up resources for business line rollouts and mission-critical deadlines.

This solution provides a simple starting point for deploying Oracle database workloads that otherwise would be a complex, risk-prone and time-consuming process.

Dell Technologies and the authors of this document welcome your feedback on the solution and the solution documentation. Contact the Dell Technologies Solutions team by email.

Author: Tom Dau

Contributors: Ava English

Note: For links to additional documentation for this solution, see the Dell Technologies Solutions Info Hub for Oracle.