Notice: Dell Technologies announced the availability of new PowerStore models in July 2022. The new PowerStore models offer greater performance, more features, and capacity compared to previous storage arrays. In this paper, we optimize Oracle database on the PowerStore 1000T. All the recommendations and best practices in this paper also apply to the new PowerStore arrays.

Solution introduction

Dell understands the importance of delivering successful outcomes for our customers. Database management best practices tend to be product specific, making it difficult for customers to understand how to be successful. The primary goal of this design guide is to drive greater success for our customers by moving best practices from product silos to easy-to-use recommendations. The recommendations for our best practices apply across various products.

This program groups best practices based on the layers in the database management stack: storage, servers, VMware vSphere, operating system, and database configuration. We implemented these groups of best practices, monitored the system, and evaluated the test findings by running OLTP workloads to mimic business application systems.

This guide describes design and configuration best practices of a VMware-based virtualized Oracle database solution that uses PowerEdge servers and PowerStore storage. The guide provides an overview of the architecture design and deployment best practices of the entire stack, including the compute, network, and storage design. It highlights the best practices that are based on the layers in the database management stack: storage, servers, network, VMware vSphere, operating system, and database configuration.

Audience

This guide is intended for infrastructure architects, storage administrators, virtualization administrators, system administrators, IT managers, and other personnel who evaluate, acquire, manage, maintain, or operate Oracle database environments.

The goal of the Oracle best practices program is to improve customer outcomes using Dell infrastructure. It can be challenging for customers to determine how an Oracle database should be optimally designed, as research can be time intensive and confusing. This Oracle best practice program centralizes and validates recommendations that will help customers successfully run an Oracle database on our infrastructure. 

We began validating best practices by gathering recommendations from multiple sources and categorizing the guidelines into different layers of the database infrastructure. For example, in the case of the optimal PowerEdge configuration for databases the team used best practices from current manuals and held meetings to discuss the findings with PowerEdge specialists. We actively worked with experts and engineers responsible for each part of the Oracle infrastructure. Below is a list of each infrastructure layer that was validated for best practices, in the order tested:

• Optimal Storage Configuration
  • Create multiple volumes
• Optimal Linux and Database Configuration
  • Configure Linux kernel using TUNED
  • Enable Huge Pages
  • Configure disk I/O scheduler
  • Enable NTP
  • Configure Oracle redo log size
  • Enable Oracle initialization parameter
• Optimal CPU configuration
  • Enable Database Optimized Performance BIOS profile + VT + x2APIC enabled
  • Enable Dell Controlled Turbo
  • Disable unused integrated devices

Before starting the validation work, the Engineering team conducted internal load testing to determine the workload profile. HammerDB was used to generate an Online Transaction Processing Workload (OLTP) that simulates enterprise applications. The goal of generating a significant load on the Oracle infrastructure was to ensure that the system was sufficiently taxed to show how best practices optimized performance. The Hammer DB workload configuration is shown in the following table:

Table 1: Hammer DB workload configuration

With this HammerDB configuration, each best practice was validated in an hour-long workload test: 10 minutes ramp up time plus 50 minutes test duration. This approach of running a workload for one hour to validate each best practice was used to ensure the database system reached a consistent state of execution. Reaching a consistent run state has the benefit of validating whether the configuration is stable and if the best practice shows value over time. 

The HammerDB parameter “Use all warehouses” forces the use of all 5,000 warehouses. This causes the workload to generate more I/O on the storage array. The first set of best practices compares baseline database performance without an optimal storage configuration to a database configuration with an optimal storage configuration. 

New Order per Minute (NOPM) and Transaction per Minute (TPM) provide metrics to interpret the HammerDB results.  These metrics are from the TPC-C benchmark and indicate the result of a test.  During our best practice validation, we compare those metrics against the baseline to ensure that there is an increase in performance.

Within each layer of the infrastructure, the team sequentially tested each component and documented the results.  For example, within the storage layer, the goal was to show how optimizing the number of volumes for DATA, REDO, and FRA disk groups improve performance of an Oracle database.  

With each additional test configuration, the working theory was that we would observe a gain in performance. Thus, with the last test in changing the server CPU parameters, an overall optimal Oracle database solution would be achieved.

To enhance the value of best practices, we have identified which configuration changes produced the greatest results. This should enable anyone reviewing the best practices to easily identify the recommendations that will have the most value. Best practices were categorized as follows:

Day 1 through 3: Most enterprises implement configuration changes based on the delivery cycle:

• Day 1: Indicates configuration changes that are part of provisioning a database. The business has defined these best practices as an essential part of delivering a database.
• Day 2: Indicates configuration changes that are applied after the database has been delivered to the customer. These best practices address optimization steps to further improve system performance. 
• Day 3: Indicates configuration changes that provide small incremental improvements in database performance. 

Highly, moderately, and fine-tuning recommendations: Customers want to understand the impact of the best practices and these terms are used to indicate the value of each best practice. 

• Highly recommended: Indicates best practices that provided the greatest performance in our tests. 
• Moderately recommended: Indicates best practices that provide modest performance improvements, but which are not as substantial as the highly recommended best practices. 
• Fine-tuning: Indicates best practices that provide small incremental improvements in database performance.  

When reviewing the best practices, the day and value of the recommendations are combined. Here are some examples:

1. Day 1, highly recommended: Indicates these best practices should be part of provisioning an Oracle database as they provide the greatest performance. 
2. Day 2, moderately recommended: Indicates these best practices should be considered as part of post-delivery activities that provide modest performance gains. 
3. Day 3, fine-tuning: Indicates these best practices should be considered when small incremental improvements are required to further tune the system. 

With this way of ranking best practices, the hope is that customers can very quickly decide which recommendations will provide the best value. For example, one or two best practices from the section on Linux optimization might provide most of the value depending on the customer’s needs. Investing time in implementing these best practices provides the greatest return. This approach can be taken for each layer of the database system until all Day 1, highly recommended best practices are completed. This differentiates how we present best practices and provides a pathway for customers to have the best return on investment. 

Best practices are broad recommendations designed to apply to most Oracle databases using Dell infrastructure. Every database workload and system is different, meaning the value of these best practices will vary from system to system. As with any configuration or change to a database system, the best approach is to test and validate the change prior to implementing the best practice on a production system. We recommend testing all best practices before implementing the changes in production. 

Architecture design includes various aspects of the solution such as components of physical layer, software layer, and storage volumes design. The architecture was designed to broadly represent the infrastructure that customers use for their Oracle databases. Two nodes of Dell PowerEdge R750xs servers were used for the compute layer, and a single PowerStore 5000T was used for storage needs.

This two-node architecture is built to provide a high availability design. It can be extended to three or more nodes to consolidate more databases.

Figure 1: Architecture design

The server layer consists of database ESXi servers and two R750xs PowerEdge servers running ESXi 7.3 hypervisors for four VMs running Red Hat Enterprise Linux 8.5 guest operating system. Only one of the two servers is actively used for benchmarking and the second server is reserved for high availability.

Each ESXi server consists of:

• Two 25 GbE network interface cards (NICs): Two dual-port 25 GbE network interface controllers for Oracle public and vMotion
• Two 32 Gbps host bus adapters (HBAs): Two dual-port 32 Gbps HBAs for SAN traffic in database servers
• One GbE management rNDC: At least one 1 GbE rNDC or LOM port for in-band management of the server from within the operating system
• One GbE integrated Dell Remote Access Controller (iDRAC): A dedicated iDRAC Ethernet port for out-of-band (OOB) management of servers
• R640 management/tool server: The R640 server runs VMware ESXi 7.3 as the hypervisor to host multiple VMs, as follows:
• VMware vCenter Server Appliance (VCSA) deployed as a VM
• VM with Red Hat Enterprise Linux 8.5 guest operating system installed to run the HammerDB test tool

The network layer consists of the following types of connectivity:

• Management network between vCenter and ESXi hosts. This network also connects the iDRAC Ethernet ports of all the servers
• vMotion network for VM migration using vMotion between ESXi hosts
• Public network that connects the database server and the Oracle database instance to with the rest of the data center network
• SAN FC network that connects database servers and PowerStore SAN storage

The networks are implemented with the following switches:

• One 1 GbE switch for management switches
• Two 25 GbE ToR switches for the Oracle public and vMotion network
• Two 32 Gbps FC switches for FC SAN connectivity

The storage layer consists of the PowerStore T array as the FC SAN storage for databases and the data stores for VM operating system volumes.

The solution is built using two PowerEdge R750xs servers and one PowerEdge R640 server. The following tables describe these components: 

Table 2: PowerEdge R750xs hardware component details 

The PowerEdge R640 is configured as the vCenter server and management server. It includes the following hardware components:

Table 3: PowerEdge 640 hardware component details 

The network hardware consists of one 1GbE management switch, ToR switches, and two FC switches, as shown in Table 4. 

Table 4: Network switches details 

One PowerStore 5000T is used for the storage to virtual machine operating system images and the Oracle database files. Table 5 describes the storage configuration.

 Table 5: PowerStore storage details 

Oracle offers one of the most popular databases in the world. We chose Oracle 19c for our best practices tests because it is a version of the database that many of our customers use. The Oracle database has many features and capabilities. In our database validation tests, the engineering team tested changes to the database configuration that will optimize performance.

Virtualization of databases has become increasingly popular over the years. There are many advantages to virtualizing the database, including consolidation, agility, and ease of management. We chose to virtualize the Oracle database with VMware vSphere version 7.0. With virtualization, another layer of configuration that applies to the database. As virtualization impacts every part of the database infrastructure, our approach was to integrate VMware best practices with each physical layer rather than having a dedicated section for virtualization. For example, storage best practices include recommendations for VMware latency sensitivity and the storage I/O control threshold. This should simplify reviewing best practices, as both physical and virtualization recommendations are presented for each part of the infrastructure.

The Linux operating system has been widely accepted for supporting Oracle databases. Red Hat Enterprise Linux version 8.5 was used in the tests as this Linux flavor provides the stability, reliability, and security required for databases. The same best practice approach was taken with Red Hat Linux, as used for VMware. Linux best practices are integrated into each physical layer of recommendations. It is easy to review Linux recommendations by reading CPU best practices or adding another physical layer.

The software stack includes a ESXi hypervisor, vCenter, VM guest operating system, Oracle grid infrastructure, Oracle database, HammerDB performance test kit, and Nexpose agent for security vulnerability scans.

The following table summarizes the software architecture details used in this solution:

Table 6: Software component list

There are many combinations possible for the software architecture. In testing with Oracle Database19c, Red Hat 8.5, and VMware vSphere 7.3, the goal was to have a design that applies to most of our customers’ databases. Future best practices programs will address new versions of the Oracle database and advances in both the operating system and virtualization.

This Oracle database solution uses a PowerStore T model with a base enclosure that has 21 NVMe SSD drives and four extreme high-performance NVMe NVRAM drives that are designed for the PowerStore caching system. The size of each of the 21 NVMe SSDs was 1.92 TB, providing a total of 28.3 TB physical storage capacity.

The following design principles were applied for the Oracle database storage volumes and related VM operating system storage volumes:

The volumes in this solution are logically grouped and placed in different volume groups (folders in storage user interface). The volume group categories and volume design are explained in the following list:

• VM OS volume group(orabp22-vm1-os) has four volumes: one for each VM Linux Guest operating system
• GRID volume group (orabp22-vm-grid) has 4x3 volumes: Three volumes for Oracle Grid Infrastructure ASM instance parameter file and ASM password file for each instance. A total of 12 volumes are created in this group to account for a total of four VMs in this solution.  
• Four database volume groups (orabp22-vm1-db1, orabp22-vm1-db2, orabp22-vm1-db3 and orabp22-vm1-db4). Each database volume group consists of DATA volumes, REDO volumes, and FRA volumes.
• TEMP volume group (orabp22-vm1-temp) has a total of four volumes. Each database instance is mapped with one TEMP volume.

Note: To show the application project for each volume, we used the application project code name ‘orabp22’ as the prefix for the volume group names which denotes Oracle Best practices in the year 2022.

The baseline configuration determined the initial workload and allowed to properly evaluate and compare best practice results.  

Server settings in BIOS

The baseline BIOS settings for the two PowerEdge R750xs servers are described in the following table.

Table 7: PowerEdge Server R750 BIOS settings 

To check or change these BIOS settings, login to iDRAC and perform the following steps:

Select Configuration > Process Setting

Select Configuration > Memory Setting

Select Configuration > System Security

Select Configuration > Process Setting

Select Configuration > System Profile Settings

ESXi system configuration

Use the following steps to configure the physical servers that host the database VMs:

1. Use the BIOS settings for the PowerEdge servers described in 0.
2. Create a single virtual disk in a RAID 1 configuration with two local disks (SSDs or HDDs) per server to install the bare-metal operating system (ESXi 7.3).
3. Install ESXi 7.3 by using the Dell customized ISO image.
4. Configure, monitor, and maintain the ESXi host, virtual networking, and the VM by using the VMware vSphere web client and VCSA, which is deployed as a VM on the management server.
5. Zone two dual-port 32 Gb/s HBAs (four initiators in total) and configure these HBAs with the PowerStore front-end Fibre Channel ports for high bandwidth, load-balanced, and highly available SAN traffic.

PowerStore storage configuration details

The PowerStore volume group feature enables the administrator to create a configuration that facilitates ease of management, efficient snapshots, and replication. This section explains the design principles used to create volumes. For the baseline storage configuration, all four virtualized databases (DB1 to DB4) used the same design. The following table shows the baseline storage configuration for DB1. The database used was named ORABP22:

Table 8: PowerStore storage volumes

The naming convention using VMn and DBn allowed the database administrators to increment the value to create volume groups for other databases. For example, the second copy of the database had a volume group name of ORABP22_VM2_DB2.  Using this naming convention allowed the administration team to repurpose copies of the database quickly. Each LUN was configured as a VMware datastore. For detailed PowerStore best practices and features, see Storage Best Practices.

SAN FC network zoning configuration

Each ESXi host (R750xs server) has two dual-port Fibre Channel HBAs.  The PowerStore 5000T under test has two Fibre Channel front-end I/O modules (one per node).  Each I/O module has four 32Gbps ports with a total of eight 32Gbps front-end ports.  There are two Connectrix Fibre Channel switches. Each Connectrix switch is its own fabric.  For connectivity, each ESXi host was zoned to all eight front-end ports of the PowerStore across the two switches. This configuration should be the default setup. This configuration ensures high availability against a single component failure of the stack.  Figure 2 shows the SAN zoning configuration.

Figure 2: Baseline SAN FC connectivity and zoning

VM network: Distributed Network switches

We configured the following two distributed network switches: ESXI-Mgmt DS ESXi server management that connects to the vCenter and ESXi hosts and OraPub-vMotion that supports the VM network for Oracle database public network and vMotion.

Table 9 describes the details of these two distributed network switches.

Table 9: Distributed network switches

Table 10: VMware Network settings

Server physical NIC port, HBA port, and switch connectivity

Table 11 shows the physical NIC ports, their associated switches, and port groups. The last two columns, SAN FC A and SAN FC B, are the two dual port HBAs that are connected to the Fibre channel switches. The last row describes the switch ports that are connected the corresponding NICs ports or the HBA ports.

Table 11: Switches ports, NIC ports and HBA ports in Ethernet network and SAN FC network

Figure 3 describes the configuration of the physical and virtual network and the SAN Fibre Channel Network. In the middle of the figure, two virtually distributed switches established the mapping of the virtual network port groups and physical uplinks:

Distributed switch vDS-01-ESXi-Mgmt connects the port group ESXi-Mgmt-3780dPG and the two physical uplinks vmnic0 and vmnic1.

Distributed switch vDS-02-Orapub-Vmotion connects two port groups: pub-25GbE-vlan382-dPG (for VM public network) and vMotion-25GBE-vlan99-dPG (for Motion) with two physical uplinks vmic2 and vmnic4. 

The right side of the figure shows how these physical links are mapped to the physical NIC ports of each server and how these NICs ports are connected to the 25GBE TOR physical switches that connect to outside network.

The lower right side of the figure shows how the two dual ports. The HBAs are connected to the ports of two FC switches.

Figure 3: Physical and virtual network design

ESXi system configuration

Use the following steps to configure the physical servers that host the database VMs:

1. Use the BIOS settings for the PowerEdge servers described in 0.
2. Create a single virtual disk in a RAID 1 configuration with two local disks (SSDs or HDDs) per server to install the bare-metal operating system (ESXi 7.3).
3. Install ESXi 7.3 by using the Dell customized ISO image.
4. Configure, monitor, and maintain the ESXi host, virtual networking, and the VM by using the VMware vSphere web client and VCSA, which is deployed as a VM on the management server.
5. Zone two dual-port 32 Gb/s HBAs (four initiators in total) and configure these HBAs with the PowerStore front-end Fibre Channel ports for high bandwidth, load-balanced, and highly available SAN traffic.

PowerStore storage configuration details

The PowerStore volume group feature enables the administrator to create a configuration that facilitates ease of management, efficient snapshots, and replication. This section explains the design principles used to create volumes. For the baseline storage configuration, all four virtualized databases (DB1 to DB4) used the same design. The following table shows the baseline storage configuration for DB1. The database used was named ORABP22:

Table 8: PowerStore storage volumes

The naming convention using VMn and DBn allowed the database administrators to increment the value to create volume groups for other databases. For example, the second copy of the database had a volume group name of ORABP22_VM2_DB2.  Using this naming convention allowed the administration team to repurpose copies of the database quickly. Each LUN was configured as a VMware datastore. For detailed PowerStore best practices and features, see Storage Best Practices.

SAN FC network zoning configuration

Each ESXi host (R750xs server) has two dual-port Fibre Channel HBAs.  The PowerStore 5000T under test has two Fibre Channel front-end I/O modules (one per node).  Each I/O module has four 32Gbps ports with a total of eight 32Gbps front-end ports.  There are two Connectrix Fibre Channel switches. Each Connectrix switch is its own fabric.  For connectivity, each ESXi host was zoned to all eight front-end ports of the PowerStore across the two switches. This configuration should be the default setup. This configuration ensures high availability against a single component failure of the stack.  Figure 2 shows the SAN zoning configuration.

Figure 2: Baseline SAN FC connectivity and zoning

VM network: Distributed Network switches

We configured the following two distributed network switches: ESXI-Mgmt DS ESXi server management that connects to the vCenter and ESXi hosts and OraPub-vMotion that supports the VM network for Oracle database public network and vMotion.

Table 9 describes the details of these two distributed network switches.

Table 9: Distributed network switches

Table 10: VMware Network settings

Server physical NIC port, HBA port, and switch connectivity

Table 11 shows the physical NIC ports, their associated switches, and port groups. The last two columns, SAN FC A and SAN FC B, are the two dual port HBAs that are connected to the Fibre channel switches. The last row describes the switch ports that are connected the corresponding NICs ports or the HBA ports.

Table 11: Switches ports, NIC ports and HBA ports in Ethernet network and SAN FC network

Figure 3 describes the configuration of the physical and virtual network and the SAN Fibre Channel Network. In the middle of the figure, two virtually distributed switches established the mapping of the virtual network port groups and physical uplinks:

Distributed switch vDS-01-ESXi-Mgmt connects the port group ESXi-Mgmt-3780dPG and the two physical uplinks vmnic0 and vmnic1.

Distributed switch vDS-02-Orapub-Vmotion connects two port groups: pub-25GbE-vlan382-dPG (for VM public network) and vMotion-25GBE-vlan99-dPG (for Motion) with two physical uplinks vmic2 and vmnic4. 

The right side of the figure shows how these physical links are mapped to the physical NIC ports of each server and how these NICs ports are connected to the 25GBE TOR physical switches that connect to outside network.

The lower right side of the figure shows how the two dual ports. The HBAs are connected to the ports of two FC switches.

Figure 3: Physical and virtual network design

CPU reservation

We ran four virtualized databases in parallel to achieve a meaningful test result for a single R750xs server. After several tests using HammerDB and an OLTP workload, a baseline configuration was developed. The following table shows the vCPU and memory allocation for each virtualized database.

Table 12: Virtual machine configuration

A VMware reservation is a guaranteed allocation of CPU or memory for the virtual machine. In the case of the vCPU, each virtual machine received a 16-core reservation. With four total virtualized databases, the total vCPU reservation was 64-cores (four virtual machines each with sixteen cores) on the server. The PowerEdge R750xs servers are two socket servers and in the configuration for our tests, two Intel Xeon Gold 6338 CPUs were used. Each Intel CPU has 32 cores so there was a total of 64 physical cores.

While the virtual machines used 64 physical cores by default, the PowerEdge R750xs servers enable logical processors. When logical processors are enabled, the hypervisor presents twice as many processor cores. In our configuration, the 64 physical processor cores are presented as 128 cores (64 x 2). Logical processors can boost performance by enabling more executions in parallel. In the case of our four virtual databases, we used 64 of the available logical processor cores from the total available of 128 logical processor cores.

Memory reservation

The memory configuration per virtual machine included a reservation of 128 GB. With four virtual machines, a total of 512GB of memory was reserved. The total memory available in the server was 1TB, meaning the virtual machines used half of the available physical memory in the server. Allocating memory to each virtual machine ensured that each VM fit into a physical NUMA (Non-uniform memory access) node. NUMA refers to a multiprocessor configuration in which each physical CPU has a dedicated memory bank. Information can be retrieved faster from the local memory bank than having to pull the data from another memory bank. When a virtual machine’s memory spans two memory banks, wait times can incur and impact performance.

To prevent sub-optimal NUMA design, each virtual machine has been allocated 128GB of memory, enabling the use of only the local memory bank. The VMware ESXi hypervisor is responsible for the placement of memory on a processor and has been designed to optimize performance. By reserving memory that enables the virtual machine to use a local memory bank, performance is optimized for all the virtual machines on the server.

We created VMFS data stores using the storage volumes that are presented to the database server ESXi hosts. These datastores host VM operating systems and other database volumes. The following table represents the datastore configuration of a virtualized database DB1:

Table 13: Data store configuration

Similar design principle is used for other virtualized databases DB2, DB3, and DB4. 

Use the following steps to set up and configure your virtual machine.

1. Create VM from a Red Hat Enterprise Linux 8.5 ISO image to orabp22-vm1-os-ds data store.

 Figure 4: Create VM with Red Hat Enterprise Linux 8.5 ISO image

2. Select Minimal Installation as base environment. 

Figure 5: Software selection display page

3. Configure public network from the Distributed switch.

Figure 6: Select the Distributed switch for Public Network 

4. Add three additional SCSI drivers. 
   1. From VCenter, select the VM and click Edit Settings > Add Devices > SCSI controller.
   2. Add three more SCSI controllers to SCSI controller 1 through 3.

Figure 7: Select SCSI controllers 

5. Add hard disks. 
   1. From vCenter, select the VM and click Edit Settings > Add Devices > Hard disks.
   2. Add all the VM hard disks on the list.

Figure 8: Add hard disks 

1. For each virtual disk, create a single partition that spans the entire disk and has a starting offset of 2,048 sectors. This ensures that the disk is correctly aligned, such that the start of each partition that is created on volume is aligned with the start of a physical sector of a disk or logical block of a LUN. Creating a partition on a disk also marks the disk that is used and can prevent the disk from being used for other purposes. 

parted -s /dev/sdh mklabel msdos

parted -s /dev/sdh mkpart primary 2048s 100%

2. Use the UDEV rule to assign Linux 0660 permission and ownership of these disks to the grid user, that is, the owner of the Oracle GI and Oracle ASM instance. In the following code sample, the UDEV rule is set for one of the disks as an entry of the custom UDEV rule script in /etc/udev/rules.d/95oracleasm.rules:

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:= "nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="368ccf09800cd6a0eaf106e5a2155cf92", SYMLINK+="oraudev/disks/ora-data1",  OWNER="grid", GROUP="asmadmin", MODE="0660"

The string “368ccf09800cd6a0eaf106e5a2155cf92” is the SCSI ID of the device. 

SCSI ID of devices can be found using the following command:

/usr/lib/udev/scsi_id -g -u -d /dev/sdX

Note: Oracle also provides ASMLIB and ASM Filter driver, which supersedes ASMLIB to configure disks. This driver is required to install the additional oraclesm kernel driver, oracleasm-support package, and oracleasmlib library. 

3. Run the Linux command # udevadm trigger and set the ownership of the device /dev/sdh1 as the following:

grid:asmadmin with 0660 permission and also create a soft link alias /dev/oraudev/disks/ora-data1 that points to /dev/sdh1.

You can use this soft link alias as the path of this device to create an Oracle ASM disk.

4. Repeat step 2 and step 3 to create UDEV rules for all the database disks in the custom UDEV rule script.

 /etc/udev/rules.d/95oracleasm.rules. 

For example, creating these rules creates the device links: 

/dev/oraudev/disks/ora-XXX as the following list.  

Prepare the local yum repository

Oracle installation requires additional RPMs due to the minimal operating system base installation. A local yum repository enables the automatic installation of this additional RPM package installation. The following steps describe how to configure the local yum repository using the Red Hat Enterprise Linux 8.5 ISO image.

1. Mount the Red Hat Enterprise Linux ISO image to the local file system.

mount -o loop RHEL-8.5.0-20211013.2-x86_64-dvd1.iso /opt

2. Add /etc/yum.repos.d/rhel8.repo with repository entries.

[InstallMedia]

name=Red Hat Enterprise Linux 8.5.0

mediaid=None

metadata_expire=-1

gpgcheck=1

enabled=1

baseurl=file:///opt/BaseOS/

#baseurl=file:///opt/

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

3. Clean Yum / DNF and Subscription Manager Cache.

#dnf clean all

4. Verify that the Yum / DNF is getting packages from the local repository.

# dnf repolist

Load the Oracle 19c preinstall package with the following yum command:

 # yum install oracle-database-preinstall-19c-1.0-2.el8.x86_64.rpm

This Oracle 19c preinstall package not only loads the required Linux packages, but also modifies the operating system configuration. These modifications include soft and hard limits in /etc/security/limits.conf and Linux operating system kernel settings that are required for Oracle database installation.

Oracle grid infrastructure installation 

Use the following steps to install Oracle Grid Infrastructure and ASM on each VM operating system.

1. Extract the LINUX.X64_193000_grid_home.zip  $unzip LINUX.X64_193000_grid_home.zip
2. Set the operating system environment variable $CV_ASSUME_DISTID=OL7 and install the Oracle GI 19c base version to the Oracle GI home by running:

                 $gridSetup.sh.

3. Specify ’/dev/oraudev/disks/*’ as the ASM Disk Discovery Path and select disks ’/dev/oraudev/disks/ora-ocr1’, ’/dev/oraudev/disks/ora-ocr2’, and ’/dev/oraudev/disks/ora-ocr3’ for the DISK group OCR with the Normal Redundancy setting.
4. The root user should perform the following command to complete the grid infrastructure installation: 

/u01/app/19.0.0/grid/root.sh 

Oracle grid infrastructure upgrade

Use the following steps upgrade Oracle grid infrastructure and ASM to 19cR15.

1. Apply the latest version of patch 6880880 to the Oracle grid infrastructure home to upgrade the Opatch utility from 12_2.0.1.17 to 12.2.0.1.30.
2. Upgrade the Oracle GI home from 19c base version to 19cRU15 by applying 33803476. To apply this patch, run the following opatchauto command as the Linux root user:

[root@dbhost1 ~]# opatchauto apply /home/grid/patches/33803476 -oh $ORACLE_HOME

3. See update instructions in Oracle Support Document 19202204.9 and Patch 33803476 - GI Release Update 19.15.0.0.220419 readme file for details. 

Oracle database 19.3.0.0 Installation

Perform the following command to install Oracle database 19.3.0.0 on Red Hat Enterprise Linux operating system 8.5 in virtual machine:

oracle@dbhost1 dbhome_1]$ export CV_ASSUME_DISTID=OL7

[oracle@dbhost1 dbhome_1]$ ./runInstaller & 

 Select software only.

Figure 9: Oracle Database 19c installation

Use the installer UI steps to complete the installation.

Use the following steps to upgrade Oracle Database to 19.15.0.0:

1. Apply patch 6880880 to the Oracle database home to upgrade the Opatch utility from 12_2.0.1.17 to 12.2.0.1.30
2. Upgrade the Oracle database home from 19c base version to 19cRU15 by applying patch 33806152. Follow the update instructions in patch 33806152 readme. 
   The following example is the main patch apply step:

[oracle@obpvm4 patches]$cd /home/oracle/patches/33806152

[oracle@obpvm4 33806152]$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./ 

          [oracle@obpvm4 33806152]$$  opatch apply

ASM disk groups creation

Before creating a test database, we created four disk groups for DATA, REDO, FRA, and TEMP for the datafiles, REDO logs files, FRA/archive logs files and Temp files for the HammerDB test database. 

Table 13 shows the volume mapping of ASM disk groups: DATA, REDO, FRA, and TEMP. For the baseline configuration, we included one volume for each of the DATA, REDO, and FRA disk groups. 

Table 14: ASM disks and disk groups

To create these ASM disk groups, log in as the grid user and run the following asmca command: 
$ asmca

Figure 10: ASM disk groups layout

Select the proper DISK Path for each disk group, as shown in Figure 11. After creating these disk groups, all disk groups should appear as shown in the following display:

Figure 11: Available ASM disks

We used HammerDB to generate an OLTP workload that simulates enterprise applications. We generated a significant load on the Oracle infrastructure to determine the performance difference between baseline configuration and best practices configuration. 

Create a HammerDB test database

We began the test database environment configuration by creating a test database. We performed the dbca command as the Oracle user to create an Oracle database using the following disk groups:

• DATA disk group for database datafile and control files  
• REDO disk group for online redo logs  
• TEMP disk group for TEMP file 
• FRA for Flash Recovery Area

Figure 12: Creating a test database with dbca 

Test database default settings

We used the following main performance related database settings or database limit settings for this HammerDB test in the baseline configuration. 

Table 15: Test database default configuration

In addition to these database parameters, we maintained the ASM disk group parameter AU Size to the default value of 4 MB in the baseline configuration.

The Hammer DB workload configuration is shown in the following table:

Table 16: HammerDB TPCC workload configuration

In reviewing the configuration parameters, the overall profile can be labeled as an entry-level database. An entry-level database configuration provides a foundation to build upon best practices. Over the course of validating best practices the database configuration will change to optimize performance. This approach of starting with a minimal configuration and adding best practices has the benefit of enabling customers to address database growth challenges using these validated recommendations.

The following list highlights default settings that we recommend maintaining for optimal storage performance:

• For mission critical Oracle databases, we recommend separating the following database components into different ASM disk groups: 
  • GRID disk group for Oracle grid infrastructure and Voting disks (if RAC is configured).
  • DATA disk group for database files
  • REDO disk group for online Redo logs
  • TEMP disk group for Temp and FRA disk group for flashback logs and archive logs
• Follow Oracle ASM best practices to create ASM disk groups for GRID, DATA, REDO, FRA, and TEMP.
• Use multiple I/O controllers in VM hard disk configuration to distribute storage volumes and VM hard disks to multiple I/O controllers for I/O load balance.

As part of the storage layer tuning, we have tested the following parameters:

1. PowerStore Multiple Volumes 
2. PowerStore Performance Policy

Multiple volume ASM Disk Groups

Storage configuration can directly influence database performance. This best practice validates that adding multiple storage LUNs for ASM disk groups optimizes database performance. 

Overview

To determine if a best practice provides value, it had to be tested against a baseline configuration of the database to validate performance.  In the baseline configuration for database volumes, the DATA, REDO, and FRA disk group each consists of one LUN. Table 16 shows the baseline volume groups and LUNs configuration. 

The baseline storage configuration followed the principle of separating database files into dedicated volume groups. For example, the Data files, Online Redo logs, and FRA files are in the same volume group, but separated from the Grid volume group and the Temp volume group. This allows the DBA to only snapshot the database without including the Grid and Temp files which are not necessary for a functional database snapshot.  This also provides space savings if replication is used.  

Table 17: Baseline volume group configuration

Table 18 shows the optimized PowerStore storage configuration. The volume groups remain intact, but additional LUNs were added for the DATA, REDO, and FRA disk groups. Using multiple LUNs for Oracle ASM disk groups improves database performance. The primary benefit of using multiple LUNs is that the operating system creates an I/O queue path per LUN. Since Oracle ASM spreads data evenly across all disks in a disk group, this would reduce any I/O bottleneck and provide higher I/O concurrency providing performance optimization and utilization. The following table shows the optimized PowerStore storage configuration for the Oracle database. In the optimized storage configuration, DATA, REDO, and FRA disk groups each has four LUNs. Compared to the baseline configuration, there are four times the number of I/O queue paths. 

Table 18: Optimized PowerStore storage configuration

Once the additional LUNs are created, these LUNS can be added to the ASM disk groups. The size of the LUNs for each ASM disk group should be determined by the size of the database.  The PowerStore storage array has thin provisioning feature, meaning LUNs should be created larger than required.  However, to truly benefit using thin-provisioned LUNs on the PowerStore, the use of Oracle “autoextend” clause when creating tablespaces is required.  For detailed PowerStore best practices and features, see Storage Best Practices. 

Recommendation

Our results indicated that having multiple LUNs for the ASM disk groups increased performance significantly for the following metrics:

• New Order per Minute (NOPM)
• Transactions per Minute (TPM)
• PowerStore IOPS
• Server CPU utilization

Based on these results, it is considered as a Day 1, Highly Recommended best practice.

Implementation steps

Before adding additional three LUNS/volumes to each ASM disk group, we performed the following tasks to present the additional volumes to the guest VM RHEL OS where the Oracle database resides.

• Present these volumes to the ESXi host and created the corresponded datastores based on these volumes
• Created the hard disks from these data stores and presented these hard disks to the guest VM Red Hat Enterprise Linux OS as VMDKs. 
• Use these devices as a whole disk (/dev/sd?)
• Created udev rules to assign Linux 0660 permission and ownership of these devices to the grid user and create alias name such as ‘/dev/oraudev/disks/ora-data2’ for each device. 

After performing these tasks, the following table shows the device names, their corresponding volume groups, and the ASM disk groups. 

Table 19: Volume groups, data stores, UDEV devices and ASM disk groups

For the baseline configuration, each ASM disk group of DATA, REDO, and FRA only has a single volume. For example, DATA disk group has one volume: ‘/dev/oraudev/disks/ora-data1’. To add the additional three volumes to the ASM disk groups, DATA, REDO, and ARCH, ensure that all four LUNs in the same disk group are of the same size.  Perform the following SQL statement in the ASM instance to add the three ASM disk groups, DATA, REDO, and FRA:

SQL> ALTER DISKGROUP data ADD DISK

'/dev/oraudev/disks/ora-data2' NAME DATA_0001,

'/dev/oraudev/disks/ora-data3' NAME DATA_0002,

'/dev/oraudev/disks/ora-data4' NAME DATA_0003;

SQL> ALTER DISKGROUP redo ADD DISK

'/dev/oraudev/disks/ora-redo2' NAME REDO_0001,

'/dev/oraudev/disks/ora-redo3' NAME REDO_0002,

'/dev/oraudev/disks/ora-redo4' NAME REDO_0003;

SQL> ALTER DISKGROUP fra ADD DISK

'/dev/oraudev/disks/ora-fra2' NAME FRA_0001,

'/dev/oraudev/disks/ora-fra3' NAME FRA_0002,

'/dev/oraudev/disks/ora-fra4' NAME FRA_0003;

It is recommended to set up the ASM disk groups with multiple LUNs during the initial configuration but if this is an existing database, you can run these SQL statements to add volume to ASM disk groups while the database is still running. It is preferable to run these statements outside of the database workloads’ peak time, as adding disk volume operations results in data rebalancing and may cause a significant performance overhead. The rebalancing power is defined by the ASM_POWER_LIMIT initialization parameter (default value = 1, range to 1024). The higher the limit, the faster a rebalance operation may complete, and the rebalance results in more performance overhead.

To monitor the status of the rebalancing operation, run this SQL statement to check the operation status:

SQL> select INST_ID, OPERATION, STATE, POWER, SOFAR, EST_WORK, EST_RATE, EST_MINUTES from GV$ASM OPERATION;

   INST_ID OPERA STAT      POWER      SOFAR   EST_WORK   EST_RATE EST_MINUTES

---------- ----- ---- ---------- ---------- ---------- ---------- -----------

         1 REBAL WAIT          1          0          0          0           0

         1 REBAL RUN           1      14185     179546       7085          23

         1 REBAL DONE          1          0          0          0           0

In this example, the output shows three rebalancing operations: the first is done (bottom), the second is running (middle), and the third is waiting (top). For the second row, the report shows:

• EST_WORKD=179,546 - estimated number of Allocation Units that must be moved
• SOFAR = 14,185 - number of Allocation Units that have been moved so far
• EST_RATE=7,085 - number of Allocation Units that are moved per minute
• EST_MINUTES= 23 – remaining estimated amount of time (23 minutes) =

PowerStore: Volume Performance Policy

Volume performance can be managed in PowerStore by adjusting the performance policy to high, medium, or low. Selecting a performance policy sets a relative priority for the volume by using a share-based system that prioritizes the higher performance policy when resources are constrained. The performance policy does not impact on system behavior unless some volumes have a low performance policy setting while other volumes are set to medium or high.

This section discusses the Volume Performance Policy of the PowerStore. This best practice validates that changing the volume performance policy for the database volumes from the default (medium) to “High” may optimize database performance in certain configurations.

Overview

To determine if changing the volume performance policy provides value, it had to be tested against the baseline using the default value (medium). In the baseline configuration, the database volumes for the Oracle DATA, REDO, and FRA ASM disk groups were configured with the “medium” performance policy.

Table 19 below shows which volumes were configured with the performance policy of “High”.

Table 20: Volume Performance Policy Settings

Recommendation

There was no change in performance based on the test results after changing the volume performance policy to High for the Oracle REDO disk group.  Although, changing the volume performance policy for the REDO disk group yielded no gain in performance, it is still recommended as this could potentially help improve database performance when the PowerStore storage array is heavily used due to workload consolidation.  Therefore, this is considered as a Day 3, Fine Tuning best practice.  

Implementation steps

To change the volume performance policy for the volumes, use the PowerStore Manager and follow the instructions from PowerStore Manager Overview.

The following figure shows the performance policy for the volumes of VM1.

Figure 13: Volume Performance Policy settings

The “tuned-adm” command line tool enables switching between different tuning profiles in Linux. In this best practice, we used the “tuned-adm” tool to switch to an optimized Linux profile for Oracle. 

Overview

The Linux operating system provides a system tuning tool called “tuned”.  It is a profile-based tool that uses udev device manager.  The “tuned-adm” tool enables the Linux administrator to switch between profiles for the operating system. Pre-defined profiles enable the application of Linux configuration settings in a package that can be downloaded. In this best practice, the Red Hat operating system tuned profile TUNED-PROFILES-ORACLE was downloaded and installed in the Linux OS.  As the result of installing this TUNED-PROFILES-ORACLE profile, the following settings were added to the system:

• Transparent Hugepages was set to “never” (disabled)

Transparent HugePages (THP) is a Linux memory management that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on machines with large amount of memory by using larger memory pages. Oracle databases do not use THP.

• VM.SWAPPINESS was set to 1

Swap space is used by the Linux kernel as virtual memory. The swappiness setting defines how aggressively the kernel will swap memory pages to disk.  By changing VM.SWAPPINESS to 1, the kernel is instructed to swap as little as possible to disk and keep more memory pages in RAM. 

• VM.DIRTY_BACKGROUND_RATIO was set to 3

The DIRTY_BACKGROUND_RATIO is the percentage of memory that contains free pages and reclaimable pages, that can be filled with dirty pages before the pages are written to disk. Databases can aggressively use the kernel page cache thus lowering the background ratio forces the kernel to flush the dirty pages to disk more frequently. 

• VM.DIRTY_RATIO set to 80

The DIRTY_RATIO is the percentage of total system memory that contain dirty pages before the pages are written to disk. By setting dirty_ratio to 80 the kernel is instructed to use 80 percent of total system memory for dirty pages before having to flush the data to disk. 

• VM.DIRTY_EXPIRE_CENTISECS is set to 500

The DIRTY_EXPIRE_CENTISECS define in centisecs, one hundredth of a second, when data is old enough to be written to disk. By setting DIRTY_EXPIRE_CENTISECS to 500 a dirty page can stay in memory for 500 centisecs before it is written to disk. 

• VM.DIRTY_WRITEBACK_CENTISECS is set to 100

The DIRTY_WRITEBACK_CENTISECS parameter defines the frequency the kernel will check to write dirty pages to disk. In this case, a value of 100 centisecs instructs the kernel to check every 100 centisecs to write dirty pages to disk.

VM.TCP_RMEM_MAX is set to 4194304

The VM.TCP_RMEM_MAX parameter defines the maximum amount of receive socket memory. 

In addition to the VM memory settings, this best practice also included network and kernel settings updated by the MSSQL tuned profile. The following list describes the network and kernel settings that were updated by the MSSQL tuned profile: 

• Net.core.rmem_default = 262144
• Net.core.rmem_max = 4194304
• Net.core.wmem_default = 262144
• Net.core.wmem_max = 1048576
• Kernel.numa_balancing = 0

Implementation steps

In this test case, we are going to install Red Hat tuned-profiles-oracle profile which is recommended and designed to run Oracle Database. To install tuned-profiles-oracle profile , use yum install command:

# yum install tuned-profiles-oracle

View available profiles by running the following command:

# tuned-adm list

Enable the profile

# tuned-adm profile oracle

Verify the active profile

# tuned-adm active

You should see an output like

Current active profile: oracle

Additional resources

Monitoring and Managing System Status and Performance

In this best practice we used Oracle scripts to implement HugePages in the Linux operating system. 

Overview

The Linux operating system default memory size is 4 KB. While this size works well for many applications, the Oracle Shared Global Area (SGA) memory pool can be very large. The memory size of the SGA is large because it supports other components, like the database buffer cache, shared pool, large pool, and many other memory pools. Thus, management of the SGA using 4 KB memory pages can cause excessive processing and wait times to access memory pages. 

HugePages is a feature integrated into the Linux kernel 2.6.  Enabling HugePages makes it possible for the operating system to support memory pages greater than the default.  Using very large page sizes can improve system performance by reducing the amount of system resources required to access page table entries.  For Oracle databases, using HugePages reduces the operating system maintenance of page states, and increases Translation Lookaside Buffer (TLB) hit ratio.  Oracle recommends using HugePages on Linux platform installations to obtain the best performance for Oracle Databases.

Implementation steps

1. To enable the HugePages, complete the following steps:
2. Run the following command to determine if the kernel supports HugePages:

# grep Huge /proc/meminfo

AnonHugePages:         0 kB

ShmemHugePages:        0 kB

HugePages_Total:       0

HugePages_Free:        0

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

Hugetlb:               0 kB

3. Edit memlock setting in /etc/security/limits.conf. The memlock setting is specified in KB, and the maximum locked memory limit should be set to at least 90 percent of the current RAM when HugePages memory is enabled. For example, if the server RAM size is 128GB, the memlock can be set 128GB * 90% = 120,795,955 KB and add the following lines into this file:

*   soft   memlock    120,795,955

*   hard   memlock   120,795,955

4. Log in as oracle user again and run the ulimit -l command to verify the new memlock setting:

   $ ulimit -l

     120795955

5. As root user, create a script “hugepages_settings.sh” provided by Oracle that computes recommended values for HugePages configuration for the current shared memory segments:

  !/bin/bash

#

# hugepages_settings.sh

#

# Linux bash script to compute values for the

# recommended HugePages/HugeTLB configuration

#

# Note: This script does calculation for all shared memory

# segments available when the script is run, no matter it

# is an Oracle RDBMS shared memory segment or not.

# Check for the kernel version

KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

# Find out the HugePage size

HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`

# Start from 1 pages to be on the safe side and guarantee 1 free HugePage

NUM_PG=1

# Cumulative number of pages required to handle the running shared memory segments

for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`

do

   MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`

   if [ $MIN_PG -gt 0 ]; then

      NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`

   fi

done

# Finish with results

case $KERN in

   '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;

          echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;

   '2.6'|'3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;

    *) echo "Unrecognized kernel version $KERN. Exiting." ;;

esac

# End

6. Provide the permission of this script:

# chmod +x hugepages_settings.sh

And execute this script to get the values for hugepages configuration:

# ./hugepages_settings.sh

7. Set the following kernel parameter in the /etc/sysctl.conf file, where value is the HugePages value determined in step 5:

vm.nr_hugepages=value

8. Load the parameter:

# sysctl -p

9. Check the available HugePages.

# grep Huge /proc/meminfo

Additional resources

Oracle® Database Administrator's Reference: 19c for Linux and UNIX System-Based Operating Systems 

In this best practice, we changed the default Linux disk I/O scheduler per Red Hat guidelines. 

Overview

There are several choices for the type of disk scheduler used by the operating system for Red Hat Enterprise Linux 8. The function of the disk scheduler is for ordering, delaying, or merging I/O requests to storage to achieve better throughput and latency. Each disk scheduler provides a different method for managing storage requests:

• None: Implements a first-in first-out (FIFO) scheduling algorithm. The none scheduler is recommended for systems that have high performance storage like Solid State Drives (SSD) or Non-volatile Memory Express (NVMe) drives. The default schedule for NVME is none. 
• Mq-deadline: This scheduler groups queued I/O requests into batches. The I/O in the batches are organized by logical block addressing so that writes are organized by location to storage. The mq-deadline is suited for traditional HDD storage. 
• Bfq: This scheduler prioritizes latency rather than maximum throughput. The bfq disk scheduler is recommended for desktop or interactive tasks and traditional HDD storage. 
• Kyber: This scheduler tunes itself by analyzing I/O requests. With each I/O request a calculation is done to determine of the I/O can be satisfied with the least amount of latency. Kyber is recommended for high performance storage like SSDs and NVMe drives. 

In this best practice, we followed Red Hat’s recommendation to set the disk I/O scheduler to none, as the PowerStore contains NVMe drives and the database access the storage volumes as virtual devices through virtual guest operating system and through host bus adapters (HBAs).

Implementation steps

Use the following steps to set disk I/O scheduler using tuned:

1. Find the system unique identifier of the selected block device. Substitute the word “device” in the command below with the device name (ex. sdc).

 $ udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)'

2. Use a device name such as sdc to in “–name = /dev/sdc”
3. Create the /etc/tuned/my-profile/tuned.conf configuration file.

[main]

 Include= existing-profile

 [disk]

   devices_udev_regex=ID_WWN=device system unique id

elevator=selected-scheduler

4. Replace device system unique id with the value of the chosen identifier (for example,

       0x6000c2982a4e742c for /dev/sdc).

5. To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars.  For example, if we have ID_WWN =0x6000c297fc627d0f for /dev/sdd and ID_WWN= 0x6000c2982a4e742c for /dev/sdf, the UDEV entry should look like this:

devices_udev_regex=(ID_WWN=0x6000c297fc627d0f)|(ID_WWN0x6000c2982a4e742c)
elevator=none

6. Enable your profile.

# tuned-adm profile my-profile

7. Verify that the Tuned profile is active and applied:

$ tuned-adm active

$ tuned-adm verify

Additional resources

Red Hat Enterprise Linux 8: Monitoring and managing system status and performance 

In this best practice, we implemented Network Time Protocol (NTP). The NTP synchronizes the clocks on the servers to ensure they are consistent across all databases. 

Overview

NTP is used to synchronize clocks between computer systems using the network. By implementing NTP our Oracle database, the servers will be synchronized. Time synchronization is important for analyzing the systems, scheduling jobs, and implementing Oracle Real Application Clusters. This is a configuration best practice as the goal is to have all our database servers synchronized from the same NTP server. 

Implementation steps

In Red Hat Enterprise Linux 8, the NTP protocol is implemented by the chrony daemon through the chrony package.

The steps to configure the daemon:

1. Install the chrony package if it is not already installed: 

# yum  install chrony -y

2. Edit /etc/chrony.conf file by adding a NTP server entry. 

For example, adding NTP server 10.230.82.70 with this entry: “server 10.230.82.70 iburst”

3. Start chronyd:  

# systemctl start chronyd

4. Stop chronyd:  

# systemctl stop chronyd

5. To ensure chronyd starts automatically at system start, 

# systemctl enable chronyd

6. To check if chrony is synchronized: 

# chronyc tracking

7. To check chrony source: 

# chronyc sources

8. To check chrony source statistics: 

# chronyc sourcestats

Additional resources

Configuring Basic System Settings, Red Hat Enterprise Linux 8

In this best practice, we increased the size of the Oracle redo logs to minimize log switching frequency to improve database performance. Proper sizing of the database redo logs can reduce wait events in the database providing an optimal database system. However, the redo logs sizing also depends on Recovery Time Objective (RTO) and Recovery Point Objective (RPO) based on the business SLA.

Overview

The most crucial structure for database recovery operations is the online redo log, which consists of two or more pre-allocated files that store all changes made to the database as they occur. Every instance of an Oracle Database has an associated redo log to protect the database in case of an instance failure.

A log file sync wait event is triggered when a user session issues a commit or a rollback.  It is the time it takes for the log writer (LGWR) to write to the redo log file and post the user session. The log file sync wait event for the Oracle database was used to validate the best practices.

In this case the original size of the log files was 200MB and they were replaced with log files of 8 GB in size.

Note: Not every workload will show significant performance increase when increasing the size of the redo logs. Customers should evaluate the AWR to determine if the redo logs have an associated high number and duration of wait times. If the redo logs show a possible bottleneck, increasing the size of the redo logs could improve performance.

Implementation steps

We recreated five new redo log groups, each 8 GB in size, and dropped the three original smallest size of redo log groups by performing the following SQL commands with system or sys privilege:

SQL> ALTER DATABASE ADD LOGFILE GROUP 4('+REDO') SIZE 8G; 

SQL> ALTER DATABASE ADD LOGFILE GROUP 5('+REDO') SIZE 8G; 

SQL> ALTER DATABASE ADD LOGFILE GROUP 6('+REDO') SIZE 8G;  

SQL> ALTER DATABASE ADD LOGFILE GROUP 7('+REDO') SIZE 8G; 

SQL> ALTER DATABASE ADD LOGFILE GROUP 8('+REDO') SIZE 8G;  

SQL> ALTER DATABASE DROP LOGFILE GROUP 1,

SQL> ALTER DATABASE DROP LOGFILE GROUP 2;                              

SQL> ALTER DATABASE DROP LOGFILE GROUP 3 ;

In this best practice, we used an Oracle undocumented parameter _high_priority_processes to improve database performance. This parameter gices  the LGWR process a higher CPU priority.  We recommend that customers check with Oracle Support before using this parameter in production.  For more information, see the Oracle support document: Doc ID 1373500.1.

Overview

The Oracle hidden parameter “_high_priority_processes” enables the database administrator to assign priority to database processes. The following is the default value of the parameter:

_high_priority_processess = ’LMS*|LM1*|LM2*|LM3*|LM4*|LM5*|LM6*|LM7*|LM8*|LM9*’

1. LMSn and LMn - relates to the global cache service process.
2. VKTM – is the virtual keeper of time process and provides a wall clock time for measurements. 
3. LGWR – is the LGWR process responsible for redo log buffer management – writing the redo log buffer to a redo log file on disk. 

The LMSn and LMn process relates to the global cache service process. These processes receive, process, and send requests for the global cache service and the buffer cache resources. 

To optimize the database configuration, we configured the parameter to prioritize these processes:

_high_priority_processess = ’LMS*|VKTM|LGWR’

Changing the _high_priority_processess parameter requires a database instance restart. 

Implementation Steps

Perform the following script.

SQL> ALTER SYSTEM SET “_high_priority_processess”=’LMS*|VKTM|LGWR’ SCOPE=SPFILE;

Restart the database instance. 

Check the current value of this parameter, run this query as sys or system user.

SQL> select x.ksppinm name, x.ksppdesc description, y.ksppstvl value

from sys.x$ksppi x, sys.x$ksppcv y

where 1=1 and x.inst_id = y.inst_id and x.indx = y.indx

and x.ksppinm ='_high_priority_processes’;

Recommendation

Results from validating the best practices of the Linux operating system and Oracle database showed a significant performance improvement. Test findings showed increases in:

• New Order per Minute (NOPM)
• Transaction per Minute (TPM)
• PowerStore IOPS

Oracle metrics that showed improvement are:

• Lower “db file sequential read” average wait time
• Lower “log file sync” average wait time

Based on the results, these best practices are considered as Day 1, Highly Recommended.

This chapter focuses on server configuration best practices. A server configuration can be optimized by tuning the sub-components such as CPU, memory etc. Dell servers offers a wide range of sub- component parameters as part of the server BIOS which can be configured. 

 As part of Server layer optimization, we have tested multiple parameters that are available in server BIOS. The test cases and results are discussed in detail here.

By default, the Database Optimized Performance (DOP) profile is not enabled on the PowerEdge R750xs server. In this best practice we selected the DOP profile in BIOS and evaluated its impact on performance. 

Overview

A BIOS profile is an easy way to apply firmware values that optimize the system. Below is a list of BIOS settings that are automatically changed by selecting the DOP profile:

• Virtualization Technology (VT): Default is ENABLED, changed to DISABLED with the DOP profile. When VT is enabled, firmware settings are configured for enhanced support of hypervisors. Thus, with VT there is greater integration with virtualization that can optimize virtualization on the server. The DOP profile disables VT as many databases are not virtualized. As the Oracle databases have been virtualized in our best practice infrastructure, the Engineering team enabled virtualization technology after it had been disabled by the DOP profile. 
• System Profile: Default is Performance Per Watt (DAPC) and changed to Performance. DAPC is Dell’s proprietary implementation of dynamic processor management. Selecting the DOP profile changes the system profile from the default to Performance. Instead of the server balancing performance per watt to drive cost savings, the system is configured for maximum performance.  
• CPU Power Management: Default system DBPM and changed to Max Performance. CPU Power Management controls use of processor performance states. By default, CPU Power Management uses Demand-Based Power Management (DBPM), which enables processor operating frequency and voltage to be dynamically adjusted. The DOP profile changes CPU Power Management to a static state of Maximum Performance. 

Other system profile changes include:

• C1E: Default Enabled, changed to Disabled. Disables DIMM CKE power down and QPI power management. 
• C States: Default Enabled, changed to Disabled. Disables C-states and DIMM self-refresh. 
• Uncore Frequency: Default is Dynamic UFS, changed to Maximum. Setting to Maximum enables best performance of the cores and uncore during runtime. 
• Energy Efficient Policy: By default the balanced performance policy is used. This setting was changed to Performance, which instructs the processors to operate at the highest performance level. 
• CPU Interconnect Bus Link Power Management: Default Enabled, changed to Disabled. 
• PCI ASPM L1 Link Power Management: Default Enabled, changed to Disabled. 

Overall, the benefit of the Data Optimized Performance is setting several firmware settings to maximum performance. The only exception made in our validation tests was changing Virtualization Technology back to its default value of Enabled. However, for customers with physical (non-virtual) database deployments it is recommended to keep the DOP profile change for VT to Disabled.  

Recommendation

Setting the Database Optimization Performance profile in the BIOS resulted in a performance improvement in our testing with virtual machines for the following metrics: 

1. New Order per Minute (NOPM)
2. Transactions per Minute (TPM)
3. PowerStore IOPS
4. Server CPU utilization

This practice is categorized as a Day 1, Highly Recommended best practice.

Implementation steps

To set the Database Optimized Performance Workload Profile: 

1. Log into the iDRAC console and navigate to BIOS settings 
2. In System Profile Settings, select the Database Optimized Performance Profile in the Workload profile dropdown list

To enable Virtualization Technology on the iDRAC console: 

1. Navigate to Processor Settings
2. Select Enabled for Virtualization Technology

These settings are described in Table 22. 

Table 22: Database Optimized Performance profile

Use the following steps to set the DOP Workload Profile: 

1. Log in to the iDRAC console and go to BIOS settings 
2. In System Profile Settings, select DataBase Optimized Performance Profile in the Workload profile dropdown list

Figure 14: Set the Database Optimized Performance Workload Profile

Additional Resources

Setting up BIOS on 15th Generation (15G) Dell PowerEdge Servers

By default, the x2APIC BIOS setting is disabled on the PowerEdge R750xs server. In this best practice, we enabled the x2APIC setting to evaluate the impact on performance. 

Overview

The x2APIC is Intel’s Advanced Programmable Interrupt Controller design to improve efficiency in multiprocessor computer systems. An interrupt is a request by the software for the processor to respond to an event. For example, if an operating system requests that the processor suspend current activities and the processor accepts this request, the state of the current activities is saved, and the new request is processed. 

The x2APIC is Intel’s most recent Advanced Programmable Interrupt Controller. Enhancements to x2APIC include support for more processors and improved performance. The PowerEdge R750xs we used for testing best practices had two Intel Xeon Gold 6338 processors each with 32 cores for a total of 64 cores in the server. We also used VMware vSphere 7.3 to virtualize the Oracle database. Enabling x2APIC should create efficiencies for the PowerEdge multi-processor system and optimize the interrupt management of virtual machines.

Recommendation

Overall, the best practice of enabling x2APIC in BIOS can provide a minor improvement in the system performance. Enabling x2APIC is a Day 3, Fine-Tuning recommendation with these considerations:

1. For new PowerEdge servers, consider enabling the x2APIC BIOS setting if the virtualization and/or operating system supports the new interrupt controller.
2. For existing PowerEdge servers, consult support documentation and consider enabling this BIOS setting at the next maintenance window. 

Implementation steps

To enable the x2APIC BIOS setting, log in to iDRAC and do the following:

1. Click the Configuration pulldown menu
2. Click the BIOS Settings and Open Processor Settings tab
3. Navigate to x2APIC Model and select the Enabled radio button
4. At the bottom of the page, click Apply and Reboot

Figure 15: Enable Virtualization Technology Setting

Additional resources

Setting up BIOS on 15th Generation (15G) Dell PowerEdge Servers

The PowerEdge servers come with the Dell Controlled Turbo feature which controls the turbo engagement of the CPUs. This feature can increase performance in both single-threaded and multithreaded applications.

Overview

The Dell Controlled Turbo setting allows the CPUs to run at maximum speed.  This option only works if the system profile is set to performance in the BIOS. In this best practice, we enabled the Dell Controlled Turbo with the Database Optimization Performance profile.  

Recommendation

After enabling the Dell Controlled Turbo option, the following test metrics gained an increase in performance and is considered as a Day 1, Highly Recommended best practice.

1. New Order per Minute (NOPM)
2. Transactions per Minute (TPM)
3. PowerStore IOPS
4. Server CPU utilization

Implementation steps

To enable Dell Controlled Turbo, log in to iDRAC and do the following:

1. Click the Configuration pulldown menu 
2. Click the BIOS settings 
3. Navigate to Processors
4. Select Dell Controlled Turbo
5. Select Enabled in the dropdown list

Additional resources

Enable Dell Controlled Turbo on 15th Generation (15G) Dell PowerEdge Server.

Disable unused Integrated devices and turn off serial communication

By default, PowerEdge servers have USB and other ports enabled. In this best practice USB and serial ports were disabled as both a security and performance best practice. 

Overview

It is not expected that disabling USB ports and serial communication on servers will significantly improve performance, however, it is a good security practice. In this best practice the following ports were disabled in BIOS:

1. User accessible USB ports
2. Internal USB ports
3. iDRAC Direct USB port
4. Serial communication

Recommendation

Although disabling unused integrated devices showed no improvement, it is still considered as a Day 3, Fine Tuning best practice for security purposes.

Implementation Steps

Go to IDRAC > Configuration > BIOS settings > Integrated Devices and make the following changes:

1. User Accessible USB Ports setting from All Ports On to All Ports Off
2. Internal USB Port setting from On to Off
3. iDRAC Direct USB Port setting from On to Off

Additional Resources

Setting up BIOS on 15th Generation (15G) Dell PowerEdge Servers

vVols is a new technology from VMware that offers a better way of provisioning, managing, and accessing virtual disks. vVols access and management are enabled by a software component called vStorage APIs for Storage Awareness (VASA) provider, which is also called as vendor provider. VASA providers are developed by storage vendors. The vendor providers enable the creation of protocol end points which act as access points between the hosts and storage systems. vVols offer multiple benefits such as:

• Storage policy-based management (SPBM) which allows to create and apply policies based on storage tier 
• Improved datastore management
• Enhanced fine-grained storage operations and services 

For more information about the benefit of vVols, see vSphere Virtual Volumes (vVols) and vVols Getting Started Guide.

Migrating to vVols can be achieved easily by implementing the steps outlined in this chapter. 

The migration process can be achieved by using the following procedure:

vVols configuration and migration process

• Time synchronization
• Storage provider registration
• Create storage container
• Create virtual datastore for vVols
• Migrate the datastore and disks from VMFS to vVols using storage vMotion

It is recommended to synchronize the time for all the hosts and vCenter Server instances.

Time Synchronization

Perform the following steps for all the hosts that will use vVols.

1. Select the host in the vSphere inventory.

2. Click the configure tab.

3. In the System section, select Time Configuration.

4. Click Add service or Edit and set up the NTP server.

Figure 20.        vSphere Time Configuration

5. Select Use Network Time Protocol (Enable NTP client).

6. Enter the IP addresses of the NTP servers to synchronize with.

Figure 21.       Time Configuration: NTP Servers Setting

7. Click OK. The host synchronizes with the NTP server.

8. Stop/Start as required.

Storage Provider Registration

To use vVols, The VASA provider must be registered with the vCenter. The storage capabilities are exported and presented to VMware based virtual infrastructure by using the protocol end points enabled by VASA.

Figure 22.       VASA Storage Provider Registration

Storage Provider Registration Procedure

1. Login to vSphere client.

2. Go to vCenter Server in the vSphere Web Client navigator.

3. Click the Configure tab and click Storage Providers.

4. Click the + Add icon to register a storage provider.

5. Enter the connection details for the new storage provider, including the name, URL, and credentials.

6. Click OK to complete the registration.

Figure 23.       VASA Storage Provider Registration

7. The storage provider VASA has been successfully registered with the vCenter as shown in Figure 22.

8. Once the VASA is registered, create a vVol based datastore to utilize vVol based services

Virtual datastore creation procedure

1. Login to the vSphere Web Client.

2. Select the host in the vSphere inventory.

3. Right-click on the host and select the storage menu.

4. Click New Datastore.

5. Enter a unique datastore name.

Figure 24.       vVol Virtual Datastore Creation

6. Select vVol as the virtual datastore type.

Figure 25.       vVol Virtual Datastore Creation

7. Select the appropriate backing storage container which refers to the VASA that has been previously registered. This backing container will be used to host the virtual volumes.

Figure 25.             vVol Virtual Datastore Creation

8. Click next to review the selections and click Finish.

Figure 26.         vVol Virtual Datastore Creation

9. At the end of the procedure, the newly created datastore can be viewed on the host.

10. To access the datastore on additional host(s), the datastore must be explicitly mounted on the required hosts based on the need.

Mounting the Virtual Datastore on additional hosts

1. Perform this procedure from the Storage view of the vSphere Client.

2. From the vSphere Client, go to the Storage view tab.

3. Right-click on the preferred vVol virtual datastore and click Mount Datastore to Additional Hosts

4. Select the available hosts to mount the Virtual Datastore and click OK.

Figure 27.             vVol Virtual Datastore Host Connectivity

Virtual Machine Migration with Storage vMotion

1. Select the required virtual machine in the vSphere inventory and click Migrate.

2. Select Change storage only and click Next.

Figure 28.       Storage Migration: VMFS to vVol

3. Select the vVol datastore destination storage, virtual disk format (thick provisioning), and click Next.

Figure 29: Storage Migration: VMFS to vVol

4. Review the migration details including migration type, virtual datastore target, and disk format settings. Click Finish.

Figure 30.             Storage Migration: VMFS to vVol

5. Depending on the number and size of the source virtual disks, it can take several minutes to complete the migration from VMFS to vVols.

6. You can check the status of the migration under the recent tasks tab of the vSphere client.

Figure 31.       Storage Migration: VMFS to vVol

7. This migration process will create the necessary virtual volumes in the storage container of the storage system, as shown in Figure 32.

Figure 32.             Storage Migration: Virtual Volumes

8. Upon the completion of migration, we can confirm the availability and mapping of the new vVol datastore by browsing to the datastores tab of the vm.

Figure 33.             Post Storage Migration: VM with vVol Datastore

See vSphere Virtual Volumes (vVols) and vVols Getting Started Guide for more information about implementing and migrating vVols.

The Integrated Solutions team developed, validated, and consolidated Oracle database best practices on 15G PowerEdge servers with Intel processors, PowerSwitches, and PowerStore 5000T storage. Improving customer outcomes continues to be a key tenet across all Oracle solutions and the validated best practices demonstrate  our commitment to ensuring customer success.

By grouping best practices by their return on performance, we streamline how customers can achieve the best results. This reduces complexity and risk by providing a plan our customers to follow: Day 1, then Day 2 and, if necessary, Day 3 best practices.

We value your feedback

Dell Technologies and the authors of this document welcome your feedback on the solution and the solution documentation. Contact the Dell Technologies Solutions team by email.

We used UDEV rules to assign Linux 0660 permission and ownership of disks to the grid user (the owner of the Oracle GI and Oracle ASM instance). The following scripts include the UDEV rules set for the disks as an entry of the custom script in: /etc/udev/rules.d/95oracleasm.rules: 

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29dfb32d8c38abe3cc89cb46ab1", SYMLINK+="oraudev/disks/ora-ocr1",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c298e41c93e74d5bdbdbd7712790", SYMLINK+="oraudev/disks/ora-ocr2",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c2927b8c006c81f140601bcb77b2", SYMLINK+="oraudev/disks/ora-ocr3",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29278ba48aaed314d4386265b59", SYMLINK+="oraudev/disks/ora-data1",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c296e4e4e36df967fcc50aa0699d", SYMLINK+="oraudev/disks/ora-data2",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29d0929bb25a7855f1c2b8529e1", SYMLINK+="oraudev/disks/ora-data3",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29290b303b4ecaa771fb8992649", SYMLINK+="oraudev/disks/ora-data4",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29742d6c21ad60006fec5a328f0", SYMLINK+="oraudev/disks/ora-redo1",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29373beaca55d865c77a3e68c99", SYMLINK+="oraudev/disks/ora-redo2",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29030f50f193d11448ac05f5397", SYMLINK+="oraudev/disks/ora-redo3",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29434bf958b9a5b3eb2ac03ce25", SYMLINK+="oraudev/disks/ora-redo4",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29d2f3cc94b6f73cecee16d2ca1", SYMLINK+="oraudev/disks/ora-fra1",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29314f941860f660a53328e4425", SYMLINK+="oraudev/disks/ora-fra2",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29404582742e38a9682fc2651bb", SYMLINK+="oraudev/disks/ora-fra3",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c295c31d03eee81d528b52fb3bff", SYMLINK+="oraudev/disks/ora-fra4",  OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd[a-z]*[1-9]", SUBSYSTEM=="block", OPTIONS:="nowatch", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29885601765374bae4633290905", SYMLINK+="oraudev/disks/ora-temp",  OWNER="grid", GROUP="asmadmin", MODE="0660"