In this best practice we used Oracle scripts to implement HugePages in the Linux operating system. 

Overview

The default memory page size used by the Linux operating system is 4 KB. While the 4 KB works well for many applications, the Oracle Shared Global Area (SGA) memory pool can be very large. The memory size of the SGA is large because it supports components like the database buffer cache, shared pool, large pool, and many other memory pools. Thus, management of the SGA using 4 KB memory pages can cause excessive processing and wait times to access memory pages. 

HugePages are the capability of the Linux administrator to configure the operating system to use 2 MB page sizes. The larger memory pages mean the operating system can manage less pages and access memory pages faster thus, optimizing memory management. The additional benefit of using HugePages is the larger pages cannot be swapped out to disk. Over time HugePages has become an accepted best practice that many Oracle systems use to optimize performance. 

Recommendation

We implemented 2 MB HugePages in the Linux operating systems for all the Oracle databases and then tested the configuration by running an OLTP workload. Test results showed no changes for the following performance metrics: 

• New Orders per Minute (NOPM) 
• Transactions per Minute (TPM) 
• PowerMax IOPS

Test results show moderate savings for:

• Server CPU Utilization

HugePages optimizes the memory configuration of the Linux operating system. The optimized memory configuration did not improve NOPM, TPM, and IOPS.  

Server CPU utilization showed a moderate decrease.

The benefit of HugePages is that the operating system manages fewer larger pages, which eases the management. The overall CPU utilization of the server showed a moderate savings when using HugePages. The savings in CPU utilization validates that HugePages optimizes page management. 

DB File Sequential Read remained the same and Log File Parallel Write increased slightly as a result of the increases in NOPM and TPM. 

Overall, HugePages optimized memory of the Linux operating system leading to improved performance and a decrease in CPU utilization. For these reasons we recommend implementing HugePages is a Day 2, Moderate recommended best practice. 

Implementation Steps

To enable the Hugepages, complete the following steps:

1. Run this command to determine if the kernel supports HugePages:

   $ grep Huge /proc/meminfo
   AnonHugePages: 0 kB
   ShmemHugePages: 0 kB
   HugePages_Total: 0
   HugePages_Free: 0
   HugePages_Rsvd: 0
   HugePages_Surp: 0
   Hugepagesize: 2048 kB
    Hugetlb:               0 kB

2. Edit memlock setting in /etc/security/limits.conf. The memlock setting is specified in KB, and the maximum locked memory limit should be set to at least 90 percent of the current RAM when HugePages memory is enabled. For example, if the server RAM size is 128GB, the memlock can be set  128GB * 90% = 120,795,955 KB and add the following lines into this file:

   *    soft   memlock    120,795,955
    *   hard   memlock    120,795,955

3. Log in as oracle user again and run the ulimit -l command to verify the new memlock setting:

       $ ulimit -l
         120795955

4. Get the HugePage size by running this command:

   $ grep Hugepagesize  /proc/meminfo

5.  Create a script hugepages_settings.sh that computes recommended values for hugepages configuration for the current shared memory segments:

    !/bin/bash
   #
   # hugepages_settings.sh
   #
   # Linux bash script to compute values for the
   # recommended HugePages/HugeTLB configuration
   #
   # Note: This script does calculation for all shared memory
   # segments available when the script is run, no matter it
   # is an Oracle RDBMS shared memory segment or not.
   # Check for the kernel version
   KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`
   # Find out the HugePage size
   HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`
   # Start from 1 pages to be on the safe side and guarantee 1 free HugePage
   NUM_PG=1
   # Cumulative number of pages required to handle the running shared memory segments
   for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`
   do
   MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
   if [ $MIN_PG -gt 0 ]; then
   NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
   fi
   done
   # Finish with results
   case $KERN in
   '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
   echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
   '2.6'|'3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
   *) echo "Unrecognized kernel version $KERN. Exiting." ;;
   esac
    # End

6. Provide the execute permission of this script

   $chmod +x hugepages_settings.sh

   1. And execute this script to values for HugePages configuration:

      $./hugepages_settings.sh

7. Set the following kernel parameter, where value is the HugePages value that you determined in step 6

   # sysctl -w vm.nr_hugepages=value
   
   
   Check the available hugepages:
    $grep Huge /proc/meminfo

8. Restart the database instance and run the command to check the available HugePages

    $grep Huge /proc/meminfo

Additional Resources

Oracle Database Online Documentations 12c, Huge Pages

Administering Oracle Database 19c on Linux: Overview of HugePages

The TUNED_ADM command line tool enables switching between different tuning profiles in Linux. In this best practice, we used the TUNED_ADM tool to switch to an optimized Linux profile for Oracle. 

Overview

The TUNED_ADM tool enables the Linux administrator to switch between profiles for the operating system. Predefined profiles enable the application of Linux configuration settings in a package that can be downloaded. In this best practice the TUNED-PROFILES-ORACLE was downloaded and installed in the Linux OS. The tuned Oracle profile made the following changes:  

• Disabled Transparent Hugepages (THP) 

 THP automates the promotion, small to large, and demotion, large to small, of memory pages. Oracle databases do not use transparent hugepages. 

• VM.SWAPPINESS was set to 1

Swap space is used by the Linux operating system as virtual memory. The way this works is the Linux kernel writes data from RAM to the swap space, allowing for greater use of memory on the system. By changing VM.SWAPPINESS to 1 the kernel is instructed to more aggressively reclaim mapped memory rather than using swap space. 

• VM.DIRTY_BACKGROUND_RATIO was set to 3

The DIRTY_BACKGROUND_RATIO is the percentage of memory that contains free pages and reclaimable pages that can be filled with dirty pages before the pages are written to disk. Databases can aggressively use the kernel page cache thus, lowering the background ratio forces the kernel to flush the dirty pages to disk more frequently. 

• VM.DIRTY_RATIO set to 40

The DIRTY_RATIO is the percentage of total system memory that contain dirty pages before the pages are written to disk. By setting DIRTY_PAGES to 40 the kernel is instructed to use 40% of total system memory for dirty pages before having to flush the data to disk. 

• VM.DIRTY_EXPIRE_CENTISECS is set to 500

The DIRTY_EXPIRE_CENTISECS defined in centisecs, one hundredth of a second, when data is old enough to be written to disk. By setting DIRTY_EXPIRE_CENTISECS to 500, a dirty page can stay in memory for 500 centisecs before it is written to disk. 

• VM.DIRTY_WRITEBACK_CENTISECS is set to 100

The DIRTY_WRITEBACK_CENTISECS parameter defines the frequency the kernel will check to write dirty pages to disk. In this case, a value of 100 centisecs instructs the kernel to check every 100 cs to write dirty pages to disk. 

• VM.TCP_RMEM_MAX is set to 4194304

The VM.TCP_RMEM_MAX parameter defines the maximum amount of receive socket memory. 

Recommendation

The approach of using the TUNED_ADM tool to apply the TUNED-PROFILES-ORACLE package automates the application of many best practices in the Linux operating system. After applying the tuned Oracle settings, we load tested the new configuration and compared it to the prior performance test. The load OLTP load test showed moderate performance improvements in the following metrics:

• New Orders per Minute (NOPM) 
• Transactions per Minute (TPM) 
• PowerMax IOPS
• Server CPU utilization
• DB File Sequential Read 
• Log File Parallel Write 

The TPC-C NOPM and TPM metrics showed a moderate increase, indicating the changes to the Linux operating system improved performance. PowerMax IOPS and overall CPU utilization showed slight improvements. Finally, DB File Sequential Read and Log File Parallel Writes show slight increases in average times. The slight increase in times for DB file sequential read and log file parallel writes indicate the database is processing more new orders and transactions per minute causing an uptick the two-file metrics.  

Overall, the configuration changes made to the Linux operating system fine tuned how memory is managed. These changes resulted in more efficient memory usage thus, improving the number of New Orders per Minute the database was able to process. The benefits of using the tuned settings for Oracle make this a moderate best practice recommendation. 

Implementation Steps

In this test case, we are going to install Red Hat tuned-profiles-oracle profile which is recommended and designed to run Oracle database. Use the following procedure to install and enable tuned-profiles-oracle profile:

1. We used a local Yum repository created based on Red Hat Enterprise Linux 8.3 to enable the profile.
   1. Make sure the repository is available

      # yum list

2. Install the tuned profile for oracle

   # yum install tuned-profiles-oracle

3. To see which profiles are available, run the following command:

   # tuned-adm list
    
   Enable the profile
   # tuned-adm profile oracle
   Verify the active profile
   # tuned-adm active
   You should see an output like
   Current active profile: oracle

Additional Resources

Red Hat Enterprise Linux 8: Tuned Profiles