This paper delivers straightforward guidance to customers using Dell Unity All-Flash storage systems in an Oracle 21c database environment running Oracle Linux. Oracle is a robust product that can be used in various solutions. The relative priorities of critical design goals such as performance, manageability, and flexibility depend on your specific environment. This paper provides considerations and recommendations to help meet your design goals.

This paper was developed using the Dell Unity 880F All-Flash array, but the information is also applicable to other Dell Unity All-Flash array models (x80F and x50F). Oracle Linux (OL) 8 with the Unbreakable Enterprise Kernel (UEK) was used for this paper.

These guidelines are recommended, but some recommendations may not apply to all environments. For questions about the applicability of these guidelines in your environment, contact your Dell Technologies representative.

Dell Unity x80F models provide an excellent storage solution for Oracle workloads regardless of the application characteristics and whether file or block storage is required. This paper discusses the best practices and performance of the Dell Unity 880F array with block storage, but also presents best practices with native xNFS or Oracle dNFS.

In addition to file and block storage support, the Dell Unity x80F arrays provide several other features. Some of the standard features are point-in-time snapshots, replication (local and remote), built encryption, compression, and extensive integration capabilities for an Oracle standalone or RAC environment.

Some recommendations in the paper might not apply to all environments. For questions about the applicability of these guidelines in your environment, contact your Dell Technologies representative.

This document is intended for IT administrators, storage architects, partners, and Dell Technologies employees. This audience also includes individuals who evaluate, acquire, manage, operate, or design a Dell networked storage environment using Unity XT systems. It is assumed that readers have prior experience with or training in the following areas:

• Dell Unity storage systems
• Linux operating environment
• Multipath software
• Oracle Automatic Storage Management (ASM)
• Oracle standalone or RAC environment

Dell Technologies and the authors of this document welcome your feedback on this document. Contact the Dell Technologies team by email.

Authors: Mark Tomczik

Note: For links to other documentation for this topic, see the Unity XT Info Hub

Dell Unity XT storage is a virtually provisioned Hybrid Flash Array (HFA) or All-Flash-optimized Array (AFA) designed for ease of use. This paper covers the AFA models with emphasis on Dell Unity XT x80F arrays. This section describes the foundational array technologies that support the application-specific sections that follow. For additional information, see Dell Unity XT Unified Storage Solutions. For general Dell Unity XT best practices, see the Dell Unity XT: Best Practices Guide.

All Dell Unity All-Flash arrays provide an embedded I/O module (optional on some models) and two optional I/O modules per storage processor (SP). This section provides an overview of the different options available in each of the I/O modules. Recommendation on choosing I/O modules for an Oracle environment where performance and throughput are of interest is also given.

I/O ports used for transferring user data are available on the embedded module or additional I/O modules on each SP.

Table 1.      Embedded I/O module

For Oracle environments that require FC front-end connectivity on the Dell Unity 480F, 680F, or 880F models, consider using a filler blank in place of the embedded I/O module. Also consider using one or two optional I/O modules that support FC.

If Oracle dNFS is used, consider using the optional four-port mezzanine card on the Dell Unity 480F, 680F, or 880F models. A four-port mezzanine card can be in both Link Aggregation Control Protocol (LACP) and fail-safe networking (FSN) configurations. LACP is discussed in section Configuring LACP – NAS client, and Configuring LACP—NAS server.

Table 2.      Optional I/O modules for Dell Unity All-Flash arrays

In high-demand Oracle environments where IOPs, latency, or capacity are a concern, consider the option of using a 4-port 12 Gb SAS I/O module. The 4-port module will increase the number of configurable hard drives in the array which can help lower latency and increase IOPS and capacity.

I/O modules must be installed in pairs (one in SPA and one in SPB), are of the same type and reside in the same slots between SPA and SPB.

With Dell Unity 480F, 680F, and 880F models, slot 0 I/O modules have x16 PCIe lanes while slot 1 has x8 PCIe lanes. For this reason, slot 0 should be reserved for environments needing greater bandwidths.

The Ethernet/iSCSI card can be in both Link Aggregation Control Protocol (LACP) and fail-safe networking (FSN) configurations).

Once the Dell Unity array is configured, all I/O modules are persistent and cannot change type.

Dell Unity storage supports two types of storage pools on All-Flash storage systems: traditional pools and dynamic pools. Dynamic pools were introduced in Dell Unity OE version 4.2 for all-flash storage models and became the default pool type in Dell Unisphere. While traditional pools are still supported on all-flash models, they can only be created through the Unisphere CLI or REST API.

Dynamic pools offer many benefits over traditional pools. The new pool structure eliminates the need to add drives in the multiples of the RAID width, allowing for greater flexibility in managing and expanding the pool. Dedicated hot spare drives are also no longer required with dynamic pools. Data space and replacement space are spread across the drives within the pool. Spreading space across drives within the pool provides better drive utilization, improves application I/O, and speeds up the proactive copying of failing drives and the rebuild operation of failed drives.

In general, create dynamic pools with large numbers of drives of the same type and use few storage pools within the Dell Unity system. However, it may be appropriate to configure additional storage pools in the following instances:

• Separate workloads and resources from competing databases or applications
• Dedicate resources to meet specific performance goals
• Create smaller failure domains

Additional information can be found in the documents, Dell Unity: Dynamic Pools and Dell Unity Family Configuring Pools.

Storage pool capacity

Storage pool capacity is used for multiple purposes:

• To store all data written into storage objects — LUNs, file systems, datastores, and VMware vSphere Virtual Volumes (vVols) — in that pool
• To store data that is needed for snapshots of storage objects in the pool
• To track changes to replicated storage objects in that pool

Storage pools must maintain free capacity to operate properly. By default, a Dell Unity system will raise an alert if a storage pool has less than 30% free capacity. If the alert is raised, the system will begin to automatically invalidate snapshots and replication sessions if the storage pool has less than 5% free capacity. Dell Technologies recommends that a storage pool always has at least 10% free capacity.

More drives can be added to a storage pool online. However, to optimize the performance and efficiency of the storage, add drives with the same specification, type, and capacity of the existing drives in the pool. Though not required, add several drives equal to the RAID width + 1, which allows the new capacity to be immediately available. Data is automatically rebalanced in the pool when drives are added.

Note: Once drives are added to a storage pool, they cannot be removed unless the storage pool is deleted.

All-Flash pool

All-flash pools provide the highest level of performance in Dell Unity systems. Use an all-flash pool when the application requires the highest storage performance at the lowest response time. Note the following considerations with all-flash pools:

• All-flash pools consist of either all SAS flash 3 or all SAS flash 4 drives of the same capacity.
• Dell FAST Cache and FAST VP are not applicable to all-flash pools.
• Compression is only supported on an all-flash pool.
• Snapshots and replication operate most efficiently in all-flash pools.
• Dell Technologies recommends using only a single drive size and a single RAID width within an all-flash pool.

For example: For an all-flash pool, use 800 GB SAS flash 3 drives and configure them all with RAID 5 8+1.

Hybrid pool

Hybrid pools (including a combination of flash drives and hard disk drives) are not supported with Dell Unity All-Flash arrays.

This section describes some of the native features available on the Dell Unity platform. Not all are applicable to Dell Unity All-Flash arrays and are noted in this document. Additional information about each of these features can be found in the Dell Unity XT: Best Practices Guide.

Dell FAST VP accelerates the performance of a specific storage pool by automatically moving data within that pool to the appropriate drive technology based on data access patterns. FAST VP is only applicable to hybrid pools within a Dell Unity Hybrid flash system.

Dell FAST VP accelerates the performance of a specific storage pool by automatically moving data within that pool to the appropriate drive technology based on data access patterns. FAST VP is only applicable to hybrid pools within a Dell Unity Hybrid flash system.

Dell Unity compression reduces the amount of physical storage needed to save a dataset in an all-flash pool for block LUNs and VMFS datastores. This capability was added to Dell Unity OE version 4.1 for thin block storage resources and was called Dell Unity Compression. Thin file storage resource support was added in Dell Unity OE version 4.2 for file systems and NFS datastores in an all-flash pool.

In Dell Unity OE version 4.3, the Dell Unity Data Reduction feature replaces compression. It provides more space-savings logic to the system with the addition of zero-block detection and deduplication. In Dell Unity OE version 4.5, data reduction includes an optional feature called Advanced Deduplication, which expands the deduplication capabilities of the data reduction algorithm. With data reduction, the amount of space required to store a dataset for data reduction-enabled storage resources is reduced when savings are achieved. Data reduction and advanced deduplication are supported on LUNs, file systems, NFS, and VMFS datastores. Starting with OE 4.5, an 8 KB Dell Unity block within a resource is subject to compression. The block will be compressed if a 1% savings or higher can be obtained.

Dell Unity Data Reduction savings are not only achieved on the storage resources it is enabled on, but on snapshots and thin clones of those resources. Snapshots and thin clones inherit the data reduction setting of the source storage resource, which helps to increase the space savings that they can provide.

If Dell Data Reduction is enabled, the storage system intelligently controls it. Configuring data reduction and reporting savings is simple, and can be done through Unisphere, Unisphere CLI, or REST API.

Dell Unity Data Reduction is licensed with all physical Dell Unity systems at no additional cost. Data reduction is not available on the Dell Unity VSA version of the Dell Unity platform as data reduction requires write caching within the system. Dell Unity must be at OE version 4.3 or later to use data reduction with block and file resources (thin LUNs, thin file systems, VMware vStorage VMFS datastores, and NFS).

By offering multiple technologies of space saving, Dell Unity provides flexibility for the best balance of space savings and performance.

Table 3.      Dell Unity All-Flash arrays: data reduction and advanced deduplication

* Resource can be created on either a traditional or a dynamic pool (for systems that support dynamic pools).

Note: Data reduction is disabled by default and needs to be enabled before advanced deduplication is an available option. After enabling data reduction, advanced deduplication is available, but is disabled by default.

While data reduction helps to optimize storage investments by maximizing drive utilization, data reduction:

• Increases the overall CPU load on the Dell Unity system when storage objects service reads or writes of compressible data, and
• May increase latency when accessing the data.

Consider these best practices before enabling data reduction on a storage object:

• Monitor the system to ensure it has available resources to support data reduction. See “Hardware Capability Guidelines” section and Table 2, Recommended sustained CPU utilization, in the Dell Unity XT: Best Practices Guide.
• Enable data reduction on a few storage objects at a time. Then monitor the system to be sure it is still within the recommended operating ranges before enabling Data Reduction on more storage objects.
• With Dell Unity x80F models, consider that data reduction will provide space savings if the data on the storage block is at least 1% compressible. Before the new x80F models and OE 5.0, data reduction would provide space savings if the data on the storage block was at least 25% compressible.
• Before enabling data reduction on a storage object, determine if it contains data that will compress. Do not enable data reduction on a storage object if there will be no space savings.
• Contact your Dell Technologies representative for tools that can analyze the data compressibility.

For additional information regarding compression and data reduction, see the Dell Unity: Data Reduction document.

Advanced deduplication

Advanced deduplication, an optional extension of data reduction released in OE 4.5, increases the capacity efficiency of data reduction. Advanced deduplication can be enabled on storage and is only performed on compressed blocks. With OE 4.5, that meant that advanced deduplication would be performed on compressed blocks that sustained as little as a 1% savings or higher. In cases where a Dell Unity block did not compress, advanced deduplication would not be performed. So if there were multiple copies of an uncompressed block, deduplication would not be applied to the uncompressed multiple block copies to realize further storage savings.

This restriction of not performing advanced deduplication on blocks with less than a 1% compression savings no longer exists in OE 5.0. With OE 5.0, advanced deduplication can deduplicate to an uncompressed block whenever a write or overwrite occurs on the block even if the block has 0% compression.

For more information regarding advanced deduplication, see the Dell Unity: Data Reduction document and the Dell Unity: Best Practices Guide document.

Many Oracle database applications have data encryption requirements. Data at Rest Encryption (D@RE) is a controller-based encryption solution that can be used for Oracle databases without requiring any database or application changes.

Note: D@RE is a license-able feature and must be selected during the ordering process and licensed at system initialization. D@RE can only be enabled at the time of system installation with the appropriate license and cannot be enabled later.

If encryption is enabled, Dell Technologies recommends making external backups of the encryption keys after system installation, and immediately following any change in the system’s drive configuration. Changes to the system’s drive configuration would include such things as creating or expanding a storage pool, adding new drives, or replacing a faulted drive.

For more information about D@RE, see the Dell Unity: Data at Rest Encryption document.

Typically, a Dell Unity system is used to service multiple hosts and applications. These applications can have different service levels and different storage demands. In addition, a single array can provide services to multiple environments such as development, test, and production. Traditionally, these scenarios have been difficult to manage to ensure critical applications get the resources they need while managing less critical resources to ensure they do not over consume.

Host I/O limits, like Quality of Service (QoS), provide an excellent means to manage these types of workloads. Instead of trying to manage workloads with multiple storage pools, use host I/O limits. Host I/O limits allow LUNs to be restricted to a specified amount of IOPS or bandwidth, so they do not adversely impact other applications. Host I/O limits allow storage administrators to ensure applications and environments adhere to budgeted limits which greatly simplify planning and management.

Host I/O limits are recommended for Oracle database environments for several reasons. First, storage administrators can ensure that demanding Oracle databases instances do not overwhelm the entire array by setting limits on database volumes. Also, if the Oracle database is the priority application, they can set limits on other LUNs on the system to ensure that the Oracle database gets the required resources. Another great component of host I/O limits is the ability to burst for a given limit for a specific period, which is user configurable. In this way, small exceptions can still be allowed while maintaining balanced performance.

In development and testing environments, it can be difficult to determine if an application meets performance requirements. Typically, these environments are smaller than production environments and it is not always feasible to keep a copy of production data in these environments due to costs or privacy concerns. An issue with smaller datasets is that the application can run faster and then encounter serious performance issues when deployed on a real dataset in production.

Host I/O limits can be used to restrict the I/O on smaller datasets to highlight I/O-intensive queries. Setting limits on databases in development and testing environments will help identify problem areas so they can be resolved before production deployment. The result is improved Oracle databases service levels and greater scalability.

For additional information, see the Dell Unity: Unisphere Overview document.

Unity provides both a REST API and CLI interface which can be used for ad hoc and batch transactions or reporting in Unity. Batched processes provide a method that can be performed multiple times without introducing typographical errors between multiple executions. A use case for batch processes is the creation of a particular volume layout that needs to be the same between environments. Another use case is a batch process that generates a particular report across multiple environments.

For more information about Unity REST API, see:

• ‎REST API and Dell Technologies Storage - Unity | DELL Technologies,
• https://<unisphere_management_address>/apidocs/index.html, and
• https://<unisphere_management_address>/apidocs/programmers-guide/about_notes_cautions_warnings.html
• Dell Unity: Unisphere Overview (delltechnologies.com)
• Dell Unity™ Family Unisphere® Management REST API Programmer's Guide
• Dell Unity Unisphere Management REST API Reference Guide (PDF)

For more information about Unisphere CLI for Unity, see:

• Dell Unity: How to use Unisphere CLI (UEMCLI) commands | Dell US
• Dell Unity™ Family Unisphere® Command Line Interface User Guide

Dell Unity can run service commands from the Unity service account. Some service commands provide more ability to monitor and manage Unity. Some service commands provide the ability to:

• In unified configurations, determine the current setting of Unity’s rsize and wsize of the NFS buffer. Knowing the current setting of Unity’s rsize and wsize is useful when configuring and tuning Oracle dNFS.
• Check NFS network performance metrics.

Service commands are intended for the Storage Administrator, field services, and support personal. Since the scripts require a lower level of system control, use care when processing the scripts.

NOTE: Users should be familiar with Linux, Unity installed hardware, and the Unity operating system. Some service scripts must not be run without the approval from your authorized service representative or support personnel. Contact your service representative or support for guidance in using the service commands.

Service commands are run from Unity’s service account and require SSH to be enabled in Unity. The service account password is initially defined when Unity is deployed and configured. By default, the password is the same as the admin password but can be changed during deployment.

When connecting to Unity using SSH, the SSH client should use Unity’s Unisphere’s IP address.

Once connected to the service account, to see a list of available service commands, type svc_help.

The remainder of this section shows several use cases with several service commands:

• Enable/disabling/viewing nfs.transChecksum in Unity when using Oracle direct nfs (dNFS)
• Display/change Unity’s current nfs v3 buffer sizes

Unity’s nfs.transChecksum parameter

To use Oracle dNFS with Unity storage, use NFS protocol NFSv3, and enable Unity’s nfs.transChecksum parameter (set to 1). This parameter ensures that each transaction carries a unique ID and avoids the possibility of conflicting IDs that result from the reuse of relinquished ports.

Determine if nfs.transChecksum is enabled or disabled

To display the current value of nfs.transChecksum, perform the following service command in an ssh session to the IP address of Unisphere:

svc_nas <NAS Server name> -param -facility nfs -info transChecksum

This service command will return a number of parameters and their current value. If the value returned for parameter current_value is 0, then nfs.transChecksum is disabled. If its value is 1, then nfs.transChecksum is enabled.

Enable Unity’s nfs.transChecksum

To enable nfs.transChecksum, shutdown all Oracle resources that reside on the NAS Server where the nfs.transChecksum will be enabled. Then proceed with the following steps:

1. Record the current nfs.transChecksum value (see Determine if nfs.transChecksum is enabled or disabled)

2. Perform the following service command to change the value of nfs.transChecksum to 1 which will enable nfs.transChecksum.

svc_nas <NAS Server name> -param -facility nfs -modify transChecksum -value 1

3. Perform the following service command to verify that the value of nfs.transChecksum was changed. The service command will return a number of parameters. If parameter current_value is 1, then nfs.transChecksum was changed correctly.

svc_nas <NAS Server name> -param -facility nfs -info transChecksum

4.  Perform the following service command to restart the NAS server that had its transChecksum value changed.

svc_nas <NAS Server name> -restart

Disable Unity’s nfs transChecksum

To disable nfs.transChecksum, shutdown all Oracle resources that reside on the NAS Server where the nfs.transChecksum will be enabled. Then, perform the following to disable nfs.transChecksum:

svc_nas ORA-ASM-NAS -param -facility nfs -info transChecksum

svc_nas <NAS Server name> -param -facility nfs -modify transChecksum -value 0

svc_nas ORA-ASM-NAS -param -facility nfs -info transChecksum

svc_nas <NAS Server name> -restart

svc_nas ORA-ASM-NAS -param -facility nfs -info transChecksum

Note: Only disable nfs.transChecksum if Oracle dNFS will no longer be used. nfs.transChecksum ensures that each database transaction carries a unique ID and avoids the possibility of conflicting IDs that result from the reuse of relinquished ports.

Unity’s nfs.v3xfersize parameter

It may be advantageous to change the size of Unity’s nfs buffer. When mounting a Unity NFS filesystem on a client, the filesystem should be mounted with wsize and rsize equal to the size of the Unity NFS buffer.

Display Unity’s current nfs v3 buffer sizes

To display the current size and range of Unity’s NFSv3 I/O transfer buffer size, process the following service command.

svc_nas <NAS server name> -param -facility nfs -info v3xfersize

The output from this command is helpful when configuring NFS, especially when using Oracle dNFS. When mounting the NFS filesystem serviced by the Unity NAS Server, set the rsize and wsize to the value of parameter current_value reported by the Unity service command:

Figure 1.        Mount the filesystem on the client and set w/rsize to Unity’s NFS buffer size.

It is recommended to test these changes in a nonproduction environment before changing them in production.

Changing Unity’s nfs v3 buffer sizes

To change the size of Unity’s NFSv3 I/O transfer buffer, perform the following steps.

Notes:
1) Contact your Dell Technologies service representative for guidance in this process.

2) The process will change the buffer sizes for all NAS servers on SP A and SP B, and will require a reboot of both the local and peer SPs.

1. Perform the following service command and record the value of nfs.v3xfersize’s parameter “configured_value”.

svc_nas <NAS name> -param -facility nfs -info v3xfersize

2. Change the NFSv3 buffer size by performing the following service command. Set the value of v3xfersize’s parameter “configured_value” to <new-value>, the new wanted size.

svc_nas ALL -param -facility nfs -modify v3xfersize -value <new-value>

3. Verify the change was recorded and configured correctly by performing the following service command. The returned value of parameter “configured_value” should be set to the new value.

svc_nas <NAS name> -param -facility nfs -info v3xfersize

4. Reboot the local SP by performing the following service command.

svc_shutdown -r

5. After the reboot, what was the peer SP will become the primary SP. Perform the following service command to verify that parameter “configured_value” on the new primary SP is the value set in step 2.

svc_nas <NAS name> -param -facility nfs -info v3xfersize

6. Make the new value active by rebooting the new local SP. To reboot the SP, perform the following service command (SP A will again become the new local SP)

svc_shutdown -r

7. Verify the nfs.v3xfersize buffer size, parameter current_value, is active on SP A, SP B, and on all NAS Servers. To verify, perform the following service command.

svc_nas ALL -param -facility nfs -info v3xfersize

The storage system is a critical component of any Oracle database environment. Sizing and configuring a storage system without understanding the I/O requirements can have adverse consequences. This section discusses the types of database workloads and some of the common tools available to measure, collect, and analyze database system performance which helps define the I/O requirements. For an Oracle environment, capacity requirements can be as important as the number of I/Os per second (IOPS) and throughput requirements.

An online transaction processing (OLTP) workload typically consists of small random reads and writes. The I/O sizes are generally equivalent to the database block size. The primary goal of designing a storage system for this type of workload is to maximize the number of IOPS while keeping the latency as low as possible. Depending on the business and application requirement, a latency of less than 1 millisecond is typical in a high performing environment.

Consider using 16 Gb Fibre Channel (FC) or 25 GbE optical I/O models in each SP. If higher drive counts are necessary to achieve higher IOPS, use 12 Gb SAS IO modules for backend connectivity, such as from the storage processors (controllers) to the disk enclosures. 12 Gb SAS is only available in the 480F, 680F, and 880F models and 25 GbE optical is only available in Dell Unity x80/x80F models.

OLTP performance

For best results, capture performance statistics for at least 24 hours that include the system peak workload.

An OLTP workload typically consists of small random reads and writes. The backend storage system servicing this type of workload is primarily sized based on capacity and the number of IOPS required.

Unlike an OLTP workload, an online analytic processing (OLAP) or decision support system (DSS) workload typically has a relatively low volume of transactions. Most of the activities involve complex queries and aggregates a large dataset. The volume of data tends to grow steadily over time and is kept available for a longer time. OLAP workloads generally have large sequential reads or writes.

The primary goal of designing a storage system that services this type of workload is to optimize the I/O throughput. The design needs to consider all components in the entire I/O path between the hosts and the drives in the Dell Unity system. For best throughput, consider using:

• 16 Gbps FC or 25 GbE optical (10 GbE is also an option) iSCSI connectivity to the array, and
• 12 Gbps SAS connectivity from the controllers to the disk enclosures.

To meet high throughput requirements, multiple HBAs may be required on the server, the array, or both.

Oracle database workloads may not have I/O patterns that can be strictly categorized as OLTP or OLAP because several applications can reside within the same database. Multiple databases with different workloads can also co-exist on the same host. Choose and design a storage system that can handle different types of workloads. When testing the I/O systems, the combined workload of these databases should be accounted for and measured against the expected performance objectives.

The Dell all-flash midrange storage portfolio offers storage systems that scale in both IOPS and throughput. Combined with the advanced architecture and storage-saving, the Dell all-flash midrange platform is ideal for any type of Oracle workload.

In general, it is recommended to use fewer storage pools within Dell Unity systems. Fewer storage pools reduce complexity and increases flexibility. Dell Technologies recommends using a single virtual disk pool for hosting volumes for Oracle databases. A single virtual pool provides better performance by leveraging the aggregate I/O bandwidth of all disks to service I/O requests from Oracle databases. A single drive pool is easier to manage, allowing an administrator to easily adapt the storage system to satisfy ever-changing workloads that are common in Oracle databases environments. Before creating multiple storage pools to separate workloads, understand the various Dell Unity features that are available for managing and throttling specific workloads.

RAID configurations

By default, the Dell Unity system chooses RAID 5 as the protection level when creating a storage pool which contradicts traditional guidance advocating RAID 1/0 for database workloads. However, this traditional guidance assumes the storage system contains spinning disks and does not consider SSDs or flash-optimized storage such as Dell Unity systems. Testing Dell Unity All-Flash systems in most RAID 5 and RAID 1/0 configurations has shown negligible performance gains unless the workload is extremely write-intensive for an extended period. Usually, the small performance gain of RAID 1/0 is not worth the reduced capacity and therefore it is recommended to use the default configuration of RAID 5. For heavy write workloads where maximum write performance is required, RAID 1/0 can be used.

The I/O requirements need to be clearly defined to size storage correctly. The RAID type chosen will be determined by comparing availability and performance requirements. The small footprint and high IO density will typically allow a smaller drive size, reducing drive rebuild times, which means RAID 5 would be preferred over RAID 6 usually.

Once the I/O requirements have been defined, I/O performance should be validated and tested before putting the environment into full production mode. After the system goes into production, it is imperative to continue to collect and analyze the performance data periodically to ensure the storage system is meeting the expected performance level. Ensure that a baseline is established and recorded so that comparison can be made.

There are many tools in the market that provide a comprehensive set of features to exercise and measure the storage system and other components in the I/O stack. It is up to administrators to decide the testing requirements and which tools work best for their environment. When choosing tools, consider the capabilities in the following subsections. Several performance testing utilities are shown below:

• I/O subsystem

• dd
• Iozone (fs benchmarking)
• Iometer
• ORION
• winsat
• FIO (fs benchmarking)
• Vdbench
• bonnie

• Relational Database Management System (RDBMS) level

• SLOB
• Oracle Database I/O calibration feature
• DBMS_RESOURCE_MANAGER.CALIBRATE_IO
• dbbenchmark

• Application Level testing tools (database side)

• Benchmark factory
• hammerdb: Supports TPC-C and TPC-H workloads
• Swingbench
• Simora: Mines Oracle SQL Trace files and generates SQL to be run to reproduce the load
• Oracle Real Application Testing: An enterprise database option from Oracle that records a database load on the source system and replays it on a destination environment

• Application Level testing tools (app side)

• HP LoadRunner

Testing the I/O path

The first item to test on a new configuration is the path between the server and the array. Running a large block sequential read test using small files should saturate the path between the server and the array. This test verifies that all paths are fully functional and can be used for I/O traffic. Run this test on a dedicated server and array; using a production system could cause significant performance issues.

To validate the I/O path, run a large block sequential read test using the following guidelines as a starting point and vary as necessary:

• Create one LUN per storage processor
• Format the volumes using a 64 KB allocation unit
• Use a block size of 512 KB for the test
• Configure the test for 32 outstanding I/Os
• Use multiple threads. Eight is the recommended starting point

If the throughput matches the expected throughput for the number of HBA ports in the server, the paths between the server and Dell Unity array are set up correctly.

Testing the drives

Once the I/O path has been validated, the next step is to test the drives. For best results when testing drives on a Dell Unity array, use the following guidelines when configuring the test.

• In a dual-controller system, use at least one volume per controller to ensure that I/O will be distributed across both controllers. Using both controllers more closely simulate real-world activity. For best results, use the same number of volumes on each controller. More LUNs might be better and may be required to achieve maximum performance.
• When performing I/O tests on any storage platform, use files that are larger than the Dell controller cache. For more accurate results, use a file size that matches the amount of data being stored. If using larger files is not practical due to a large dataset, use a file size of at least 100 GB.
• Some I/O test tools (Oracle ORION is an example) generate files full of zeros. This behavior causes inaccurate results when testing. Avoid using test utilities that write zeros for drive validation or configure the tool to avoid writing zeros.

The purpose of this type of testing is to validate that the storage design will provide the required throughput and IOPS with acceptable latency. It is important that the test does not exceed the designed capacity of the array. For example, an array designed for a workload of 5,000 IOPS is likely to perform poorly with a workload of 10,000 IOPS. If a test is generating a workload higher than the designed capacity, adjust the workload being generated by reducing the number of threads, outstanding I/Os, or both.

The results of the Live Optics analysis provide an I/O target to simulate using these tests. To get an idea of the performance capabilities of the array, run I/O tests with a range of I/O sizes commonly seen with Oracle. When testing random I/O, test with I/O sizes of 8 KB, 16 KB, and 32 KB. When testing sequential I/O, test with 8 KB, 16 KB, 32 KB, and 64 KB. Since processes like read ahead scans and backups can issue larger sequential I/O, it is a good idea to also test block sizes larger than 32 KB. To truly test the array the designed workload should be simulated at a minimum, and slightly higher if possible. To ensure the array has headroom for load spikes the throughput should be tested slightly beyond estimated production loads.

I/O simulation

The primary objective of I/O simulation is to stress the storage system. I/O simulation tools are typically easy to use and configure because they do not require a fully configured database. These tools generally allow workloads to increase or decrease during the tests by specifying different parameters:

• I/O block size
• Queue depth (outstanding requests)
• Number of test files and sizes
• Number of I/O threads
• Read/write patterns (read-only, write-only, read/write mix)
• Ability to easily step through a series of tests with various settings (for example, run sets of 8k tests, 16k tests, 32k tests)
• Ability to write random data (nonzero) to the storage
• Ability to generate test reports or export data to other applications (such as Microsoft Excel)

Oracle ORION, Vdbench, and FIO are three IO simulation tools. The software is free to download and use. Oracle ORION has a unique advantage over others because it is explicitly designed to simulate Oracle database I/O workloads using the same I/O software stack as Oracle. It also provides both OLTP and OLAP simulation modes which simplify the setup and execution of the test. ORION has been bundled with the Oracle database software and can be found in the $ORACLE_HOME/bin directory.

Database transaction generation

These tools focus on generating a combination of workloads with different types of database transactions to simulate a typical OLTP, OLAP, or both. They require a higher degree of configuration and customization in the tools and databases. SLOB, HammerDB, Swingbench, and Quest Benchmark Factory are commonly used to perform database transactional I/O benchmarks. Except for Quest Benchmark Factory, all these tools are freely available for multiple operating systems.

Performance monitoring

Performance data can be monitored by the operating system, the Dell Unity system, and in Oracle databases. Ideally, monitoring the performance continuously offers the most detail and allows in-depth analysis of the environments. At a minimum, performance statistics should be captured for at least 24 hours and during the time periods when there are heaviest activities. The following subsections describe popular software and cloud-based platforms for monitoring and analyzing performance.

Operating system monitoring

The following utilities are freely available from the operating system vendors and can perform basic system and I/O monitoring. See the operating system manual and the online resources for each tool to find more information.

• sar (Linux)
• iostat (Linux)
• top (Linux)
• atop and netatop (Linux)
• collectl (Linux)
• Performance Monitor (Microsoft Windows)

Oracle database monitoring

For an Oracle database, the utilities used most are statspack and AWR. AWR is preferred to the older statspack, but either can provide abundant performance statistics of a database. Both utilities come bundled with the database software.

Oracle Enterprise Manager (OEM) is a separate application offered by Oracle. It provides a centralized management and monitoring platform for many Oracle applications and databases. Configuration and performance data are collected through Oracle agents running on an individual host stored in a common management database. OEM provides a plethora of performance and utilization charts and many other advanced features to manage the environment.

Additional information can be found at the Oracle Enterprise Manager page. 

Unisphere performance dashboard

This web-based unified management software comes with every Dell Unity storage system and can manage every aspect of the storage. The performance dashboard in Dell Unisphere provides both real-time and historical performance charts. Administrators can easily modify existing dashboards and charts or add new dashboards and charts according to their needs.

Metric data ages over time and gets aggregated into longer sampling intervals. The data is kept for historical referencing for up to 90 days.

Additional information can be found in the Dell Unity: Unisphere Overview.

Dell Live Optics

Dell’s Performance Analysis Collection Kit (DPACK) has evolved into a new product called Dell Live Optics and is a platform-agnostic analysis service freely available from Dell Technologies. It works on Linux, Microsoft Windows, and VMware environments and collects performance data such as processor utilization, memory utilization, storage utilization, IOPS, I/O throughput, and more. Live Optics analyzes these data and provides a comprehensive in-depth report on server workloads and capacity.

Find additional information at the Live Optics for Service Providers page with a download available at Live Optics - Real-world data for IT decisions : Live Optics.

Dell CloudIQ

Dell CloudIQ is a software as a service (SaaS) application that is freely available. When it is enabled for the Dell Unity storage system, it allows administrators to monitor multiple Dell Unity storage systems remotely. CloudIQ provides continuous monitoring of performance, capacity, configuration, and data protection, and enables administrators to manage storage proactively by receiving advanced notification for potential issues.

Find additional information, see CloudIQ: A Detailed Review | Dell Technologies Info Hub.

This section discusses best practices for architecture and configuration of Oracle storage for Oracle databases to realize the optimal performance and manageability of the environment.

Starting with Oracle 21c, after the Oracle installation, ORACLE_HOME is configured as read-only. With a read-only ORACLE_HOME, all user- and instance-specific files, log files, trace files reside in a location known as ORACLE_BASE_HOME.

With a read-only ORACLE_HOME, ORACLE_BASE_HOME is set to:

$ORACLE_BASE/homes/<HOME_NAME from inventory.xml>

Performing the following will return the value of ORACLE_BASE_HOME:

[oracle@r730xd-1 ~]$ cd $ORACLE_HOME/bin

[oracle@r730xd-1 bin]$ ./orabasehome

/u01/app/oraclel/homes/OraDB21Home1

[oracle@r730xd-1 bin]$

For more information, see Oracle 21c Database Installation Guide.

Oracle databases are commonly deployed on Linux operating systems. The following subsections describe best practices when working with Dell Unity storage systems on Linux operating systems.

Discovering and identifying Dell Unity LUNs on a host

After creating and enabling host access of the LUNs in the Dell Unity system, the host operating system needs to scan for these LUNs before they can be used. On Linux, install the following rpm packages which contain useful utilities to discover and identify LUNs: sg3_utils and lsscsi.

Identifying LUN IDs on Dell Unity storage

When creating a LUN, if the Host LUN ID is not specified, Dell Unity automatically assigns LUN IDs, starting from 0 and incrementing by 1 thereafter. LUN 0 typically represents the first LUN allowed access to a host.

Perform the following in Unisphere to view the LUN ID information:

1. In Unisphere’s navigation menu under ACCESS, click Hosts.

2. Click the checkbox associated with the name of the host.

3. Click Host Properties (pencil icon) > LUNs tab.
4. If column Host LUN ID is hidden from the default view, click the columns filter (gear icon) > Columns > Host LUN ID.

Scanning for LUNs

LUNZ is known as a host LUN and is not a real LUN. LUNZ appears on the Linux operating system for making the Dell Unity system visible to the host when no LUNs are assigned to the host. LUNZ always takes up ID 0 on the host. Any I/Os sent down the LUNZ paths result in errors.

To see if there are any LUNZ on the system, run the lsscsi command.

# lsscsi|egrep DGC

[13:0:0:0]   disk    DGC      LUNZ            4201  /dev/sde

[13:0:1:0]   disk    DGC      LUNZ            4201  /dev/sdaf

[14:0:0:0]   disk    DGC      LUNZ            4201  /dev/sdd

[14:0:1:0]   disk    DGC      LUNZ            4201  /dev/sdag

[snipped]

If the host detects LUN ID 0 and LUNZ, run rescan-scsi-bus.sh with the --forcerescan option. LUNZ will be removed and allows the real LUN 0 to show up on the host. For example:

# /usr/bin/rescan-scsi-bus.sh --forcerescan

# lsscsi |egrep DGC

[13:0:0:0]   disk    DGC      VRAID            4201  /dev/sde

[13:0:1:0]   disk    DGC      VRAID            4201  /dev/sdaf

[14:0:0:0]   disk    DGC      VRAID            4201  /dev/sdd

[14:0:1:0]   disk    DGC      VRAID            4201  /dev/sdag

[snipped]

When the Dell Unity LUNs have nonzero IDs, use -a option instead.

# /usr/bin/rescan-scsi-bus.sh -a

Note: Omitting the --forcerescan option might prevent the operating system from discovering LUN 0 because of the LUNZ conflict.

Identifying LUNs by WWNs

The most accurate way to identify a LUN on the host operating system is by its WWN. The Dell Unity system assigns a unique WWN for each LUN. WWN information can be found using the same steps in Identifying LUN IDs on Dell Unity storage. However, instead of using column HOST LUN ID, use WWN.

Querying WWNs using scsi_id command

To query the WWN on a Linux operating system, run the following commands against the device file.

# /usr/lib/udev/scsi_id --page=0x83 --whitelisted --device=<device>

Where <device> can be one of the following:

• Single path device (/dev/sde)
• Linux multipath device (/dev/mapper/mpathe)
• Dell PowerPath device (/dev/emcpowerc)

The string returned by the scsi_id command indicates the WWN of the Dell Unity LUN, prepended with a 3:

36006016010e0420093a88859586140a5

Querying WWNs using multipath command

If the system has Linux device-mapper-multipath software enabled, the multipath command displays the multipath device properties including the WWN. For example:

Figure 2.        multipath -ll

Querying WWNs using powermt command

Similarly, when PowerPath is enabled on the system, the powermt command displays the multipath device properties including the WWN.

# powermt display dev=all

Multipathing

Multipathing is a software solution implemented at the host operating system level. While multipathing is optional, it provides path redundancy, failover, and performance-enhancing capabilities. It is recommended to deploy the solution in a production environment or any environments where availability and performance are critical.  

Main benefits of using an MPIO solution

• Increase database availability by providing automatic path failover and failback
• Enhance database I/O performance by providing automatic load balancing and capabilities for multiple parallel I/O paths
• Ease administration by providing persistent user-friendly names for the storage devices across cluster nodes

Multipath software solutions

There are a few multipath software choices available to choose from. It is up to the administrator's preference to decide which software solution is best for the environment. The following list provides a brief description of some of these solutions.

• Native Linux multipath (device-mapper-multipath)
• Dell PowerPath
• Symantec Veritas Dynamic Multipathing (VxDMP)

The native Linux multipath solution is supported and bundled with most popular Linux distributions in use today. Because the software is widely and readily available at no additional cost, many administrators prefer using it compared to other third-party solutions.

Unlike the native Linux multipath solution, both Dell PowerPath and Symantec VxDMP provide extended capabilities for some storage platforms and software integrations. Both solutions also offer support for numerous operating systems in addition to Linux.

Only one multipath software solution should be enabled on the host, and the same solution should be deployed in a cluster on all cluster hosts.

See the vendor's multipath solution documentation for more information. For information about operating systems supported by Dell PowerPath, see Support for PowerPath | Overview | Dell US.

Connectivity guidelines

The following list provides a summary of array-to-host connectivity best practices. It is recommended to review the documents, Configuring Hosts to Access Fibre Channel (FC) or iSCSI Storage and Dell Unity: High Availability.

• Have at least two FC/iSCSI HBAs or ports to provide path redundancy.
• Connect the same port on both Dell Unity storage processors (SP) to the same switch because the Dell Unity system matches the physical port assignment on both SPs.
• Use multiple switches to provide switch redundancy.

Configuration file

To ease deployment of the native Linux multipath, it comes with a set of default settings. The default settings list storage models from different vendors including the Dell Unity system. The default settings allow the software to work with the Dell Unity system without additional configuration. However, these settings might not be optimal for all situations and should be reviewed and modified if necessary.

The multipath daemon configuration file needs to be created on newly installed systems. A basic template can be copied from /usr/share/doc/device-mapper-multipath-<version>/multipath.conf to /etc/multipath.conf as a starting point. Any settings that are not defined explicitly in /etc/multipath.conf would assume the default values. The full list of settings (explicitly set and default values) can be obtained using the following command. Specific Dell Unity settings can be found by searching for DGC from the output. The default settings generally work without any issues.

# multipathd –k"show config"

Creating aliases

It is generally a good idea to assign meaningful names (aliases) for the multipath devices though it is not mandatory. For example, create aliases based on the application type and environment it is in. The following snippet in the multipaths section assigns an alias of ORA-DATA-00 to the Dell Unity LUN with WWN 36006016010e04200271a8a594a34d845.

Figure 3.        Multipath device configuration

Asymmetric Logic Unit Access

Dell Unity systems support ALUA for host access. ALUA allows the host operating system to recognize optimized paths from nonoptimized paths. Optimized paths are the ones connected to the LUN’s SP owner, and they are assigned a higher priority. The default multipath settings reflect the support of the ALUA feature on Dell Unity storage. The following example shows LUN ORA-DATA-00 with 12 paths divided into two groups. The optimized paths have a priority of 50, and nonoptimized paths have a priority of 10.

Figure 4.        Optimized and nonoptimized multipath paths

I/Os are sent down the optimized paths when possible. If I/Os are sent down the nonoptimized paths, the peer SP redirects the I/Os to the primary SP through the internal bus. When the Dell Unity system senses large amounts of nonoptimized I/Os, it automatically trespasses the LUN from the primary SP to the peer SP to optimize the data paths.

LUN partition

A LUN can be used as a whole partition or it can be divided into multiple partitions. Certain applications, such as Oracle ASMLib, recommend partitioning over whole LUNs. Dell Technologies recommends configuring whole LUNs without partitions wherever appropriate because it offers the most flexibility for configuring and managing the underlying storage.

See sections Oracle Automatic Storage Management and File systems on choosing a strategy to grow storage space.

Partition alignment

When partitioning a LUN, it is recommended to align the partition on the 1 MB boundary. Either fdisk or parted can be used to create the partition. Only parted can create 2 TB or larger partitions.

Creating partition using parted

Before creating a partition, label the device as GPT. Then, specify the partition offset at sector 2048, which will be a 1 MB offset from the beginning of the device. The following command labels device /dev/mapper/orabin-std and creates a single partition that consumes the entire LUN. Once the partition is created, the partition file, in this example /dev/mapper/orabin-std1, should be used for creating a file system or ASMLib volume.

parted --script /dev/mapper/orabin-std mklabel gpt mkpart primary 2048s 100% quit ;

Note: parted might issue a misalignment warning which can be safely ignored.

Partitioned devices and filesystems

When creating a file system, create the file system on a properly aligned partitioned device.

# mkfs.ext4 /dev/mapper/orabin-std1

I/O scheduler for Oracle ASM devices

The function of the disk I/O scheduler is to order, delay, or merge I/O requests submitted to a storage device to achieve performance improvements with throughput and latency. There are several different I/O schedulers that can be used in Linux. Each provides a different method for managing I/O requests. The recommended I/O scheduler will depend on several factors. Some factors are: Operating system version, UEK version, Oracle version, storage media used in the storage array, and type of database processing.

Oracle Linux 8 has four different I/O schedulers:

none: Implements a first-in-first-out (FIFO) scheduling algorithm. When high-performance SSDs are used, or when a CPU-bound system with fast storage exists, the 'none' scheduler is recommended. Scheduler ‘none’ is the default I/O scheduler.

mq-deadline: Sorts queued I/O requests into read or write batches. Batches are then performed in increasing logical block address (LBA) order so that writes are organized by location to storage. mq-deadline is suitable for most use cases, but is suited best for traditional hard drive storage with a SCSI interface, and where write operations are mostly asynchronous.

kyber: Tunes itself by analyzing each I/O request to obtain the least amount of latency for each request. kyber is recommended, as an alternative to the 'none' scheduler, for high-performance SSDs or CPU-bound systems with fast storage.

bfq: Prioritizes on providing lowest latency rather than maximum throughput. bfq is recommended for copying large files, systems with traditional hard drive with a SCSI interface, and desktop or interactive activities.

For additional use cases, see Red Hat Enterprise Linux 8, Optimizing system throughput, latency, and power consumption.

Earlier Linux versions implemented I/O schedules: noop, deadline, and cfq.

Before changing the default I/O scheduler in production, review the latest Oracle documentation and confer with Oracle support for recommended values. Then change the scheduler in a test environment, simulate production activity, and monitor performance before changing the value in production.

To verify the I/O schedule, use the following commands:

# egrep "*" /sys/block/sd*/queue/scheduler

/sys/block/sdaa/queue/scheduler:[none] mq-deadline kyber bfq

/sys/block/sdab/queue/scheduler:[none] mq-deadline kyber bfq

/sys/block/sdac/queue/scheduler:[none] mq-deadline kyber bfq

<snippet>

To set the I/O schedule persistently, create an udev rule that updates the devices. See the section Linux dynamic device management (udev) for more information about using udev to set persistent ownership and permission.

The following example shows setting the deadline I/O scheduler on all /dev/sd* devices. The rule is appended to the 99-oracle-asmdevices.rule file.

# cat /etc/udev/rules.d/99-oracle-asmdevices.rules

ACTION=="add|change", KERNEL=="sd*", RUN+="/bin/sh -c '/bin/echo mq-deadline > /sys$env{DEVPATH}/queue/scheduler'"

# udevadm control --reload-rules

# udevadm trigger

Dell Technologies and Oracle recommend using Oracle Automatic Storage Management (ASM) to manage Dell Unity LUNs for the database and clusterware. This section reviews the general guidelines and additional considerations for an Oracle database.

Preparing storage for Oracle ASM

LUNs intended for Oracle ASM must have their user and group ownership and permissions set correctly. Group and ownership must be set to the group and owner of the ASM instance. Permissions must be set to read/write. For example, if user grid with primary group oinstall is the owner of the ASM instance, grid:oinstall should be assigned to the LUNs. There are different methods to set the ownership and permissions and keep these settings persistent across host reboot.

Persistent device ownership and permissions

Persistent device ownership and permission can be managed through various software. Some of the commonly used software is:

• Linux dynamic device management (udev)
• Oracle ASMLib
• Oracle ASMFD

Linux dynamic device management (udev)

The Linux udev facility comes with every Linux distribution and is easy to set up for persistent device ownership and permission by creating rules in the udev rule file. System rule files are in directory /usr/lib/udev/rules.d, and user-defined rule files are in directory /etc/udev/rules.d. There are many ways to define a device in the rule file. Two examples are provided as follows.

Example 1: Set device ownership and permission by WWNs

Define a rule for each Dell Unity LUN using its unique WWN. With this approach, each LUN requires an udev rule. Rules are defined in /etc/udev/rules.d/99-oracle-asmdevices.rules. The following example shows an udev rule that sets grid:oinstall ownership and 660 permission on a dm (multipath) device that matches the WWN 36006016010d04200b584ce59557ba84a.

# cat /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL=="dm-*",PROGRAM=="/lib/udev/scsi_id --whitelisted --device=/dev/$name",RESULT=="36006016010d04200b584ce59557ba84a",ACTION=="add|change",OWNER="grid",GROUP="oinstall",MODE="0660"

<snippet>

# udevadm control --reload-rules

# udevadm trigger

If PowerPath is used, change KERNEL=="dm-*" to KERNEL=="emcpower*".

Example 2: Set device ownership and permission by device name pattern

The udev rule can be simplified if multipath device aliases are created with a consistent string pattern. For example, use prefix ORA- in all multipath device aliases for LUNs intended for Oracle ASM. A single udev rule can be used to set ownership and permission on all ORA* multipath devices. See section Creating aliases.

# cat /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL=="dm-*",ENV{DM_NAME}=="ORA*",OWNER="grid",GROUP="oinstall",MODE="0660"

# udevadm control --reload-rules

# udevadm trigger

The advantage of this approach is that only multipath.conf needs to be updated when new LUNs are added to the system for Oracle ASM.

Oracle ASMLib

Oracle ASMLib simplifies storage management and reduces kernel resource usage. It provides device file name, ownership, and permission persistency and reduces the number of open file handles required by the database processes. No udev is required when ASMLib is used.

When LUNs are initialized with ASMLib, special device files are created in the /dev/oracleasm/disks folder with proper ownership and permission automatically. When the system reboots, the ASMLib driver restarts and re-creates the device files. ASMLib consists of three packages:

• oracleasm-support-version.arch.rpm
• oracleasm-kernel-version.arch.rpm
• oracleasmlib-version.arch.rpm

Each Linux vendor maintains their oracleasm kernel driver (oracleasm-kernel-version.arch.rpm). With Oracle Linux, the kernel driver is already included with Oracle Linux Unbreakable Enterprise Kernel. For more information about ASMLib and to download the software, go to http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html.

The ownership of the ASMLib device's is defined in configuration file /etc/sysconfig/oracleasm which is generated by running:

/etc/init.d/oracleasm configure

If necessary, update file /etc/sysconfig/oracleasm to reflect proper ownership and the disk scanning order.

# cat/etc/sysconfig/oracleasm

# ORACLEASM_ENABLED: 'true' means to load the driver on boot.

ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.

ORACLEASM_UID=grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.

ORACLEASM_GID=oinstall

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.

ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER="dm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE="sd"

# If the physical block size of block devices is 4K, then

# uncomment the following line:

#ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true

# However, if the physical block size is 512, then

# uncomment the following line:

#ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false

This configuration file indicates grid:oinstall for the ownership, and it searches for multipath devices (dm) and excludes any single path devices (sd).

If PowerPath devices are used, set ORACLEASM_SCANORDER to “emcpower” rather than to “dm”:

ORACLEASM_SCANORDER="emcpower"

Note: The asterisk (*) cannot be used in the value for ORACLEASM_SCANORDER and ORACLEASM_SCANEXCLUDE.

Oracle requires the LUNs to be partitioned for ASMLib use. First, create a partition with parted, and then use oracleasm to label the partition. ASMLib does not provide multipath capability and relies on native or third-party multipath software to provide the function. The following example shows creating an ASMLib device on a partition of a Linux Multipath device. Command oracleasm writes the ASMLib header to /dev/mapper/mpathap1 and generates the ASMLib device file in /dev/oracleasm/disks/DATA01 with ownership as indicated in /etc/sysconfig/oracleasm.

# oracleasm createdisk DATA01 /dev/mapper/mpathap1

Oracle ASM Filter Driver

Oracle ASM Filter Driver (ASMFD) is a kernel module that sits between the operating system kernel and Oracle ASM. Oracle intends to replace Oracle ASMLib with ASMFD and recommends using ASMFD in Oracle 12c and above. ASMFD includes all the ASMLib benefits of storage device name, ownership, and permission persistency, and better kernel usage by reducing the number of open file handles. Also, it provides storage protection by rejecting non-oracle I/Os and hence prevents inadvertent overwrite of the ASM disks.

In a cluster environment, without ASMFD, when a cluster node is fenced, the host must be rebooted to ensure the integrity of the data. With ASMFD, the fenced node does not need to be rebooted. It is possible to restart the clusterware stack which reduces the time to recover the node.

Unlike ASMLib, ASMFD comes with the Grid Infrastructure software and there is no additional software to download. Starting with Oracle ASM 12c Release 2, the installation and configuration for Oracle ASMFD have been simplified by integrating the option into the Oracle Grid Infrastructure installation. Administrators need to select the option Configure Oracle ASM Filter Driver during the Grid installation.

The installation of ASMFD automatically creates an udev rule file in /etc/udev/rules.d/53-afd.rules that sets the afd devices with the proper ownership and permission. Do not attempt to modify or delete this file directly. Use command asmcmd adf_configure to make updates instead.

# cat /etc/udev/rules.d/53-afd.rules

#

# AFD devices

KERNEL=="oracleafd/.*", OWNER="grid", GROUP="oinstall", MODE="0775"

KERNEL=="oracleafd/*", OWNER="grid", GROUP="oinstall", MODE="0775"

KERNEL=="oracleafd/disks/*", OWNER="grid", GROUP="oinstall", MODE="0664"

Either whole LUNs or LUN partitions can be used for ASMFD devices. Dell Technologies recommends using whole LUNs because of certain restrictions with partitions which affect database availability during storage expansion. See the section Expand Oracle ASM storage for more details.

The following example shows creating an ASMFD device on a Linux multipath device. The asmcmd afd_label command writes the ASMFD header to /dev/mapper/mpathb and generates the ASMFD device file in /dev/oracleafd/disks/DATA01. The udev rule ensures the afd devices are set to grid:oinstall and 0664 permission.

# asmcmd afd_label DATA01 /dev/mapper/mpatha

The other advantage of using ASMFD is that it supports thin-provisioned disk group starting in Oracle release 12.2.0.1.

To find out which operating system platforms ASMFD is supported on, see Oracle KB Doc ID 2034681.1 at Oracle Support.

For more information about installing and configuring ASMFD, refer to the Oracle Automatic Storage Management Administrator’s Guide.

Setting the asm_diskstring ASM instance parameter

ASM instance parameter asm_diskstring tells ASM the location of the ASM devices. During the Grid Infrastructure installation, it defaults to null and it should be updated to reflect the correct location of the device files.

Table 4.      Example of asm_diskstring setting

Oracle ASM guidelines

Dell Technologies and Oracle recommend using Oracle ASM as the preferred storage management solution for either single-instance or Real Application Clusters (RAC) databases. ASM takes the place of the traditional Linux Logical Volume Manager (LVM) and file system and manages the disks and disk groups where Oracle datafiles reside.

Benefits of using Oracle ASM

ASM offers many advantages over the traditional Linux LVM. The main benefits include:

• Automatic Oracle file management
• Online Oracle data files rebalance across ASM disks
• Online addition and removal of ASM disks without downtime
• Single solution for both volume and file management integrated with Oracle software
• Improved I/O performance because ASM stripes all files across all disks in a disk group
• Transparent integration with Dell Unity system features such as snapshots, thin-provisioning, thin clones, compression, and Data at Rest Encryption

ASM disk and disk group guidelines

When creating an Oracle ASM disk group, consider the following guidelines:

• For ultimate flexibility and maintaining configuration consistency, create separate disk groups for each of the following:

• Create a disk group for the Oracle Cluster Registry (OCR) and voting files.
• Create a disk group for Grid Infrastructure Management Repository (GIMR).
• Use one or more disk groups for database datafiles for each database.
• Use a disk group for a fast recovery area for each database.
• Configure a database which can span across multiple and exclusive disk groups to optimize the storage and snapshot configuration for each individual database.

• Create LUNs for an ASM disk group with the same capacity and services, such as compression, consistency group, and snapshot schedule.
• Use fewer but larger LUNs to reduce the number of objects to be managed.
• Create a minimum of two LUNs for each disk group. Distribute the LUNs evenly on both Dell Unity storage processors to allow even I/O distribution to both processors, to maximize the performance and I/O bandwidth for the environment.
• To take an array-based snapshot on a multivolume Oracle database, ensure that all LUNs belonging to the same database are snapped together. To snap the LUNs together, group the LUNs in a Consistency Group (see more information in section Consistency group).
• While ASM can provide software-level mirroring, it is not necessary because data protection is integrated in Dell Unity RAID protection. Use External Redundancy for ASM disk groups to enable substantial storage savings, reduce overall IOPS from ASM, and results in better I/O performance.
• When creating datafiles on ASM disk groups, set an initial size of each datafile and specify the autoextend clause to include an extent size for growth. Dell Unity system allocates storage for the initial datafile size and as data are written to datafiles. When needed, more space is allocated in the amount of autoextend size. An example of the Oracle SQL*Plus CREATE TABLESPACE statement is shown in the following:

create tablespace DATATS

   datafile '+DATADG'

   size 10G

   autoextend on

   next 1024M

   maxsize unlimited;

• By default, each LUN has unlimited I/O limits in the Dell Unity system. When a database requires higher performance and another does not, consider creating different host I/O limit policies in Dell Unisphere that limit I/O performance based on IOPS and bandwidth. Assign the policy to the LUNs corresponding to the level of performance required. The host I/O limit is applied on the LUN level.
• Starting with Oracle 12c releases, ASMFD supports thin provision ASM disk group. The feature allows unused space to be released back to the Dell Unity system after deleting or shrinking the datafiles. To enable the feature, set the THIN_PROVISIONED attribute to 'TRUE' on the Oracle SQL*Plus ALTER DISKGROUP command. For example:

ALTER DISKGROUP DATADG

   SET ATTRIBUTE 'THIN_PROVISIONED'='TRUE';

• When ASM rebalances the disk group, data is moved to higher performing tracks of spinning disks during the compact phase. Since the Dell Unity system virtualizes the physical storage devices, and with the use of the flash devices, there is no real benefit to compacting the data. Starting with Oracle 12c, it is possible to disable the compact phase on individual disk group by setting the _rebalance_compact attribute of Oracle SQL*Plus command ALTER DISKGROUP to 'FALSE'.

ALTER DISKGROUP DATADG

   SET ATTRIBUTE '_rebalance_compact'='FALSE';

For more information about ASM compact phase rebalancing, see Oracle KB Doc ID 1902001.1 on Oracle Support.

Table 5 demonstrates an example of how ASM disk groups are organized. Figure 5 illustrates the storage layout on the database, ASM disk group, and Dell Unity system levels.

Table 5.      Example ASM disk group configuration

Figure 5.        Oracle ASM storage layout on the Dell Unity System

Consistency group

For performance reasons, it is common for a database to span multiple LUNs to increase I/O parallelism to the storage devices. Dell Technologies recommends grouping the LUNs into a consistency group for a database to ensure data consistency when taking storage snapshots. The Dell Unity system snapshot feature is a quick and space-efficient way to create a point-in-time snapshot of the entire database. Section Data protection and recovery discusses using Dell Unity system snapshots and thin clones to reduce database recovery time and create space-efficient copies of the database.

In Figure 5, for example, the RAC database consists of disk group +DATADG and +FRADG. All ASM volumes in those disk groups are configured in a single consistency group, testdb_cg. The single instance database consists of disk groups +DATA2DG and +FRA2DG. The ASM devices of both disk groups are configured in a consistency group, devdb_cg.

The consistency group feature allows taking a database-consistent snapshot across multiple LUNs. On the database side, use the ALTER DATABASE BEGIN BACKUP clause before the snapshot is taken and END BACKUP clause after the snapshot is taken.

Note: Storage snapshots taken on a multiple-LUN database without a consistency group might be irrecoverable by Oracle during database recovery.

Expand Oracle ASM storage

As the storage consumption grows over time, it is necessary to increase and grow the existing storage capacity both in the Dell Unity system and in the database. It is most desirable to add capacity online with minimal business interruptions. The Dell Unity system has the flexibility to expand the current storage system with no interruption to the application. The following nondisruptive operations can be performed online in Unisphere:

• Adding flash devices
• Expanding the storage pool
• Increasing the size of existing LUNs
• Creating and adding new LUNs to existing hosts

The following subsections discuss the different ways to increase ASM storage capacity.

Increase Oracle ASM storage by adding new LUNs

More storage capacity can be added to an ASM disk group by adding new LUNs to the disk group. The advantage of this method is that the process is relatively simple and safe because no changes are made to the existing LUNs.

The following outlines the general process:

1. Create LUNs in Unisphere.

2. Ensure the size of new LUNs and other features such as compression, and that the consistency group matches the existing LUNs.
3. Allow access to new LUNs to the host systems.
4. Perform a SCSI scan on the host system (see section Discovering and identifying Dell Unity LUNs on a host).
5. Configure multipath for the new devices (see section Multipathing).
6. Prepare the LUNs for ASM (see section Preparing storage for Oracle ASM).
7. Add the LUNs to the ASM disk group.

Since ASM automatically rebalances the data after new LUNs are added, it is recommended to add the LUNs in a single operation to minimize the amount of rebalancing work. The following example shows the SQL*Plus ALTER DISKGROUP ADD DISK statement to add multiple devices to a disk group.

ALTER DISKGROUP DATADG

   ADD DISK 'AFD:DATADG_VOL1',

            'AFD:DATADG_VOL2'

   REBALANCE POWER 10

   NOWAIT;

8. Verify the status and capacity of the disk group with the Linux ASM command asmcmd.

asmcmd lsdsk -gk -G datadg

asmcmd lsdg –g datadg

9. If the existing LUNs are in a consistency group, add the new LUNs to the same consistency group.

Note: Adding or removing LUNs in a consistency group is not allowed when there are existing snapshots of the consistency group. To add or remove LUNs in a consistency group, delete all snapshots and retry the operation.

Increase Oracle ASM storage by resizing current LUNs

The Dell Unity system can extend the size of existing LUNs online. However, depending on the operating system, disk partition configuration, and Oracle software chosen, resizing ASM disks online might not be possible. Customers should consult with each vendor to fully understand the capability and limitations of their software.

Note: Resizing LUNs on the operating system can cause loss of data or corruption. It is recommended to back up all data before attempting to resize the LUNs.

Resize ASM devices without ASMFD and ASMLib online

When not using ASMFD or ASMLib, and only using whole LUNs, it is possible to resize the devices online on a wide range of operating systems and Oracle versions. Customers should consult with each vendor to fully understand the capability and limitations of their software.

The following outlines the general steps to resize ASM devices online when ASMFD or ASMLib are not used to manage the ASM devices.

1. Take manual snapshots of LUNs that are going to be expanded. See the section Data protection and recovery for more information about taking snapshots and recovering from snapshots.

2. Expand the size of existing LUNs in Unisphere.
3. Perform a SCSI scan on the host system and refresh the partition table on each LUN path and reload multipath devices.
4. In Linux, if dm-multipath devices are used, reload the multipath devices

multipathd -k"resize map DATA03"

If PowerPath is used instead of dm-multipath, the new size is automatically updated.

5. Verify the new LUN size.
   1. If dm-multipath is used, use the following Linux commands:

# multipath -ll ORA-TEST3 | egrep size

size=400G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw

# multipath -ll ORA-TEST3 | awk '/sd/ {print $(NF-4)}' | xargs -i fdisk -l /dev/{} | egrep "^Disk"

Disk /dev/sdfd: 429.5 GB, 429496729600 bytes, 838860800 sectors

Disk /dev/sdff: 429.5 GB, 429496729600 bytes, 838860800 sectors

Disk /dev/sdfh: 429.5 GB, 429496729600 bytes, 838860800 sectors

[snipped]

2. If PowerPath, use the following Linux commands:

# fdisk –l /dev/emcpowerc

# powermt display dev=all|awk '/sd/ {print $3}'|xargs -i fdisk -l /dev/{}|egrep "^Disk"

6. To determine the maximum size of the LUN, run asmcmd lsdsk and extract the OS_MB value. Use this value with the ALTER DISKGROUP RESIZE DISK clause.

# asmcmd lsdsk –k

Inst_ID  Total_MB  Free_MB   OS_MB  Name          Failgroup     Site_Name  Site_GUID                         Site_Status  Failgroup_Type  Library  Label  Failgroup_Label  Site_Label  UDID  Product  Redund   Path

      1    204800   409464  409600  TEST3DG_0000  TEST3DG_0000             00000000000000000000000000000000               REGULAR         System                                                      UNKNOWN  /dev/mapper/ORA-TEST3

      2    204800   409464  409600  TEST3DG_0000  TEST3DG_0000             00000000000000000000000000000000               REGULAR         System                                                      UNKNOWN  /dev/mapper/ORA-TEST3

Total_MB represents the current size before the resize operation.

OS_MB represents the new maximum size ASM can expand to.

7. Using SQL*Plus, resize the ASM device.

ALTER DISKGROUP TEST3DG

   RESIZE DISK TEST3DG_0000

   SIZE 409600M

   REBALANCE POWER 10;

8. Verify the new ASM device size. After the resize operation completes, run asmcmd lsdsk to confirm the Total_MB value matches the OS_MB value.

# asmcmd lsdsk -k

Inst_ID  Total_MB  Free_MB   OS_MB  Name          Failgroup     Site_Name  Site_GUID                         Site_Status  Failgroup_Type  Library  Label  Failgroup_Label  Site_Label  UDID  Product  Redund   Path

      1    409600   409464  409600  TEST3DG_0000  TEST3DG_0000             00000000000000000000000000000000               REGULAR         System                                                      UNKNOWN  /dev/mapper/ORA-TEST3

      2    409600   409464  409600  TEST3DG_0000  TEST3DG_0000             00000000000000000000000000000000               REGULAR         System                                                      UNKNOWN  /dev/mapper/ORA-TEST3

Run asmcmd lsdg to confirm the Total_MB value on the disk group has increased.

# asmcmd lsdg

Inst_ID  State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

      1  MOUNTED  EXTERN  N         512             512   4096  4194304    409600   409464                0          409464              0             N  TEST3DG/

      2  MOUNTED  EXTERN  N         512             512   4096  4194304    409600   409464                0          409464              0             N  TEST3DG/

Resize ASM devices with ASMFD

When Oracle 12.2 and newer ASMFD is used with ASM devices, it is possible to resize the ASM device online without impacting the database. For Oracle version 12.1, the option afd_refresh is not available.

Resize ASM devices online:

The following outlines the general steps to resize ASM devices with ASMFD online.

1. Take manual snapshots of LUNs that are going to be expanded. See the section Data protection and recovery for more information about taking snapshots and recovering from snapshots.

2. Expand the size of existing LUNs in Unisphere.
3. Perform a SCSI scan on the host system and refresh the partition table on each LUN path and reload multipath devices.
4. In Linux, if dm-multipath devices are used, reload the multipath devices

multipathd -k"resize map ORA-TEST3"

If PowerPath is used instead of dm-multipath, the new size is automatically updated.

5. Verify the new LUN size.
   1. If dm-multipath is used, use the following Linux commands:

# multipath -ll ORA-TEST3 | egrep size

# multipath -ll ORA-TEST3 | awk '/sd/ {print $(NF-4)}' | xargs -i fdisk -l /dev/{} | egrep "^Disk"

2. If PowerPath is used, use the following Linux commands:

# fdisk –l /dev/emcpowerc

# powermt display dev=all|awk '/sd/ {print $3}'|xargs -i fdisk -l /dev/{}|egrep "^Disk"

6. Refresh the ASMFD devices.

# asmcmd afd_refresh

7. To determine the maximum size of the LUN, run asmcmd lsdsk and extract the OS_MB value. Use this value with the ALTER DISKGROUP RESIZE DISK clause.

# asmcmd lsdsk –k

8.  Using SQL*Plus, resize the ASM device.

ALTER DISKGROUP DATADG

   RESIZE DISK DATA03

   SIZE $OS_MB

   REBALANCE POWER 10;

9. Verify the new size in the ASM device and disk group. After the resize operation completes, run asmcmd lsdsk to confirm the Total_MB value matches the OS_MB value.

# asmcmd lsdsk -k

Run asmcmd lsdg to confirm the Total_MB value on the disk group has increased.

# asmcmd lsdg

Resize ASM devices offline:

As mentioned previously in this section, the online resize capability of ASMFD is available in Oracle 12.2. With Oracle 12.1, either restart the host to refresh the LUN size, or restart the clusterware, ASM instance, and the AFD driver on the host to minimize the outage window. In a cluster environment, refreshing the LUN size can be done in a rolling fashion to further minimize the impact of the outage.

1. Follow step 1 to step 5 in section Resize ASM devices without ASMFD and ASMLib online.

2. Stop all databases on the host.
3. Stop CRS.

# crsctl stop crs

4. Reload the AFD driver.

# afdload stop

# afdload start

5. Rescan the AFD devices.

# asmcmd afd_scan

6. Restart CRS.

# crsctl start crs

7. Restart the databases.
8. Continue with step 6 to step 8 in section Resize ASM devices without ASMFD and ASMLib online.
9. Repeat the process on other cluster nodes.

Another alternative to restarting the node or software is to unlabel and label the AFD devices. The database associated with the devices must be stopped, and the disk groups and devices must be unmounted before they can be relabeled. This approach increases the risk of data loss and corruption and requires extra caution.

Resize an ASM device with ASMLib

ASMLib required partitioned LUNs. To increase the size of the partition after expanding the LUN, the partition is first removed and then re-created with the new size. The database associated with the device will be impacted.

The following outlines the general steps to resize ASM devices with ASMLib online.

1. Take manual snapshots of LUNs that are going to be expanded. See the section Data protection and recovery for more information about taking snapshots and recovering from snapshots.

2. Expand the size of the existing LUNs in Unisphere.
3. Perform a SCSI scan on the host system, refresh the partition table on each LUN path, and reload the multipath devices.

# rescan-scsi-bus.sh --resize

4. In Linux, if dm-multipath devices are used, reload the multipath devices

multipathd -k"resize map TEST67_ASMLIB"

If PowerPath is used instead of dm-multipath, the new size is automatically updated.

5. Verify the new LUN size.
   1. If dm-multipath is used, use the following Linux commands:

# multipath -ll TEST67_ASMLIB | egrep size

# multipath -ll TEST67_ASMLIB | awk '/sd/ {print $(NF-4)}' | xargs -i fdisk -l /dev/{} | egrep "^Disk"

2. If PowerPath is used, use the following Linux commands:

# fdisk –l /dev/emcpowerc

# powermt display dev=all|awk '/sd/ {print $3}'|xargs -i fdisk -l /dev/{}|egrep "^Disk"

6. Stop the database across the cluster.

$ srvctl stop db -d demodb

7. Use SQL*Plus to dismount the disk group on all cluster nodes.

ALTER DISKGROUP TEST67_ASMLIBDG DISMOUNT;

8. In Linux, remove and re-create the partition.

parted /dev/mapper/TEST67_ASMLIB rm 1

parted /dev/mapper/TEST67_ASMLIB mkpart primary 2048s 100%

9. In Linux, rescan the LUNs on all cluster nodes.

rescan-scsi-bus.sh –resize

10. In Linux, if using dm-multipath, refresh the multipath device on all cluster nodes.

multipathd -k"resize map TEST67_ASMLIB"

If PowerPath is used instead of dm-multipath, the new size is automatically updated.

11. Update the partition on all cluster nodes.

partprobe /dev/mapper/TEST67_ASMLIB

parted /dev/mapper/TEST67_ASMLIB u GB p

parted command should show the new size.

12. Rescan and list the ASMLib devices on all cluster nodes.

oracleasm scandisks

oracleasm listdisks

13. Using SQL*Plus to mount the disk group on all cluster nodes.

ALTER DISKGROUP TEST67_ASMLIBDG MOUNT;

14. Run asmcmd lsdsk to extract the OS_MB value to determine the maximum LUN size.

# asmcmd lsdsk –k –g –G TEST67_ASMLIBDG

15. Use SQL*Plus to resize the ASM device.

ALTER DISKGROUP TEST67_ASMLIB

   RESIZE DISK TEST67_ASMLIB

   SIZE $OS_MB

   REBALANCE POWER 10;

16. Verify the new size in the ASM device and disk group. After the resize operation completes, run asmcmd lsdsk to confirm the Total_MB value matches the OS_MB value.

# asmcmd lsdsk -k

Run asmcmd lsdg to confirm the Total_MB value on the disk group has increased.

# asmcmd lsdg

17. Restart the database.

$ srvctl start db -d demodb

Space reclamation

Dell Unity system supports the SCSI TRIM/UNMAP feature which allows operating systems to inform which data blocks are no longer in use and can be released for other uses. For space reclamation to work, the LUNs must be thin provisioned in the Dell Unity system and the Linux kernel, and Oracle ASM must also support the feature. The TRIM/UNMAP feature has been introduced in Linux kernel 2.6.28-25 and newer. With Oracle 12.2 ASMFD, thin-provisioned ASM diskgroups allow deleted space in datafiles to be reclaimed.

To verify the availability of the feature on the Linux operating system, query /sys/block/$disk/queue/discard_granularity. If the value is zero, it means the device does not support discard functionality. For example, since device sdx has a non-zero discard_granularity value, its free space will be reclaimed with TRIM/UNMAP.

# cat /sys/block/sdx/queue/discard_granularity

8192

Prepare ASM disk group for space reclamation

1. Ensure LUNs are thin provisioned in the Dell Unity storage system.

2. Create datafiles with an initial size and enable autoextend on the datafiles.
3. Set the THIN_PROVISIONED attribute to 'TRUE' on the disk group.

Reclaim space in ASM disk group

The following outlines the general steps to reclaim storage space in Oracle ASM.

1. Delete rows, tables, objects, or tablespaces.

2. Use SQL*Plus to enable the ROW MOVEMENT attribute on the tables.

ALTER TABLE <$TABLE_NAME> ENABLE ROW MOVEMENT;

3. After deleting objects in a table, run SQL*Plus command ALTER TABLE SHRINK SPACE to repack the rows, move the high water mark, and release unused extents in the datafiles.

ALTER TABLE <$TABLE_NAME> SHRINK SPACE;

4. Determine the high water mark (HWM) of each datafile and prepare the resize statements using the following script provided by Oracle. The original post can be found in the following Oracle article: https://asktom.oracle.com/pls/apex/f?p=100:11:0::::P11_QUESTION_ID:766625833673

# cat find_datafile_hwm.sql

set verify off line 200 pages 100

column file_name format a50 word_wrapped

column smallest format 999,990 heading "Smallest|Size|Poss."

column currsize format 999,990 heading "Current|Size"

column savings format 999,990 heading "Poss.|Savings"

break on report

compute sum of savings on report

column value new_val blksize

select value from v$parameter where name = 'db_block_size'

/

select file_name,

ceil( (nvl(hwm,1)*&&blksize)/1024/1024 ) smallest,

ceil( blocks*&&blksize/1024/1024) currsize,

ceil( blocks*&&blksize/1024/1024) -

ceil( (nvl(hwm,1)*&&blksize)/1024/1024 ) savings

from dba_data_files a,

( select file_id, max(block_id+blocks-1) hwm

from dba_extents

group by file_id ) b

where a.file_id = b.file_id(+)

/

column cmd format a85 word_wrapped

select 'alter database datafile ''' || file_name || ''' resize ' ||

ceil( (nvl(hwm,1)*&&blksize)/1024/1024 ) || 'm;' cmd

from dba_data_files a,

( select file_id, max(block_id+blocks-1) hwm

from dba_extents

group by file_id ) b

where a.file_id = b.file_id(+)

and ceil( blocks*&&blksize/1024/1024) -

ceil( (nvl(hwm,1)*&&blksize)/1024/1024 ) > 0

/

The prior script generates the following output.

VALUE

--------------------------------------------------------------------------

8192

              Smallest

                  Size    Curr     Poss.

FILE_NAME                Poss.    Size   Savings

----------------------------------------------  --------  ------  --------

+DATADG/DEMODB/DATAFILE/system.257.952265543         841     850         9

+DATADG/DEMODB/DATAFILE/demots.289.952606307      16,328  17,014       686

+DATADG/DEMODB/DATAFILE/undotbs1.259.952265593    12,932  24,708    11,776

+DATADG/DEMODB/DATAFILE/undotbs2.265.952265669        37   1,024       987

+DATADG/DEMODB/DATAFILE/demots.337.952606345      17,220  17,652       432

+DATADG/DEMODB/DATAFILE/demots.320.952606331      16,712  17,462       750

+DATADG/DEMODB/DATAFILE/sysaux.258.952265577       2,008   2,030        22

+DATADG/DEMODB/DATAFILE/users.260.952265593            1       5         4

                     --------

sum                       14,666

8 rows selected.

CMD  

-------------------------------------------------------------------------------------

alter database datafile '+DATADG/DEMODB/DATAFILE/system.257.952265543' resize 841m;

alter database datafile '+DATADG/DEMODB/DATAFILE/demots.289.952606307' resize 16328m;

alter database datafile '+DATADG/DEMODB/DATAFILE/undotbs1.259.952265593' resize

12932m;

alter database datafile '+DATADG/DEMODB/DATAFILE/undotbs2.265.952265669' resize 37m;

alter database datafile '+DATADG/DEMODB/DATAFILE/demots.337.952606345' resize 17220m;

alter database datafile '+DATADG/DEMODB/DATAFILE/demots.320.952606331' resize 16712m;

alter database datafile '+DATADG/DEMODB/DATAFILE/sysaux.258.952265577' resize 2008m;

alter database datafile '+DATADG/DEMODB/DATAFILE/users.260.952265593' resize 1m;

8 rows selected.

5. To resize the datafiles, copy and paste the ALTER DATABASE RESIZE statements associated with the datafiles. For example, the previous statements in bold shrink only the demodb tablespace that resides in the DATADG disk group.
6. Use the following SQL*Plus command to manually rebalance the disk group.

ALTER DISKGROUP DATADG REBALANCE POWER 10;

7. Confirm the release of the space in Unisphere by observing the Capacity and Space Used information on the LUN properties page. It might take several minutes to see the changes depending on the amount of data and how busy the system is at the time.

Note: Deleted space is not released until either the datafiles are deleted or shrunk and a rebalance operation is run against the ASM disk groups.

Linux Logical Volume Manager (LVM) is a common general-purpose storage manager in all popular Linux distributions. Since ASM does not support storing Oracle software, the software must be installed on a Linux file system that can be configured on top of LVM. LVM mirroring is not necessary because Dell Unity systems provide storage protection. Multiple LUNs can be grouped into a single LVM volume group. Then logical volumes must be created that span across these LUNs. When taking Dell Unity system snapshots on a multi-LUN volume group, ensure the LUNs are configured in a consistency group.

A file system is created on a logical volume where the Oracle binary is installed. More space can be added to the volume groups, logical volumes, and file systems either by adding new LUNs or by expanding existing LUNS in the volume groups. Once volume groups and logical volumes are expanded, the file systems can be resized to the newly added space. LVM and many file systems, such as ext4 and xfs, allow on-demand expansion without taking down the applications.

Unlike ASM, administrators must configure LVM striping, and data is not rebalanced when extending the volume group.

LVM guidelines

• Use whole LUNs for volume groups.
• Create a dedicated volume group for storing each copy or version of Oracle software. A dedicated volume group simplifies management and allows greater flexibility on array-based snapshots on individual Oracle software copies. 
• Use two or more LUNs in a volume group when performance is of concern.
• Configure all LUNs with the same size in the same volume group and group them in the same consistency group.
• In an Oracle RAC configuration, use a dedicated local volume group for each cluster node.

Physical volume data alignment

When initializing LUNs in LVM, use the --dataalignment argument to indicate the alignment starts at 1 M.

The following example shows the tasks to create an Oracle software file system on LVM:

# pvcreate --dataalignment 1m /dev/mapper/orabin-rac

# vgcreate vgoracle /dev/mapper/orabin-rac

# lvcreate –L 50g –n lv-oracle-bin vgoracle

# mkfs.xfs /dev/vgoracle/lv-oracle-bin (for xfs)

# mkfs.ext4 /dev/vgoracle/lv-oracle-bin (for ext4)

Note: if --dataalignment is not specified, mkfs might report a warning message (see below). Reinitialize the LUN with --dataalignment to ensure proper alignment.

Misalignment warning for mkfs.xfs:

warning: device is not properly aligned /dev/vgoracle/lv-oracle-logs

Use -f to force usage of a misaligned device

Misalignment warning for mkfs.ext4:

mke2fs 1.42.9 (28-Dec-2013)

/dev/vgoracle/lv-oracle-bin alignment is offset by 512 bytes.

This may result in very poor performance, (re)-partitioning suggested.

A local file system is preferred to store Oracle software and diagnostic logs. Datafiles can be stored in local file systems, but it is recommended to use Oracle ASM on block devices or Oracle DirectNFS. Sections Dell Unity file storage, Oracle Disk Manager, and Oracle Direct NFS discuss using the Dell Unity NFS service with Oracle DirectNFS.

The Dell Unity system supports a wide range of file systems on Linux. This section focuses on two popular and stable file systems: ext4 and xfs.

For additional information about supported file systems and feature limitations, see the Dell Technologies Host Connectivity Guide for Linux located on Dell Technologies E-Lab Navigator.

File system layout

The file system can be created on top of a LUN, a LUN partition, or a logical volume in LVM. Dell Technologies recommends using the whole LUN without partition or a logical volume in LVM for ease of management.

It can be beneficial to separate the Oracle software and Oracle diagnostic logs. To separate software and logs, create a separate volume group or assign a different LUN to store Oracle diagnostic log files. The diagnostic logs can consume a large amount of space in a short time. By isolating the logs in a different file system, it reduces the risk of filling up the storage space with these logs and affects the operation of the software. Since the diagnostic logs are not mission critical to the software operation, it is not essential to enable snapshots on the LUNs used by the logs. The diagnostic logs are also good candidates to be compressed to reduce the storage consumption. Table 6 shows an example of using separate file systems for software and diagnostic logs.

Table 6.      An example of file system layout for Oracle software and diagnostic logs

File system mount options

When mounting a file system, consider the following options and guidelines.

• Identify the file system by its UUID or LVM LV device in the /etc/fstab file. Avoid using any non-persistent device paths such as /dev/sd*.
• Query the UUID with the blkid command.

# blkid /dev/vgoracle/lv-oracle-rac-home

/dev/vgoracle/lv-oracle-rac-home: UUID="83cf5726-f842-448b-a143-5f77eb0d9b37" TYPE="xfs"

• Include discard in the mount option to enable space reclamation support for the file system. More information is provided in section File systems.
• Include nofail in the mount option if the Linux operating system experiences a mount issue during system boot. nofail prevents interruption during the boot process which requires manual intervention.
• For the xfs file system, disable the file system check (fsck option) in /etc/fstab because it does not perform any check or repair automatically during boot time. The xfs journaling feature ensures the file system integrity and data is in a consistent state after abrupt shutdown. If a manual repair or check is necessary, use the xfs_repair utility to repair a damaged file system.
• Set a value of 0 in the sixth field to disable the fsck check. Here is an example of an xfs file system entry in /etc/fstab:

UUID="83cf5726-f842-448b-a143-5f77eb0d9b37" /u01 xfs defaults,discard,nofail 0 0

Expand storage for the file system

Certain file system types, such as ext4 and xfs, support the online resize operation. The following outlines the general steps to resize a file system online assuming non-partition LUN are used.

1. Take manual snapshots of LUNs that are going to be expanded. See the section  Data protection and recovery for more information about taking snapshots and recovering from snapshots.

2. Expand the size of existing LUNs in Unisphere.
3. Perform a SCSI scan on the host system, refresh the partition table on each LUN path, and reload multipath devices.

# rescan-scsi-bus.sh –resize

4. Reload the multipath devices.

# multipathd -k"resize map orabin-rac"

For PowerPath, the new size is automatically updated.

5. Expand the logical volume if the file system is on top of LVM.

# lvresize –L $NEW_SIZE /dev/vgoracle/lv-oracle-rac-home

6. Extend the file system size to the maximum size, automatically and online.

# xfs_growfs –d /u01 (for xfs)

# resize2fs /dev/mapper/orabin-rac (for ext4)

Space reclamation

For file system types that support the online SCSI TRIM/UNMAP command, enable the discard mount option in /etc/fstab or include –o discard to the manual mount command. The discard option allows space to be released back to the storage pool in the Dell Unity system when deleting files in the file system. Administrators should review the file system documentation to confirm the availability of the features.

The LUNs must be thin provisioned in Dell Unity storage system for space reclamation to work. As new data is written to the file system, space is allocated in the Dell Unity system. When files are deleted from the file system, the operating system informs the Dell Unity system which data blocks can be released. The release of storage is automatic and requires no additional steps. To confirm the release of space in the Dell Unity system, monitor the Total Pool Space Used on the LUN properties page in Unisphere.

Dell Unity storage can serve file data through virtual file servers (NAS servers) while providing many of the advanced capabilities of Dell Unity systems. Some of these capabilities are shown in the following list, while others are mentioned in the remainder of this section:

• Advanced static routing
• Packet reflect
• IP Multitenancy
• NAS server mobility
• Configurable Dell Unity system parameters

Dell Unity x80F storage systems support NAS connections on multiple 10 GbE and 25 GbE ports. In an Oracle NFS environment, 25 Gb/s is recommended for the best performance. Starting with OE version 4.4, the ports on the embedded 4-port network card and optional Ethernet I/O modules that are used for NAS servers, replication, and import. These ports can be configured to a custom MTU size from 1280 through 9216.

Figure 6.        Dell Unity 480F, 680F, and 880F front-end Ethernet ports

Ports from iSCSI interfaces must still use 1500 or 9000 for MTU. Ports with custom MTU sizes can be configured for link aggregations and fail-safe networking (FSN), if the MTU size matches on all ports. See section 2.1 Interfaces of Dell Unity: NAS Capabilites for more information. For more information about Jumbo frames/MTU 9000, see the sections:

• Jumbo frames,
• Viewing the MTU size of Unity interfaces, and
• Changing the MTU size of Unity’s interface ports

Consideration should be given to configure highly available Unity network interfaces when the interfaces are used by an Oracle database. For more information about highly available Unity interfaces, see the sections: Dell Unity NAS servers, and Configuring LACP—NAS server.

Note: See the note in section Dell Unity file system and Oracle ASM disks regarding using NFS with Oracle ASM.

For more information, see the Dell Unity: NAS Capabilities, Dell Unity: Best Practices Guide, and Dell Unity: Service Commands documents.

For security reasons, all network configuration information (IPv4 and IPv6 IP addresses, subnet, CIDR, gateway, VLAN, and MAC addresses) has been redacted with character strings. When applying the content of this paper, replace the given character strings with the appropriate network information from your target environment.

Table 7.      Obfuscated IP addresses and network information used in this paper

Figure 7.        Unity NAS Server link aggregated network interfaces

Figure 8.        Snippets of interface configuration files from the NFS client

Dell Unity NAS servers are assigned to a single SP. All file systems serviced by a NAS server will have their I/O processed by the SP on which the NAS server is resident or current. If multiple NAS servers are required, load balance NAS servers by creating them on SP A and SP B such that the front-end NFS I/O is distributed evenly between SPs. Because both SPs can be used simultaneously, do not overprovision either of the SPs so that the peer SP does not become overloaded if an SP failover occurs.

Because each NAS server is logically separate, NFS clients of a NAS server cannot access data on another NAS server. Logically separate NAS servers can provide database isolation and protection across multiple NFS clients (database servers).

Unity NAS servers must have one or more interfaces that are created on one or more Ethernet ports from either the onboard embedded four port card or optional I/O modules. Should a failure occur on a single-link Unity network, the system does not initiate an SP failover to the peer SP. In an Oracle environment, such a failure would cause the database to appear unresponsive. Therefore, Dell Technologies recommends configuring highly available NAS Server networks for Oracle environments.

Highly available Unity network interfaces can be configured either as link aggregations or fail-safe networks. Link aggregations combine multiple Ethernet ports from either the onboard four-port card or optional I/O modules into one logical link. Link aggregations provide link redundancy and increased bandwidth. If a link in the link aggregate fails, traffic is automatically routed to surviving links in the link aggregate. For information about creating a link aggregation, see the section Unity link aggregation configuration.

Fail-safe networking (FSN) was made available starting with OE version 4.2.1. FSN is a high availability feature that extends link failover by supporting switch-level redundancy. FSN can consist of Ethernet ports, link aggregations, or both.

For more information about link aggregations, FSN, or high availability and redundancy, reference the Dell Unity: NAS Capabilities, and Dell Unity; High Availability white papers on Dell Technologies Info Hub,

For brevity, Unity NAS Server network interfaces configured as Fail-safe networks are not discussed in this document.

Supported NFS protocols

Dell Unity storage supports NFSv3 through NFSv4.1, including secure NFS.

All Dell Unity OE versions support Oracle dNFS in single-node configurations. Starting with OE version 4.2, Oracle Real Application Clusters (RAC) are also supported.

Unity 4.2 and newer versions, and Oracle dNFS starting with Oracle 12c have support for NFSv3, NFSv4, and NFSv4.1 protocols, with the following exceptions:

• Unity does not support directory delegation and parallel NFS (pNFS), and
• Unity 4.2 and new versions supports Oracle dNFS with NFSv3 protocol.

Therefore, do not configure pNFS in Oracle, and use the NFSv3 protocol if dNFS is used.

Note: pNFS refers to a protocol to access an NFS server, while dNFS refers to a driver built into the Oracle Disk Manager (ODM).

Creating a NAS server and adding a link aggregation network interface

This section assumes a link aggregation has been created.

To create a NAS server in Unity, perform the following steps in Unisphere:

1. Select File > NAS Servers > +

Figure 9.        Starting the “Create a NAS Server” wizard

2. Supply the general information for the NAS server, then select Next.

Figure 10.    Specify general settings for a NAS Server

3. When prompted for Ethernet Port, select the appropriate Unity interface. The selected interface will become the preferred interface for the NAS Server. Should the NAS server only have a preferred interface, the preferred interface is used for controlling and managing the NAS server and NFS share, and NFS data traffic. The IP address associated with the interface is used to mount and manage the NAS share on the NFS client. If the NAS Server will be hosting an Oracle database, Dell Technologies recommends selecting a link aggregation.

Figure 11.    Specifying a network interface for the NAS Server

If no Link Aggregations appear in the list of available interfaces, it is because no Link Aggregations have been configured in Unity. For information about creating link aggregations in Unity, see the section Creating Unity link aggregations.

Notes:
1) The Ethernet switch interfaces that are cabled to the port members of a Unity Link Aggregation must be members of a port channel on the switch. See Figure 54 and Figure 55 in the section Unity link aggregation and Ethernet switch port channel configuration.

2) Dell Technologies recommends using two TOR switches for redundancy in production environments. The lab that we used for this paper is configured with a single TOR switch.

3) When you select the Ethernet Port interface, Unity will automatically assign the corresponding Ethernet Port from the peer SP to the peer SP interface.

4. Specify the network configuration.

Figure 12.    Specifying an interface and network information for the NAS Server interface

Note: Specifying a value for VLAN ID instructs Unity to tag the network frames with the specified VLAN ID. If you are unsure if a VLAN ID should be specified, ask the Network Administrator responsible for the candidate IP address of this Unity NAS server interface.

If you select Edit to the right of VLAN ID, the following cautionary note will be displayed.

Figure 13.           Cautionary note on specifying a VLAN ID.

Our lab uses VLANs, and tagged switch interfaces cabled to the Unity interfaces. Therefore, Edit > Yes was selected and the appropriate VLAN ID for our environment was specified (see Figure 14).

Figure 14.           Specifying a VLAN ID.

VLAN ID should be set to None if:

• VLANs are not used, or
• VLANs are used, but the switch interfaces connected to the Unity interfaces of SPA and SPB chosen for this NAS Server are untagged to the VLAN ID.

Figure 15.    No VLAN ID specified if the corresponding switch port channel is untagged

If the switch interfaces connected to the Unity interfaces are tagged to the VLAN of the IP address (fff.fff.fff.fff) of the Unity interface, set the VLAN ID (see Figure 16).

Figure 16.    VLAN ID specified if the corresponding switch port channel is tagged in the specified VLAN.

5. Select NFS protocol NFSv3.

Figure 17.           Defining sharing protocols for the NAS Server

6. Only configure a UNIX Directory Service, DNS, and Replication if they are needed.
7.  After configuring all components of the NAS Server, the Create a NAS Server wizard will display a summary of your request. If the request is correct, click Finish.

Figure 18.    Summary of request to create a NAS Server

After a NAS server is created, the Dell Unity NFS file systems and then Dell Unity NFS shares can be created. See sections Dell Unity NFS file system for more information.

To increase throughput on the NAS server, additional interfaces (single or link aggregate) can be added to the NAS server. For information about adding more network interfaces to a Unity NAS server, see the section: Adding additional interfaces to Unity NAS Servers.

Adding additional interfaces to Unity NAS Servers

There are several use cases for using multiple interfaces in a Unity NAS Server. Two of them are:

• Increasing throughput, and
• Using dedicated interfaces for NFS control traffic and NFS data traffic for Oracle dNFS channels. For more information see the sections: Ethernet networks and dNFS, and Multiple client interfaces for dNFS.

If either of these use cases exist, consider adding one or more interfaces to the Unity NAS Server. Extra interaces added to the NAS server are added as a nonpreferred network interface.

To add additional interfaces to the Unity NAS Server, perform the following in Unisphere:

1. Select File > NAS Servers > double-click the NAS server name > Network > Interfaces & Routes > +.

2. Select either Production IP Interface or IP Interface for Backup & Test.

Note: NAS server interfaces can be either configured as production, or backup and DR testing interfaces. The type of interface dictates the type of activity that can be performed. Table 8 displays the characteristics of the interface types.

Table 8.      NAS server interface types

3. When prompted for Ethernet Port, select the appropriate Unity interface.

Note: To avoid a network interface single point of failure, Dell Technologies recommends selecting a highly available network interface when the NAS Server is used to host an Oracle database.

Figure 19.    Specifying a network interface for the NAS Server

If no highly available interface appears in the list of available interfaces, it is because no highly available interface has been configured in Unity. For information about creating Unity link aggregations, see the section Configuring LACP—NAS server Creating Unity link aggregations.

Notes:
1) Ethernet switch interfaces that are cabled to the Unity Link Aggregation port members must be members of a switch port channel. See Figure 21 and the section Unity link aggregate and Ethernet switch port channel configuration.

2) Dell Technologies recommends using two TOR switches for redundancy in production environments. The lab that we used for this paper is configured with a single TOR switch. Also, see the note in the section Dell Unity file system and Oracle ASM disks regarding redundancy considerations.

3) When you select the Ethernet Port interface, Unity automatically configures the same interface on the peer SP.

4. Specify the network IP address, subnet mask, and gateway.
5. Specify the VLAN ID if the environment requires it.

Figure 20.    Specifying network information for the interface

Note: For more information about specifying a VLAN ID, see step 4 in the section Creating a NAS server and adding a link aggregation network interface.

6. Select OK.
    

Repeat this section if additional link aggregate network interfaces are needed in the NAS Server.

The NAS Server interfaces in Figure 21 were configured for this paper:

Figure 21.    NAS server interfaces and switch port channels

Link Aggregation Ocp 0 0 is the preferred interface, while Link Aggregation Iom 1 0 and Link Aggregation Iom 1 2 are nonpreferred interfaces.

Dell Unity NAS file systems are well suited for Oracle. They provide scalability, large system size, flexible file system, storage efficiency, security, isolation, availability, recoverability, virtualization, and performance.

Scalability

Dell Unity file systems provide scalability in several areas, including maximum file system size, which makes Dell Unity storage ideal for Oracle environments. Dell Unity OE version 4.2 increases the maximum file system size from 64 TB to 256 TB for all file systems. File systems can also be shrunk or extended to any size within the supported limits. Dell Technologies recommends configuring storage objects that are 100 GB at a minimum and preferably 1 TB in size or greater.

Storage efficiency

Dell Unity storage supports thin-provisioned file systems. Starting with Dell Unity OE version 4.2, Unisphere can also create thick file systems. When using Dell Unity file storage with Oracle, consider using thin-provisioned file systems. Dell Unity also provides increased storage flexibility with manual or automatic file system extension and shrinking with reclaim.

Quotas

Dell Unity includes full-quota support to allow administrators to limit the amount of space that can be consumed from a user of an NFS file system or directory. When working with Oracle, quotas are not usually necessary. If deciding to use quotas, carefully consider their impact on managing the Oracle environment.

Unity NFS file systems can host Oracle datafiles that exist on ASM (Figure 22), file system (Figure 23), or both.

Figure 22.    Using ASM disk groups on an NFS file system.

Figure 23.    Using a file system for Oracle datafiles

To create a Unity NFS file system, perform the following steps in Unisphere:

1. Select File > File Systems > +.

2. Set the protocol to Linux/Unix Shares (NFS).
3. Set NAS Server to the NAS Server that will host the file system. Click Next.

4. Set Name: to the name of the file system. Provide an optional description. Then click Next.
5. Ensure that File-level Retention is Off. Click Next.

Note: Do not use file-level retention (FLR) on a file system if it will be used by an Oracle database. FLR prevents the modification or deletion of locked files until a specified retention date is reached. When the retention date is reached, the files on the file system can either be deleted or relocked. The files cannot be modified or changed to append only. This type of FLR is known as Write Once, Read Many (WORM).

6. Select the storage pool where the file system will reside.
7. Set the size of the file system. Select Next.
8. Select checkbox NFS Share (Linux/Unix).
9. Set Name to the wanted NFS share path name. Then, select Next.

Note: Click Show NFS advanced settings. Make sure the checkbox Allow SUID is selected.

10. When prompted to configure access, set Default Access: to Read/Write, allow Root.

Note: By default, the file system will be exported so that all NFS clients can see it. If the file system should be restricted to certain NFS clients, select +ADD HOST, then add the list of NFS clients where the NFS file system access is allowed. Then, select Next.

11. Configure an optional snapshot schedule. Click Next.
12. Configure an optional replication mode and RPO. Click Next.
13. Review the request to create the file system. If the request is correct, click Finish.
14. Select Close.

After a Unity NFS file system and NFS share have been configured, verify that the database server has access to the NFS export through all the interfaces. To verify access, use the Linux showmount command on all the IP addresses shown in the list of Exported Paths interfaces (see Figure 24). If any of the IP addresses do not have access to the NFS export, resolve the issue before configuring the NFS client and Oracle dNFS.

Log in as root to the NFS client and perform showmount using all the IP addresses to verify that the client can see the NFS export and file system:

Figure 24.    Verifying NAS exports are accessible

When using NFS storage with Oracle, Oracle recommends using Oracle Direct NFS (dNFS) rather than native kNFS.

Note: According to Oracle, caution should be exercised when considering using NFS storage in an Oracle database that uses ASM. In general, such configurations should be avoided or at least carefully considered. The issue is that to prevent data corruption, NFS file systems must be hard-mounted. Hard mounting means that the database or an ASM instance must wait indefinitely when an NFS server, or the communications link it operates over, becomes unavailable. NFS on ASM might be reasonable in some situations. See Oracle MOS note doc ID 1620238.1, “Creating File Devices On NAS/NFS FileSystems for ASM Diskgroups.” Using highly available interfaces in the Unity NAS Server will help reduce the possibility of running into this situation.

To create ASM disks on Unity file systems, use the following process (change values where necessary):

1. Log in to the NFS client (database server) as root.

2. Create a mount point for the NAS export. Change the owner, group, and privileges of the mount point to the Linux user owning the Oracle Grid software installation.

mkdir /ora-asm-db3

chmod 770 /ora-asm-db3

chown grid:oinstall /ora-asm-db3

3. Mount the Dell Unity NFS export using the IPv4 address of the preferred network interface of the NFS Export Path and NFS protocol vers=3.

Note: see the subsections: Overview, and Mount options for NFS shares in the section Oracle Direct NFS, and the section Dell Unity Service Scripts for information about wsize and rsize.

4. Change the owner, group, and privileges of the root directory on the NFS export to the Linux user owning the Oracle Grid software:

chmod 770 /ora-asm-db3

chown grid:oinstall /ora-asm-db3

5. Create the raw files for the ASM disk groups and set their permissions and ownership.

ASM disk groups can be created on multiple NFS shares residing on different NAS servers. Spanning ASM disks across multiple NAS servers might provide improved load balancing and flexible capacity planning. However, to use snapshots for a database, all disk groups containing the database must reside on a single NAS server as snapshots are not supported across multiple NAS servers.

To create ASM disk groups on Unity NFS exports, follow these steps.

1. Log in to the Linux grid user and make sure that the environment is set to the +ASM instance.

2. Start asmca.
3. Update the ASM Disk Discovery Path, to include the location and name of the ASM disks created previously. The location and name used for the ASM disk devices created were: /ora-asm-db3/asm-*.
   
   This example uses asmca to change the Disk Discovery Path string:

Note: In this example, ORCL:* is in the Disk Discovery Path because part of the Oracle environment relies on block devices managed by ASMLib.

asmca displays the available disks in the disk discovery path. If the ASM disks created previously are missing, check the ownership and permission of the disks.

4. Create the ASM disk groups with either asmca or SQL*Plus. A partial SQL*Plus example follows.

In Oracle 21c, the standard Oracle ODM static library ($ORACLE_HOME/rdbms/libnfsodm.a) manages Oracle I/O and its file management infrastructure. If the dynamic ODM library containing the embedded Oracle NFS client ($ORACLE_HOME/lib/libnfsodm.so) is enabled, a copy of the dynamic ODM library will exist in directory $ORACLE_HOME/rdbms/odm/. The dynamic library allows the use of NFS devices without using the native Linux NFS kernel (kNFS).

Note: In versions prior to Oracle 21c, the version number of Oracle is part of the ODM library and Oracle NFS client file names. For example, in Oracle 19c, the respective ODM library and Oracle NFS client names are: $ORACLE_HOME/lib/libodm19.a
$ORACLE_HOME/lib/libodm19.so
$ORACLE_HOME/lib/libodmd19.so
$ORACLE_HOME/rdbms/lib/odm/libnfsodm19.so

Generally, NFS traffic can be classified as control/management traffic or I/O traffic on application data. Whether the Oracle ODM NFS client library is enabled, Linux’s kernel NFS client driver (kNFS) manages the control/management of NFS devices. When the ODM library is enabled, the Oracle environment is said to be using Oracle Direct NFS (dNFS) and all database I/O, NFS data traffic, flows through the dNFS driver. When the ODM library containing the embedded Oracle NFS client is disabled, all database I/O flows through the kNFS driver:

• get attribute
• set attribute
• access
• create
• mkdir
• rmdir
• mount
• umount

NFS is a supported storage option for Oracle and can be used for:

• Oracle clusterware binaries
• OCR and voting files
• Oracle Standalone and RAC database binaries
• Oracle Standalone and RAC database datafiles
• Oracle database recovery files

Note: Oracle dNFS does not support Oracle clusterware files (voting disks and OCR). Also, because dNFS does not support soft mounts, dNFS does not support quorum disks (quorum failure groups). For quorum disks, use an NFS soft mount. The quorum failure group feature can be used in an Oracle ASM disk group without requiring a hard mount for NFS storage. With hard mounts, ASM or the database might stop responding if the NFS server becomes unavailable.

Oracle ADVM, Oracle ACFS, and ASMFD are not supported with NFS. However, NFS disks can be used with Oracle ACFS NAS Maximum Availability Extension.

Oracle dNFS is an optimized NFS client from Oracle for database I/O and resides in the ODM library as a part of the Oracle database kernel. dNFS improves the stability and reliability of NFS storage devices over TCP/IP, more so than the native Linux NFS driver (kNFS). dNFS also improves performance to NFS storage devices by bypassing the kNFS I/O stack. When mounting the database datafiles, Oracle will first load dNFS functionality if the Direct NFS client ODM library is enabled. If dNFS cannot access an NFS storage device, dNFS silently reverts to using the kNFS client driver. However, to ensure this reversion occurs, the kNFS client mount options rsize and wsize must be used.

It is recommended to use dNFS if NFS storage devices are used by an oracle database so that the performance optimizations and multipathing built into Oracle can be exploited.

To use Oracle dNFS with Unity storage, use NFS protocol NFSv3, and enable Unity’s nfs.transChecksum NAS server parameter (set to 1). This parameter ensures that each transaction carries a unique ID and avoids the possibility of conflicting IDs that result from the reuse of relinquished ports. For more information about nfs.transChcksum and how to configure it, see the section Dell Unity Service Scripts and Dell Unity document: Dell Unity Family Service Commands Technical Notes.

The advantage of using Oracle dNFS lies within the fact that it is part of the Oracle database kernel. dNFS services all I/O to NFS storage devices. dNFS gives Oracle the ability to manage the best possible configuration, tune itself, use Oracle buffer cache, and use available resources for optimal multipath NFS data traffic I/O.

dNFS:

• Provides the ability to manage the best possible configuration
• Automatically tunes NFS data traffic for performance optimizations
• Takes advantage of the Oracle buffer cache
• Appropriately uses available resources for optimal NFS data traffic, without the overhead of the client operating system kernel software

Oracle recommends using dNFS if NFS storage devices are used so these benefits can be exploited.

During the installation of Oracle, the installer prompts for directory locations for storing software and components. These directories can reside on NFS shares. For more information, see the Oracle documentation. Table 9 provides some examples of different Oracle directories that could reside on an NFS share.

Table 9.      Example directories that could be serviced by NFS

After determining what NFS shares Oracle will use, create the NFS shares in Unity. Once the NFS shares are created in Unity, create mount points in Linux for the NFS shares. Set the privileges, owner, and group of the Linux mount points for the NFS shares per Oracle requirements. For information about mounting Unity NFS exports for Oracle, see the section Dell Unity file system and Oracle ASM disks.

The environment used for this paper used only one Unity NFS share (ora-asm-db3) to host a single Standalone Oracle database. Oracle binaries and the Grid Infrastructure including the ASM database resided on block devices.

Before the dNFS driver is configured or used on an NFS share, the NFS share must first be mounted using the native NFS driver (kNFS). Specific mounting options are required when mounting an NFS share for Oracle usage. If the NFS export will be used for Oracle services that are to be automatically restarted when the server restarts, add the NFS export and mount options to /etc/fstab. If the NFS share and mount options are not specified in /etc/fstab, Oracle will experience issues if the Oracle servers are set to auto start during a Linux boot. In an Oracle RAC cluster, ensure that all nodes in the cluster use the same mount options and /etc/fstab entry for each identical NFS share mount point.

After the NFS share is mounted using kNFS, dNFS mounts and unmounts the NFS share logically as needed. Because dNFS uses a logical mount, after Oracle unmounts the NFS share, the NFS share is still physically mounted to the server and can be accessed through kNFS.

If NFS is used for database files, the NFS buffer size for reads (rsize) and writes (wsize) must be set to at least 16,384. Oracle recommends a value of 32,768. For the NFS client, these values are set in /etc/fstab or when explicitly mounting an NFS share. For performance reasons, Dell Technologies recommends evaluating whether rsize and wsize should be changed from the Oracle default.

Unity NAS servers support a range of values for rsize, and wsize in 8 KiB increments starting from 8192 to 1048576. The default value of Unity’s rsize and wsize is in Unity parameter nfs.v3xfersize. Its value is set depending on the amount of memory in the system. For the Unity system used for this paper, a default of 262144 was set for both rsize and wsize. dNFS clients issue writes with Oracle v$dnfs_servers.wtmax granularity to the Unity NAS server. Dell Unity NAS facility parameter nfs.v3xfersize.current_value specifies the size of I/Os to and from the Unity NAS server. nfs.v3xfersize.current_value must be a multiple of v$dnfs_servers.wtmax.

Note: Changing the rsize or wsize on the Unity NAS server requires a restart of the SP. For more information, see the sections Dell Unity Service Scripts and Unity’s nfs.v3xfersize parameter.

For additional mount options and information about using NFS shares intended for Oracle, see the Oracle MOS notes listed in Oracle Support MOS notes.

Performance of Oracle on NAS devices depends in part on network performance between the NFS client and server. Therefore, consider whether to use a dedicated interface for NFS control and NFS data traffic.

If NFS and network redundancy is a concern, consider creating a bonded interface for NFS control traffic on the NFS client (database server). This bonded interface could be a bonded public network, a bonded interface for the RAC interconnect in a RAC environment, or some other bonded interface.

Using a bonded interface for NFS control traffic and an interface for NFS data traffic might not always be possible because of a limited number of available NICs. In such cases, using a single unbonded interface for both NFS control and data traffic may be the only possible option. However, that configuration might cause network performance issues under heavy loads because the server will not perform network load balancing. There will also be no redundancy for NFS control traffic.

When using dNFS, using multiple network paths is recommended. Oracle supports one to five network paths between a NAS server and NFS client—one path for NFS control traffic and up to four paths for NFS data traffic. Each of these paths should also be configured on a separate dedicated subnet. This includes not using the subnet of the public network for NFS. Using different and dedicated subnets provides:

• network traffic isolation
• simplifies operating system, network, and dNFS configuration
• eliminate the need for static routes for dNFS
• reduces the possibility of misconfiguring

If there are not enough subnets available, it is possible to configure dNFS paths to share subnets. However, this is not optimal and it requires extra configuration in the operating system and in Oracle. For the operating system, ingress filtering must be relaxed for multihomed networks, and static routes must be defined. For Oracle, file oranfstab must be created to direct dNFS data traffic to flow across paired endpoint IP addresses in the same shared subnet. This extra configuration prevents the operating system from using the default route and allows multiple NIC interfaces in a server to use the same subnet.

If a subnet is shared between NICs and the appropriate configuration does not exist in the operating system and in Oracle, the operating system chooses the dynamic route. When it does, the operating system will invariably use the first best-matched route possible from the operating system routing table. Usually, that route will be incorrect and will impact the environment negatively. Some negative impacts are:

• There will be no dNFS load balancing
• There will be no NFS traffic scalability
• dNFS path failover will not work as expected

When sharing a subnet, all paired end-point IP addresses of the different network paths for dNFS data traffic must be defined in the Oracle file oranfstab.

For additional information, see sections Single network path for dNFS, Multiple client interfaces for dNFS, Configuring LACP – NAS client, and Oracle dNFS configuration file oranfstab.

In Ethernet networking, a unit of data is referred to as a frame. Jumbo frames are frames having more than 1500 bytes of data known as a payload. Commonly, jumbo frames refer to frames that carry a payload up to 9000 bytes. To ensure the frame size of a packet stays the same size as it traverses through the network, the frame size must be set the same, end-to-end in the infrastructure. If the frame size is not consistently defined end-to-end, the maximum frame size used will be governed by the smallest frame size value defined, end-to-end, in the network path.

Jumbo frames are supported and recommended by Oracle when dNFS is used. Using jumbo frames allows the network stack to bundle payloads into larger frames and reducing TCP overhead. The frame size of any frame can vary depending on the immediate needs of the network session established between the NFS client and NFS server.

Frame size is defined by setting the maximum transmission unit (MTU). By default, Unity NAS File interfaces and Unity Ethernet interfaces are set to MTU 1500, but can support a maximum of 9216. See Figure 25.

Figure 25.    Unity MTU range

MTU 9000 might be acceptable for heavy dNFS work loads, but it should be evaluated in a nonproduction environment before changing it in production. Dell Unity best practices recommend, if possible, configuring jumbo frames (MTU 9000). For more information, see the document: Dell EMC Unity: Best Practices Guide (delltechnologies.com)

To view the current MTU size of Unity interfaces, perform the following steps:

1. Log in to Unisphere

2. Click the gear icon in Unisphere
3. Select Access > Ethernet > column Location
4. If column MTU is not shown, select the gear icon in the Setting wizard. Hover over Columns, then click the checkbox for MTU.

To change the MTU size of a Unity interface, perform the following steps:

1. Log in to Unisphere

2. Select the gear icon in Unisphere
3. Select Access > Ethernet > column Location

Note: If column MTU is not displayed, display it. For more information, see step 4 in the section Viewing the MTU size of Unity interfaces.

4. Of the MTU valus displayed, double-click the MTU value that should be changed.
5. Change the MTU size to 9000 and select OK.

Note: If you change the MTU value, consider changing the MTU value of all components of the network path, end-to-end (switch ports and database server), to match this value. If you do not, the maximum frame size used will be the smallest MTU size in the network path. Also, you cannot change the MTU size for either the management port or a synchronous replication management port.

For the simplest dNFS network configuration, a single network path can be configured. The single network path in Figure 26 is used for the database server public network and network traffic between the NAS server and NFS client. A benefit of this configuration is its simplicity, however it creates a single point of failure for NFS and potentially not enough bandwidth to support general network and NFS traffic. For these reasons, Dell Technologies does not recommended using this network configuration for a critical, highly available database having high transaction rates.

The remainder of this section covers two different single network path configurations:

• Single network path for both public network and NFS traffic
• Single dedicated network path for both NFS control and data traffic

Single client network path for both public network and NFS traffic

Figure 26.    Single Ethernet path between the NFS client and Dell Unity NAS server

Note: Figure 26 represents a network route between a single interface in Dell Unity and a single interface on the database server. This route is intended to be used by NFS control and data traffic, and public network traffic. For information about creating network routes and using specific network routes for certain network traffic, see the following sections: Shared subnets, Static routing, dNFS without oranfstab, and dNFS with oranfstab.

The following figures show:

• NAS server and interface,
• NFS share (export),
• Unity NAS file system, and
• NFS client interface configuration

using this path:

Figure 27.    Dell Unity NAS server with a single Ethernet path for NFS traffic

Note: See Dell Unity NAS servers for information about setting VLAN ID.

Figure 28.    Dell Unity NFS share

Figure 29.    Dell Unity file system

Figure 30.    NFS client Ethernet interface configuration

dNFS uses two kinds of NFS mounts: the native operating system mount of NFS (known as a kernel kNFS mount) and the Oracle database NFS mount (dNFS mount). When a single network path is used for dNFS, file oranfstab is not necessary because Oracle dNFS gleans the required information for the mounted NFS share from file /etc/mtab.

Figure 31.    /etc/mtab entry for the Unity NFS export

If dNFS is unable to find the necessary information in /etc/mtab, control is handed back to the database and file access is attempted through kNFS.

Single dedicated network path for both NFS control and data traffic

To reduce the potential of running out of bandwidth on a single interface, consider moving NFS traffic to a different interface on the database server. Moving NFS traffic to a different interface requires an additional IP address and an additional switch interface.

Figure 32.    NFS traffic on a dedicated single network path

If this additional IP address is in the same subnet used by any other NIC interface in the database server, see sections Static routing and Oracle dNFS configuration file oranfstab.

Note: A single point of failure for NFS and the potential of running into NFS bandwidth issues exists with this configuration. 

If multiple network paths are intended for NFS traffic, consider using one path for NFS control/management traffic and the remaining NFS paths for NFS data traffic. This ensures that NFS data traffic flows through the paths intended for NFS data traffic, and NFS control traffic to flow through the path intended for NFS control traffic.

Figure 33.    Multiple network routes for NFS control and data traffic

When multiple dNFS paths exist for data traffic and one of the paths fail, dNFS reissues requests over any of the remaining dNFS data paths, improving database availability. Multiple data paths also provide Oracle the ability to automatically tune the paths to the NFS storage devices. Because dNFS implements multipath I/O internally, Oracle recommends not configuring LACP and channel-bonded interfaces on the database server (NFS client) for dNFS data traffic.

In environments requiring high availability, bonded NIC interfaces for NFS control traffic are recommended.

NFS and channel-bonding configuration

If an unbonded interface is used for the NFS mount on the NFS client, or NFS filer (see Figure 33), NFS control traffic can be blocked if the interface is down. This blocked NFS traffic causes the database to appear unavailable in certain operations.

Figure 34.    Single point of failure – single interface for NFS control traffic

To mitigate this single point of failure:

• configure LACP on multiple interfaces to create a bonded interface on the NFS client,
• create a link aggregate interface on the Unity NAS server, and
• create and channel-bonded interfaces for NFS control/management traffic on the switches connected to the database server and Unity array (see Figure 35).

Also, consider configuring LACP on Unity interfaces used for NFS data traffic, and port channels on the switch connected to the Unity interfaces. Using bonded interfaces on the database server for dNFS data traffic is not recommended by Oracle. Also see the note in the section Dell Unity file system and Oracle ASM disks regarding network redundancy considerations.

The bonded interface can be shared with other network traffic like public network traffic or even traffic from an Oracle RAC interconnect. LACP needs to be configured on multiple interfaces to create a channel-bonded interface. Be mindful that NFS control or metadata traffic is minimal compared to NFS data traffic, so using 1 GbE interfaces for the bonded interface of NFS control traffic may suffice.

When LACP is configured on the NFS client interfaces for NFS control traffic, a port channel must be configured on the Ethernet switch interfaces connected to the LACP client interfaces.

Figure 35.    Network routes using bonded interfaces

When configuring dNFS with multiple network paths, if possible, use a unique network for each path. When multiple unique networks are not available, a different IP address from the same subnet can be used for each of the network paths. For more information, see the section Shared subnets.

If multiple network paths are needed for dNFS/kNFS control traffic and dNFS data traffic, Oracle recommends that each path uses a separate subnet. However, using separate subnets may not always be possible.

This section discusses the use of subnet sharing on network paths for Oracle dNFS.

When configuring dNFS to use a subnet that is shared with another network interface, additional configuration, including IPv4 network routing filter and static routing, is required.

According to Oracle, if any IP address used by dNFS is from a subnet on another NIC interface, the following additional configuration changes must be made:

• Use dontroute in oranfstab
• Configure static routing for each of the network paths used by dNFS

Note: For more information about dontroute and oranfstab, see Oracle Database—Database Installation Guide.

For more information about shared subnets, see

• The section Static routing
• How to Setup Direct NFS Client Multipaths in Same Subnet (Doc ID 822481.1)
• How to configure DNFS to Use Multiple IPs (Doc ID 1552831.1)

Ipv4 network routing filters

Linux 8 follows the recommendations of ingress filtering for multihomed networks (http://tools.ietf.org/html/rfc3704). These routing filters must be relaxed for multiple NIC interfaces in the same server to use the same subnet. Before configuring network interfaces to use the same subnet, ensure that you relax routing filters.

Before installing Oracle, if the Oracle 21c preinstall rpm is used to configure the operating system, routing filters will be relaxed appropriately.

With OL 8u8, rp_filter is set to 2 for all network interfaces including the default interface. A value of 2 (loose filtering) is acceptable. Should no filtering (rp_filter = 0) or strict filtering (rp_filter = 1) be needed, see Oracle Grid Infrastructure—Grid Infrastructure Installation and Upgrade Guide, and My Oracle Support Doc ID 1286796.1.

If dNFS multipathing is needed and there is only one subnet, loose filtering (rp_filter = 2) needs to be set.

Setting rp_filter to an incorrect value can cause interconnect packets to be blocked or discarded in a RAC configuration.

Table 10.  Reverse path filtering

To verify that IPv4 routing filters have been relaxed, look at the rp_filter settings in /etc/sysctl.conf. The values should be 2.

Figure 36.    Loose Ipv4 rp_filter

If IPv4 reverse path filtering is not set, add or change the IPv4 rp_filters in /etc/sysctl.conf and make them active.

Figure 37.    Loose IPv4 rp_filter

Starting with OL kernel 2.6.31, Oracle RAC systems that use multiple NICs for the private interconnect must have parameter rp_filter set appropriately.

For information about channel bonded interfaces for NFS control traffic, see Oracle MOS notes:

• How to Setup Direct NFS Client Multipaths in Same Subnet (Doc ID 822481.1)
• How to configure DNFS to Use Multiple IPs (Doc ID 1552831.1)

Static routing

When dNFS traffic must route traffic to an IP address from a subnet already in use on the NFS client, static routing must be configured. A static route must be defined for each of the network addresses from a shared subnet and used by dNFS as a route. This requirement includes defining a static route for dNFS data traffic should it use the same subnet as dNFS control traffic. If static routes are not defined, automatic load balancing and performance tuning of dNFS will not operate as expected per the dNFS path definitions in file oranfstab.

The remainder of this section provides two examples of static routing. The first example considers two interfaces sharing a subnet, and the second example considers four network interfaces sharing a subnet.

Shared subnet on multiple interfaces with single dedicated dNFS data path

The following figure illustrates a scenario where an NFS client has two interfaces. Interface enp132s0f0 is intended to be dedicated to dNFS traffic, and interface eno3 is for general public Ethernet traffic. Both interfaces use a network IP address from the same subnet, and default routing is defined. Figure 38 shows dNFS traffic flowing through interface eno3 rather than the intended interface enp132s0f0 when default routing is used.

Figure 38.    Incorrect network route taken by dNFS data traffic on a shared subnet with default routing

The following two figures show the corresponding routing table for Figure 38.

Figure 39.    Default routing—netstat -r: single interface for dNFS on a shared subnet not used

Figure 40.    Default routing—ip -4 route: dedicated single interface for dNFS on shared subnet not used

If default routing and IP addresses from the same subnet are used, the operating system searches the routing table for the route that best matches the destination address and mask. The operating system will use the best-matched route found. Because interfaces eno3 and enp132s0f0 share the same subnet (nnn.nnn.nnn.nnn) and CIDR (cc), the operating system considers both entries as best-matched. When multiple best-matched entries exist, the operating system selects the top-most best-matched entry for the route. Because interface eno3 is the top-most best-matched entry in both Figure 39 and Figure 40, the operating system will use interface eno3 to route dNFS data traffic to the target. However, as shown in Figure 38, interface enp132s0f0 should be the interface used for that traffic.

To mitigate this traffic misdirection, a static route must be added to the operating system routing table. The static route forces the operating system to send dNFS data traffic between the IP addresses specified in the static route. In this case, the static route is:

• IP address (xxx.xxx.xxx.xxx) of interface enp132s0f0, and
• IP address (mmm.mmm.mmm.mmm) of link aggregate Ocp 0 0 on the Unity base enclosure 4-port card

To add a static route between interface enp132s0f0 and Unity link aggregate Ocp  0 0, run the ip route add or nmcli connection modify Linux command from Linux user root. Specify interface enp132s0f0 and parts of the network configuration (IP address, CIDR, and gateway) of Unity’s link aggregate Ocp 0 0:

ip route add mmm.mmm.mmm.mmm dev enp132s0f0

or

nmcli connection modify enp132s0f0 +ipv4.routes \ "mmm.mmm.mmm.mmm/cc ggg.ggg.ggg.ggg"

To verify that the static route appears in the routing table, run either:

• netstat -r
• ip route
• nmcli connection show

command and filter for enp132s0f0. See Figure 41.

Figure 41.    Static route between enp132s0f0 and Unity Link aggregate Ocp 0 0

Because the static route defines a specific destination address (mmm.mmm.mmm.mmm) rather than the subnetwork IP address, the operating system considers the specific destination address as best-matched and uses it. The static route forces Ethernet traffic from interface enp132s0f0 (xxx.xxx.xxx.xxx) to Unity (mmm.mmm.mmm.mmm), as shown in Figure 42.

Figure 42.    Static route—correct path taken for dNFS data traffic

Manual changes to the routing table are not persistent across server reboots. If static routes should be persistent, consider creating routing configuration files. For more information, see the section Persistent static routes.

Shared subnet on multiple interfaces with multiple dedicated dNFS data paths

This example illustrates how static routing changes the paths taken on multiple interfaces configured with IP addresses from the same subnet with default routing (See Figure 43). NFS client Ethernet interfaces configured are:

• Interface bond1 is intended to be dedicated to kNFS/dNFS control traffic.
• Interfaces enp131s0f0 and enp3s0f0 are intended to be dedicated to dNFS data traffic.
• Interface eno3 is intended for general public Ethernet traffic.

Figure 43 shows that dNFS data traffic flows through interface eno3 rather than intended interfaces enp131s0f0 and enp3s0f0.

Figure 43.    Incorrect network route taken by dNFS data traffic on a shared subnet with default routing

The following two figures show the corresponding routing table for Figure 43.

Figure 44.    Default routing—netstat -r: traffic routed through interface eno3

Figure 45.    Default routing—ip -4 route: traffic routed through interface eno3

Figure 45 shows interface eno3 having the same subnet and CIDR as interfaces enp131s0f0, en3s0f0, and bond1. Since interface eno3 is the top-most best-matched entry in the routing table, all dNFS traffic flows through eno3.

To mitigate this traffic misdirection, static routes must be added to the operating system routing table. The static route forces the operating system to send dNFS traffic between the IP addresses specified in the static route. In this case, the static routes are:

• IP address (xxx.xxx.xxx.xxx) of interface bond1 of the database server, and IP address (mmm.mmm.mmm.mmm) of Unity’s Link aggregate Ocp 0 0 on the base enclosure 4-port card
• IP address (yyy.yyy.yyy.yyy) of interface enp131s0f0 and IP address (ddd.ddd.ddd.ddd) of Unity’s Link Aggregate Iom 1 0 0 on the IoModule1 4-port card
• IP address (ttt.ttt.ttt.ttt) of interface enp3s0f0 and IP address (fff.fff.fff.fff) of Unity’s Link Aggregate Iom 1 2 0 on the IoModule1 4-port card

To direct Ethernet traffic to flow through the three routes listed above, the following static routes are needed:

ip route add mmm.mmm.mmm.mmm dev bond1

ip route add ddd.ddd.ddd.ddd dev enp131s0f0

ip route add fff.fff.fff.fff dev enp3s0f0

Once the static routes are defined, the operating system chooses them because they best fit the destination and subnet mask/CIDR of the destination interface (see Figure 46).

Figure 46.    Static route—correct path taken for dNFS data traffic

Manual changes to the routing table are not persistent across server reboots. If static routes should be persistent, consider creating routing configuration files. For more information, see the section Persistent static routes.

To verify that the static routes appear in the routing table, run either:

• netstat -r
• ip route
• nmcli connection show

and filter on the interface name:

Figure 47.    Static route between database server interface bond1 and Unity link aggregate Ocp 0 0

Figure 48.    Static route between database server interface enp131s0f0 and Unity link aggregate Iom 1 0 0

Figure 49.    Static route between database server interface enp3s0f0 and Unity link aggregate Iom 1 2 0

Persistent static routes

To ensure that static routes persist across server reboots, create a static route configuration file in directory /etc/sysconfig/network-scripts for each of the static routes. Name the static route configuration file route-<NAS client interface>, substituting <NAS client interface> with the name of the network interface.

The following static route configuration files define the routes shown in Figure 50:

[root@r730xd-1 network-scripts]# cat route-bond1

ADDRESS0=mmm.mmm.mmm.mmm

NETMASK0=255.255.255.255

GATEWAY0=ggg.ggg.ggg.ggg

[root@r730xd-1 network-scripts]#

[root@r730xd-1 network-scripts]# cat route-enp131s0f0

ADDRESS0=ddd.ddd.ddd.ddd

NETMASK0=255.255.255.255

GATEWAY0=ggg.ggg.ggg.ggg

[root@r730xd-1 network-scripts]#

[root@r730xd-1 network-scripts]# cat route-enp3s0f0

ADDRESS0=fff.fff.fff.fff

NETMASK0=255.255.255.255

GATEWAY0=ggg.ggg.ggg.ggg

[root@r730xd-1 network-scripts]#

Figure 50.    Static routes for dNFS data and kNFS control traffic

Prior to OE 4.2.1, only ports belonging to the same I/O modules or onboard Ethernet ports could be used in a link aggregation. Starting with OE 4.2.1, link aggregations can be created with ports between I/O modules and the onboard Ethernet ports (see Figure 6). This requirement was added to provide greater flexibility and options to configure high availability in Unity NAS server connectivity. All ports in the link aggregation must have the same speed, MTU, and duplex setting.

The Unity ports configured in a link aggregate must be cabled to switch interfaces that are configured as a port channel (see Figure 21 or Figure 52).

Unity link aggregation configuration

Link aggregations can be defined across two, three, or four ports, on the same card or I/O modules, or between the card and I/O modules within the same SP. Ports in that configuration must have the same speed, duplex, and MTU. For more information, see the Unity documentation.

Link aggregations need to be created before specifying them when configuring a Unity NAS server.

Note: Dell Technologies recommends using two TOR switches for redundancy. The lab used for this paper used only a single TOR switch.

When a link aggregate interface is configured in Unisphere, the candidate ports for the link aggregate must be cabled and configured in both SP A and SP B. The candidate ports must be cabled to the appropriate port channels on the Ethernet switch (see Figure 21).

The following figures show three link aggregations (Ocp 0 0, Iom 1 0, and Iom 1 2). Each of the link aggregations is mirrored on both SP A and SP B.

Figure 51.    Unity link aggregations

The mirrored link aggregations are necessary because, in failover, the peer SP uses the ports residing on the peer SP that mirror the ports being used on the primary SP.

The number of switch interfaces required for a Unity link aggregate is double the number of interfaces in the Unity link aggregate. If a Unity link aggregate contains two interfaces, four switch interfaces are required: two switch interfaces for the two interfaces in SP A and two switch interfaces for the two interfaces in SP B. Figure 52 shows the interface mappings between Unity link aggregated interfaces and the Ethernet switch port channels used in this paper.

Note: Dell Technologies recommends using two TOR switches for redundancy. The lab used for this paper used only a single TOR switch. Using one switch is not recommended.

Figure 52.    Interface mappings: Unity and Ethernet switch

Creating Unity link aggregations

To configure Unity link aggregations, perform the following steps in Unisphere:

1. Log in to Unisphere.

2. Select the gear icon (Settings wizard).
3. Under Access, select High Availability. Then click the plus sign (Add) under Link aggregations.
4. Optional: For ease of selecting the primary ports for the Link Aggregation, click column heading Location to sort the ports.
5. Locate and select the port that will be used as the primary port of the link aggregation, then click Next.

6. Locate and select the checkbox of the port that will be used as the secondary port of the link aggregation, the click Next.

7. Review the request and click Finish if the request is correct.

The Create Link Aggregation wizard displays the new link aggregation.

8. After the link aggregation is created, click Close.

By default, the link aggregate is created with MTU 1500. Oracle recommends using Jumbo frames (MTU 9000). See section Jumbo frames for additional information.

Repeat the above steps as necessary to create additional link aggregations. Figure 53 shows the three link aggregations used in this paper.

Figure 53.    Three link aggregations

After a link aggregation is created, it can be used as a NAS server network interface. For information about creating a NAS server and assigning a link aggregation as the interface, see Creating a NAS server and adding a link aggregation network interface.

Unity link Aggregation and Ethernet switch port channel configuration

If NFS control traffic will flow through a bonded NFS client (database server) NIC interface and a link aggregate in Dell Unity storage, Ethernet switch ports must use LACP.

Figure 54 illustrates an example of switch interfaces and port channels in a Dell Networking S5000 switch cabled to the Unity array. The port channels will be used for NFS control traffic. Port channel 1 will be used for Dell Unity SP module A and port channel 2 will be used for Dell Unity SP module B.

Figure 54.    Cabling between Dell Unity and an Ethernet switch – single port channel per SP

Figure 55 illustrates an example of multiple port channels. Port channels 1, 3, and 5 are configured for SP A. Port channels 2, 4, and 6 are configured for SP B. While SP A is healthy, NFS control traffic flows through port channel 1 and dNFS data traffic flow though port channels 3 and 5. On failure of SP A, NFS control traffic will flow through port channel 2 and dNFS data traffic will flow through port channels 4 and 6.

Figure 55.    Cabling between Dell Unity and an Ethernet switch – multiple port channels

File oranfstab is optional. If it exists, it defines:

• Mount points that are available to dNFS
• One or more NAS server stanzas
• For each NAS server stanza, the network paths that dNFS should use for dNFS data traffic

oranfstab can reside in either /etc, $ORACLE_BASE_HOME/dbs or $ORACLE_BASE_CONFIG/dbs. If oranfstab resides in /etc, it will be global to all databases running on that server regardless of which $ORACLE_HOME the databases are running from. If oranfstab resides in $ORACLE_BASE_HOME/dbs, oranfstab it is global to any database running from the corresponding $ORACLE_HOME. If $ORACLE_HOME is shared between RAC nodes, all RAC databases running from the shared $ORACLE_HOME will use the same $ORACLE_BASE_HOME/dbs/oranfstab.

For more information about ORACLE_BASE_HOME and ORACLE_CONFIG_HOME, see Oracle 21c Database Installation Guide.

dNFS searches for mount entries in the following order and uses the first matching entry as the mount point:

• $ORACLE_BASE_HOME/dbs/oranfstab
• $ORACLE_BASE_CONFIG/dbs/oranfstab
• /etc/oranfstab
• /etc/mtab

If a database uses dNFS mount points configured in oranfstab, Oracle first verifies kNFS mount points by cross-checking entries in mtab and oranfstab. If a match does not exist, dNFS logs a message and fails to operate.

Benefits of using oranfstab

oranfstab provides dNFS with metadata to be able to perform multipathing. dNFS multipathing provides:

• Increased dNFS bandwidth
• Automatic dNFS data traffic load balancing
• Automatic dNFS channel failover

If any of these features are needed, file oranfstab is required.

When oranfstab is configured, dNFS automatically performs load balancing across all specified available channels in oranfstab. Also, if one channel fails, dNFS reissues I/O commands over any remaining available channel specified for that NAS server.

oranfstab directives

The following table shows the available parameters for oranfstab.

Note: dontroute is a POSIX option, which sometimes does not work on all Linux systems when multiple paths in the same subnet are used. Dell Technologies recommends investigating and testing the usage of dontroute and operating system static routes.

Single server stanza

The following oranfstab file was used for this paper. It defines one NAS server with two dNFS data paths:

Figure 56.    Single NAS server in oranfstab

The following example shows an Oracle instance using two channels for NAS server ORA-ASM-NFS1.

Figure 57.    Displaying the dNFS channels

Multiple server stanzas

The following oranfstab file contains two dNFS data paths to two NAS servers. Each NAS server services a different database.

Figure 58.    Multiple NAS servers in oranfstab

dNFS is enabled and disabled with the following commands from the Linux user owning the ORACLE_HOME.

Enabling dNFS

cd $ORACLE_HOME/rdbms/lib

make -f ins_rdbms.mk dnfs_on

Disabling dNFS

cd $ORACLE_HOME/rdbms/lib

make -f ins_rdbms.mk dnfs_off

If oranfstab does not exist, dNFS creates a single dNFS channel for entries found in /etc/mtab that are required for the running database. This single dNFS channel is used for both dNFS control and dNFS data traffic. The local IP address used for the dNFS channel is the IP address used to mount the NFS share. In Oracle, the dNFS channel has a name equal to the local IP address of the mount entry in /etc/mtab.

Note: If oranfstab is not used, dNFS load balancing and scaling are not available. For more information, see Benefits of using oranfstab.

Figure 59 shows the mount point, /etc/fstab, and /etc/mtab entries for the single NFS share used in this paper:

Figure 59.    Mount point, /etcfstab, /etc/mtab entries for Unity NFS share

Figure 60 shows Oracle referring to the name (filer) of the single channel as the IP address used to mount the Unity NFS export. Because a local:path IP address mapping is not defined in file oranfstab for the mount IP, dNFS reports the local IP in the alert log as a null value. In this case, dNFS uses the IP address used to mount the NFS share for dNFS data traffic.

Figure 60.    Database using mount IP address for dNFS control and data traffic

If multiple channels to a NAS server are needed for increased dNFS bandwidth, automatic dNFS data traffic load balancing, or automatic dNFS channel failover, the file oranfstab is required. dNFS automatically performs load balancing across all specified channels defined in oranfstab. If one channel fails, dNFS reissues I/O commands over any remaining available channel for that NAS server.

When oranfstab exists, dNFS control traffic uses the IP address used to mount the NFS share. dNFS data traffic is balanced across the local:path IP address mappings of the NAS server stanza specified in oranfstab to which the database belongs. local IP addresses are set to the IP address of the Ethernet interfaces on the database server that are to be used for the database. path IP addresses must be the IP addresses of the Unity NAS server network interfaces.

Figure 61.    dNFS local:path IP address mappings

A dNFS channel for each of the local:path IP address mappings is created in the database instance. For each channel, the value of the NAS server name in oranfstab is recorded in the dNFS channel metadata.

Figure 62.    dNFS filer name is the value of SERVER in oranfstab

Figure 63.    dNFS NAS server name is the value of SERVER in oranfstab

This section discusses the following topics:

• Verifying the database is using dNFS
• Displaying database files that dNFS has opened
• Verifying the instance is using the configured dNFS network endpoints
• Verifying Linux network routes are being used for dNFS data traffic

Verifying the database is using dNFS

To verify that the database instance is using dNFS, check the database alert log for the string running with ODM. If the string is found, as shown in Figure 64, the instance was started with the ODM library containing the direct NFS driver:

Figure 64.    Check if the database instance is running with the ODM library

Displaying database files that dNFS has opened

The database compares the datafile names with the NFS mount to see if the datafiles can be used by dNFS. Any datafile that dNFS has open resides in v$dnfs_files. To verify that dNFS has opened files residing on the NFS share, run the following SQL*Plus command:

Figure 65.    Displaying database files that dNFS has open

Note: Only database files that dNFS has open will be displayed in v$dnfs_files.

Verifying the instance is using the configured dNFS network endpoints

To verify the correct network endpoints are being used with dNFS, look for the string Direct NFS: channel id in the database alert log. The path and local IP addresses shown in the alert log should match all the local:path mappings in file oranfstab for the appropriate NAS server hosting the database.

If Oracle automatically detects the local host interface because oranfstab is not defined, ensure that the chosen interface is intended to be used for the dNFS channel.

Figure 66 shows that the database instance created a dNFS channel for each local:path mapping specified in oranfstab:

Figure 66.    dNFS channels and corresponding oranfstab local:path mappings

Verifying Linux network routes are being used for dNFS data traffic

If the operating system network routes are set up correctly for dNFS, a traceroute from the database server can reach the NAS network interface in Unity. Also, netstat should report increasing TX/RX metrics on the interfaces intended for dNFS during database activity.

If dNFS is enabled but oranfstab is not configured, the Ethernet traffic for dNFS will reside on the interface used to mount the NAS share.

With or without oranfstab, dNFS activity will be displayed as changes to RX-OK and TX-OK values from netstat.

Figure 67 shows two hops between the endpoints in the routes:

Figure 67.    Traceroutes between Unity NAS link aggregation interfaces and database server network interfaces

Figure 68 shows netstat send/receive (TX/RX) statistics for the database server Ethernet interfaces. The statistics can be monitored during database activity to ensure that the interfaces for dNFS traffic have TX/RX activity.

eno3 is the public Ethernet interface. Low values of RX and TX for eno3 are a good indication that the public Ethernet interface is not used by dNFS.

Figure 68.    Netstat statistics of Ethernet interfaces used by xNFS and dNFS

Note: The netstat statistics are counters that are only reset with a system boot.

The following SQL*Plus output shows dNFS send/receive (TX/RX) statistics for each dNFS channel. If dNFS is correctly configured, all channels appear in the output, and have read and write activity.

Figure 69.    Current dNFS statistics

Eight dNFS dynamic performance views are available in Oracle for monitoring ODM NFS storage devices. Four of the eight views are for Standalone deployments, and four are for RAC deployments. For a full description of the dynamic tables for Standalone deployments, see Oracle Database—Database Reference.

Table 11.  Standalone deployment dNFS dynamic performance views

Unity can create point-in-time snapshots of one or more volumes. Snapshots are space-efficient because they consist of pointers to frozen data blocks and use redirect-on-write technology. They consume no extra space unless, for example, a thin clone is created from a snapshot and mapped to a host, and new data is written to the thin clone.

Snapshots provide a fast and space-efficient way to protect Oracle databases. When using snapshots with Oracle databases, there are important considerations to ensure a successful database recovery.

• All LUNs of an Oracle database must be protected as a set using the consistency group feature. The consistency group will ensure that the snapshot is taken at the exact same time on all LUNs in that group. For NFS file systems that support an Oracle database, to ensure database consistency, the entire database must exist on the file system being snapped.
• Snapshots do not replace Oracle RMAN for regular database backup. However, it offers additional protection to the database and allows offloading RMAN processing to an alternate host.
• Snapshots can be taken on demand manually or automatically based on a schedule defined on the LUN file system. It is recommended to put the database in hot backup mode before taking a snapshot and end backup mode after the snapshot is taken.

Note: Snapshots increase the overall CPU load on the system and increase the overall drive IOPS in the storage pool. Snapshots also use pool capacity to store the older data in the snapshot, which increases the amount of capacity used in the pool until the snapshot is deleted. Consider the overhead of snapshots when planning both performance and capacity requirements for the storage pool.

• Before enabling snapshots on a storage object, it is recommended to monitor the system and ensure that existing resources can meet the additional workload requirements. (See “Hardware Capability Guidelines” section and Table 2 in the Dell Unity: Best Practices Guide.)
• Enable snapshots on a few storage objects at a time, and then monitor the system to be sure it is still within the recommended operating ranges before enabling more snapshots. Additional information can be found in the Dell Unity: Snapshots and Thin Clones document.

When recovering the database from a snapshot, the Dell Unity system offers two methods to recover a point-in-time copy of the database: restore and attach to host.

With the restore method, the data of the original LUNs are replaced in place on the original server where the snapshot was taken. The restore method is the simplest and fastest recovery method because there is no copying data, and no configuration or modification is required. The overall process to restore a snapshot is described as follows.

1. Terminate all user connections and shut down the database to be restored.

2. In Dell Unisphere, identify and select the snapshot in the LUN Snapshots properties page or in the Consistency Group Snapshots properties page.
3. Choose Restore from the More Actions drop-down menu.
4. After the restore operation is completed, restart the database on the host.
5. Oracle automatically performs database recovery during the startup.

6. Verify the data in the database.

When only a subset of the data needs to be restored, a point-in-time copy of the database can be mounted on an alternate host. Administrators can export and import the data back to the original database. The original database can remain running but access to the corrupted data should be restricted. To the operating system, the LUNs presented by the snapshot will have different WWNs than the original LUNs. However, the ASM headers are the same as the original ASM devices. It is not recommended to attach these LUNs on the same host as the original database. Attaching the LUNs on the same host as the original database will create confusion to Oracle and increase the risk of writing to the wrong LUNs accidentally. The overall recovery process:

1. Prepare the destination host to receive the LUNs.
   1. Configure the operating system similarly as the original host.
   2. Install the same version of Oracle software on the destination host.
   3. Install the same storage software (ASMLib, ASMFD) on the destination host.

2. In Unisphere, identify and select the snapshot in the LUN Snapshots properties page or in the Consistency Group Snapshots properties page.
3. Choose Attach to host from the More Actions drop-down menu.
4. Select the destination host and allow Read/Write access.
5. After the snapshot is attached to the host, scan for the LUNs using rescan-scsi-bus.sh -forcerescan or -a.
6. Set ownership, group membership, and permission on the LUNs. It is possible to set ownership and membership with chown, and permission with chmod, but the change will not be persistent across reboots.
7. Scan for ASM devices.

For ASMFD:

# asmcmd afd_scan

# asmcmd afd_lsdsk

For ASMLIb;

# oracleasm scandisks

# oracleasm listdisks

8. Mount the ASM disk groups.
9. Copy the database init parameter file to the destination host.
10. Create the database log directories on the destination host.
11. Start up the database in SQL*Plus.
12. Restore data using one of the following methods. Data is extracted from the destination database and imported into the original database.

a. RMAN

b. Datapump

c. Copy data using the database link

Find more information in the Oracle Backup and Recovery User's Guide.

13. Once the recovery is complete, shut down the database copy.
14. Dismount the ASM disk groups.
15. Remove the snapshot LUNs from the destination host.
16. Remove host access of the snapshot LUNs in Unisphere.

Thin clones are based on snapshot technology and are the preferred way to make read/write copies of databases. Similar to regular LUNs, many of the data services, such as snapshots, replications, and host I/O limit, are also available to thin clones. When thin clones are first created, they consume no storage because they share the same blocks as their parent snapshot at the beginning. As new data is written or changes are made to the existing data, new data blocks are allocated and tracked separately from the parent. The data on the thin clones are the same as the parent LUNs but they have different LUN IDs and WWNs. To the operating system, they appear to be different LUNs. However, when Oracle scans for the ASM headers, they contain the same labels and disk group information as the original LUNs. Therefore, it is recommended to attach thin clones on an alternate host to avoid confusion in Oracle and risks in overwriting data on the wrong LUNs.

Thin clone use cases:

• Create full-size development and test environments from production
• Test new code, patches, or data changes in a production replica
• Offload backup and restore processing

Find additional information about thin clones can be found in the Dell Unity: Snapshots and Thin Clones document.

The following shows the overall process to create a database copy using a thin clone.

1. Prepare the destination host to receive the LUNs.
   1. Configure the operating system similarly as the original host.
   2. Install the same version of Oracle software on the destination host.
   3. Install the same storage software (ASMLib, ASMFD) on the destination host.

2. In Unisphere, identify and select the LUN or the consistency group.
3. Choose Clone from the More Actions drop-down menu.
4. Select a snapshot to clone from.

Note: Only snapshots with no auto-delete policy and no expiration time are eligible for selection. Remove the auto-delete policy and expiration time on the snapshot before attempting the Clone action.

5. Follow the wizard to configure the thin clone’s name, host I/O Limit, host access, snapshot policy, and replication.
6. After the thin clone LUNs are attached to the destination host, scan for the LUNs using rescan-scsi-bus.sh -forcerescan or –a.
7. If the database clone is intended for long-term use, configure multipath and persistent ownership and permission on the thin clone LUNs.
8. Scan for ASM devices.
9. Mount the ASM disk groups.
10. Copy the database init parameter file to the destination host.
11. Create the database log directories on the destination host.
12. Start up the database in SQL*Plus.

It is possible to refresh thin clones from the same snapshot or a different snapshot. Refreshing thin clones from the same or different snapshot provides an easy way to reset an environment consistently to a baseline or switch to a different point-in-time copy. The refresh process is quick and simple as only the pointers are updated, and no data is being copied. The process to refresh a thin clone is as follows:

1. Shut down the database copy that is using the thin clone LUNs.

2. Dismount the ASM disk groups
3. In Unisphere, select the thin clone and select the Refresh action in the More Actions menu.
4. A snapshot is automatically created of the thin clone to preserve the thin clone data.
5. Select a snapshot to refresh from.

Note: Snapshots that have auto-delete policy or expiration time set are not eligible for selection. Remove the auto-delete policy and expiration time on the snapshot first before starting the Refresh action.

6. After the refresh is completed, perform a rescan on the operating system.
7. Perform a rescan on ASM devices.
8. Mount the ASM disk group.
9. Start up the database.
10. Verify the data in the database.

Creating a high availability solution for Oracle databases often involves creating a copy of the data on another storage device and synchronizing that data in some manner. Dell Unity replication provides data synchronization between Dell Unity systems. Data is replicated at the consistency group or at the LUN and file system providing a choice of replication settings on a per-volume basis. Using Dell Unity replication can be an effective way to protect Oracle databases due to the flexibility and configuration options that it provides. The variety of options provide a robust way to develop a replication scheme that provides the proper mix of performance and bandwidth efficiency while still meeting RTO and RPO requirements.

When using Dell Unity replication to protect Oracle databases that are on multiple volumes, contain all ASM devices for a database within a consistency group. Then configure replication on the consistency group.

Dell Unity storage supports both asynchronous and synchronous replication. A flash tier is recommended (in a hybrid pool) for both source and destination pools where replication will be active.

Asynchronous replication

Asynchronous replication takes snapshots on the replicated storage objects to create the point-in-time copy, determining the changed data to transfer and maintain consistency during the transfer. Consider the overhead of snapshots when planning performance and capacity requirements for a storage pool that will have replication objects.

Setting smaller RPO values on replication sessions will not make them transfer data more quickly but will result in more snapshot operations. Choosing larger RPOs, or manually synchronizing during non-production hours, may provide more predictable levels of performance. Additional information can be found in the Dell Unity: Replication Technologies and Dell Unity: Configuring Replication documents.

Synchronous replication

Synchronous replication transfers data to the remote system over the first Fibre Channel port on each SP. When planning to use synchronous replication, it may be appropriate to reduce the number of host connections on this port. When the CNA ports are configured as FC, CNA port 4 is defined as the synchronous replication port. If the CNA ports are configured as 10 GbE, port 0 of the lowest numbered FC I/O module is the replication port. Additional information can be found in the Dell Unity:  Replication Technologies and Dell Unity: Configuring Replication documents.

In addition to the snapshots and replication provided by Dell Unity systems, Dell Technologies offers additional data protection software that integrates with the Dell Unity data protection features. The software is optional and can be used to enhance the overall application protection.

Dell AppSync is software that enables integrated Copy Data Management (iCDM) with the Dell primary storage systems, including Dell Unity arrays. It supports many applications, including Oracle, and storage replication technologies. For the latest support information, refer to the AppSync Support Matrix at the Dell E-lab Navigator.

AppSync simplifies and automates the process of creating and using snapshots of production data. By abstracting the underlying storage and replication technologies, and through application integration, AppSync empowers application owners to manage data copy needs themselves. The storage administrator, in turn, need only be concerned with initial setup and policy management, resulting in a more agile environment.

Additional information about AppSync can be found in the AppSync User and Administration Guide and the AppSync Performance and Scalability Guidelines.

Dell RecoverPoint virtual edition provides continuous data protection with multiple recovery points to restore applications instantly to a specific point in time. RecoverPoint virtual edition consists of RecoverPoint Appliance (RPA) software deployed as a virtual appliance in an existing VMware ESXi VM environment. RecoverPoint virtual edition is a flexible deployment option which offers maximum simplicity with no dependency on a physical appliance, able to lower TCO.

dell.com/support is focused on meeting customer needs with proven services and support.

Storage technical documents and videos provide expertise that helps to ensure customer success on Dell storage platforms.

The Dell Unity area of the Dell Technologies Info Hub provides white papers and videos.

The following referenced or recommended Dell Technologies publications and resources are at dell.com or infohub.delltechnologies.com. If you do not have access to a document, contact your Dell Technologies representative.

• Dell PowerEdge R730 and R730xd Technical Guide
• Configuring Dell Networking S5248F-ON switches
• CloudIQ: A Detailed Review | Dell Technologies Info Hub
• PowerConnect 5500 Series User Guide (dell.com)
• PowerConnect 5500 Series Getting Started Guide (dell.com)
• Dell Networking S5000 Getting Started Guide
• ‎REST API and Dell Technologies Storage - Unity | DELL Technologies
• Live Optics for Service Providers
• Live Optics - Real-world data for IT decisions : Live Optics
• Support for PowerPath | Overview | Dell US
• PowerPath Installation and Administration Guide
• Dell Technologies E-Lab Navigator - Host Connectivity Guide for Linux
• Dell AppSync User and Administration Guide
• Unity XT Info Hub
• Dell Unity Info Hub Product documents and information
• Dell Unity OE 5.2: Technical Primer
• Dell Unity OE 5.2: Technical Primer (delltechnologies.com)
• Dell Unity OE Overview
• Dell Unity XT: Introduction to the Platform - H15084
• Dell Unity XT: Introduction to the Platform - H17782.5
• Dell Unity XT: Best Practices Guide
• Dell Unity: Best Practices Guide
• Dell Unity: Data Reduction
• Dell Unity: Data Reduction Analysis
• Dell Unity: FAST Technology Overview
• Dell Unity: Unisphere Overview
• Dell Unity: Unisphere Central
• Dell Unity: High Availability
• Dell Unity: Replication Technologies
• Dell Unity: MetroSync
• Dell Unity: DR Access and Testing
• Dell Unity: MetroSync and Home Directories
• Dell Unity: Snapshots and Thin Clones
• Dell Unity: Data at Rest Encryption
• Dell Unity: Performance Metrics
• Dell Unity: Dynamic Pools
• Dell Unity: NAS Capabilities
• Dell Unity: File-Level Retention (FLR)
• Dell Unity: Configuring Pools
• Dell Unity: Compression
• Dell Unity: Drive Support Matrix
• Dell Unity XT Unified Storage Solutions
• Dell Unity Family Unisphere Management REST API Programmer's Guide
• Dell Unity Family Unisphere Management REST API Reference Guide
• Dell Unity Family Unisphere Command Line Interface User Guide
• Dell Unity: Service Commands
• Dell Unity: Family Service Commands Technical Notes
• Dell EMC Unity™ Family Configuring NFS File Sharing
• Dell EMC Unity™ Family Configuring hosts to access NFS
• Dell Unity™ Family Configuring High Availability
• Dell EMC Unity: Best Practices Guide (delltechnologies.com)
• Dell Unity OE 5.2: Technical Primer (delltechnologies.com)
• Dell EMC Unity: Best Practices Guide (delltechnologies.com)
• Dell EMC Unity: Best Practices Guide (delltechnologies.com)
• Dell EMC Host Connectivity Guide for Linux (delltechnologies.com)
• Dell EMC Host Connectivity Guide for Linux (delltechnologies.com)
• E-Lab Interoperability Navigator 2.0-HOME (dell.com)
• Drawing1 - Copy - Unity 5-3.vsdx (sharepoint.com)

See the following referenced or recommended Oracle resources at the Oracle Online Documentation Portal:

• Oracle Enterprise Manager
• Oracle ASMLib
• Oracle Automatic Storage Management Administrator’s Guide 21c, F30296-05, June 2022
• Oracle Database—Database Installation Guide, 21c for Linux, F31591-06 November 2022
• Oracle Database—Database Administrator’s Guide, 21c, E31835-14 September 2022
• Oracle Database—Database Concepts 21c, F31733-04 August 2021
• Oracle Database—Database Performance Tuning Guide 21c, F32091-03 August 2021
• Oracle Database—SQL Tuning Guide, 21c, F31828-10 August 2022
• Oracle Database – Database Reference, 21c, F31274-18, February 2023
• Oracle Database – Multitenant Administrator’s Guide, 21c, F31734-10, August 2022
• Oracle Database – Utilities, 21c, F30732-13, June 2023
• Oracle Database – Database Net Services Administrator’s Guide, 21c, F31801-05, August 2022
• Oracle Database – Database Net Services Reference, 21c, F31800-06, February 2023
• Oracle Database – Security Guide, 21c, F31285-15, March 2023
• Oracle Grid Infrastructure—Grid Infrastructure Installation and Upgrade Guide, 21c for Linux, F31403-04 July 2022
• Oracle Clusterware – Clusterware Administration and Deployment Guide, 21c, F30298-02, June 2022
• Oracle Real Application Clusters – Real Application Clusters Installation Guide, 21c for Linux and UNIX, F31402-04 June 2022
• Oracle Real Application Clusters – Real Application Clusters Administration and Deployment Guide, 21c, F32002-03, May 2022
• Oracle Database, Database Performance Tuning Guide, 21c, F32091-03, August 2021
• I/O Configuration and Design (oracle.com)
• Use of Oracle Screenshots
• Oracle branding
• How Autofs Works

See the following referenced resources at My Oracle Support (Oracle support license and login required):

• Direct NFS: FAQ (Doc ID 954425.1)
• How To Setup dNFS (Direct NFS) On Oracle Release 11.2 (Doc ID 1452614.1)
• Step by Step - Configure Direct NFS Client (dNFS) on Linux (Doc ID 762374.1)
• How to Setup Direct NFS Client Multipaths in Same Subnet (Doc ID 822481.1)
• How to configure DNFS to use multiple IPs (Doc ID 1552831.1)
• How to configure DNFS to use multiple IPs using different subnets (Doc ID 1528148.1)
• Best Practices: How to configure DNFS client using Single and Multiple Subnets (Doc ID 2246252.1)
• rp_filter for multiple private interconnects and Linux Kernel 2.6.32+ (Doc ID 1286796.1)
• Creating File Devices On NAS/NFS FileSystems For ASM Diskgroups. (Doc ID 1620238.1)
• How to Mount NFS File Systems Using 'autofs' (Doc ID 2188323.1)
• Linux operating system Service 'autofs' (Doc ID 551692.1)
• Mount Options for Oracle files for RAC databases and Clusterware when used with NFS on NAS devices (Doc ID 359515.1)
• NFS-Filesystems: Hard vs Soft mounts Explained (Doc ID 1003375.1)
• ASM Certification On Thin-Provisioned NFS Files (Doc ID 1601897.1)
• How To Create ASM Diskgroups using NFS/NAS Files? (Doc ID 731775.1)
• NFS Performance Decline Introduced by Mount Option "actimeo=0 (Doc ID 1164673.1)
• How to Optimize NFS Performance with NFS options (Doc ID 397194.1)
• What is ASM rebalance compact Phase and how it can be disabled (Doc ID 1902001.1)
• ASMFD (ASM Filter Driver) Support on operating system Platforms (Certification Matrix) (Doc ID 2034681.1)
• Supported Backup, Restore and Recovery Operations using Third Party Snapshot Technologies (Doc ID 604683.1)
• How to prepare the Oracle database for Snapshot Technologies and ensure a consistent recovery (Doc ID 221779.1)
• Is It Possible To Configure 2-factor Or Multi Factor Authentication For Oracle DBA Accounts? (Doc ID 2323931.1)
• What is the Orion I/O Test Tool? (Doc ID 1905698.1)

Other assets that are referenced in the paper include:

• Red Hat Enterprise Linux 8, Optimizing system throughput, latency, and power consumption
• DB-Engines Ranking of Relational DBMS
• Using autofs with Oracle Recovery Manager (RMAN) and dNFS - Ron Ekins' - Oracle Technology, DevOps and Kubernetes Blog
• DNFS: wtmax = /proc/fs/nfsd/max_block_size (pythian.com)
• NFS file system attributes for z/OS UNIX file systems - IBM Documentation
•  

Referenced or recommended I/O benchmark resources:

• vdbench download and documentation
• FIO download
• FIO output explained
• atop and netatop homepage
• collectl sourceforge homepage
• SLOB Blog
• HammerDB homepage
• Swingbench homepage

The following referenced or recommended Veritas resources are at Veritas Online Support:

• Storage Foundation Administrator’s Guide
• Storage Foundation Tuning Guide