PowerProtect Data Manager: Oracle Agent Error "Unable to load Oracle OCI library" Due to Missing Library Files

The following error is logging in ddbmcon*.log or OpAgentSvc*.log which prevents Discovery and backup from running:

Unable to load Oracle OCI library

Library files are missing, or LDD_LIBRARY_PATH is not configured correctly.

Run ddbmcon in debug mode to get additional details on the nature of the error:

While logged in with an Oracle user account with database access, create a file named:

.ddbmcon.debug

Copy the file to these two locations:

• /opt/dpsapps/agentsvc/tmp - This is the default location for the agent service's tmp folder but it may be different.
• $RMAN_AGENT_HOME/tmp

Create a file $RMAN_AGENT_HOME/bin/input.txt.

Add the following text to the input.txt file:

{"APIVersion": "2.3", "command": "DISCOVERY", "arguments": {"discoveryType": "APPLICATION_SYSTEMS", "type": "ORACLE_DATABSE_SYSTEM"}}

Run the following command from $RMAN_AGENT_HOME:

bin/ddbmcon -d9 < bin/input.txt

Inspect the resulting ddbmcon.<epoch time>.<counter>.log debug logs.

Debug file name example:

ddbmcon.1719066794.32076.log

Here is a sequence of missing library errors:

2024-06-14T20:53:40.198Z ddbmcon: build machine = linux86w
2024-06-14T20:53:40.198Z ddbmcon: Oracle library: /u01/app/oracle/product/19.3.0/dbhome_1/lib64/libclntsh.so not found.
2024-06-14T20:53:40.198Z ddbmcon: Loading library: /u01/app/oracle/product/19.3.0/dbhome_1/lib/libclntsh.so
2024-06-14T20:53:40.198Z ddbmcon: dlopen(NULL): failure = libnnz19.so: cannot open shared object file: No such file or directory
2024-06-14T20:53:40.198Z ddbmcon: Unable to load Oracle OCI library.

Use ldd to check library dependencies for the library listed in the 'not found' error as in this example using the output from above:

[root@lab logs]# ldd -v /u01/app/oracle/product/19.3.0/dbhome_1/lib/libclntsh.so.19.1
linux-vdso.so.1 => (0x00007ffe4d7e5000)
libnnz19.so => not found
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fbb3f248000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00007fbb3ef46000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fbb3ed2a000)
libnsl.so.1 => /usr/lib64/libnsl.so.1 (0x00007fbb3eb10000)
librt.so.1 => /usr/lib64/librt.so.1 (0x00007fbb3e908000)
libaio.so.1 => /usr/lib64/libaio.so.1 (0x00007fbb3e706000)
libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007fbb3e4ec000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00007fbb3e11e000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbb435ef000)
libclntshcore.so.19.1 => not found

The output shows that the two highlighted library files are not found. Search the file system with these commands, based on the example output, to determine if the files are on the machine:

find / -type f -name libnnz.so
find / -type f -name libclntshcore.so.

If the files are not on the machine, determine what package should be installed to get the libraries installed. Consult Oracle support if needed.

If libraries are found in a non-default location, configure the LDD_LIBRARY_PATH for the PowerProtect Data Manager agent.

Rename $RMAN_AGENT_HOME/bin/ddbmcon to $RMAN_AGENT_HOME/bin/ddbmcon.orig
Locate the bash executable:

which bash

Create a file ddbmcon.bash with this content:

#!/usr/bin/bash  #determined from `which bash`
export LD_LIBRARY_PATH=<path to OCI libraries>:$LD_LIBRARY_PATH
/home/oracle/opt/dpsapps/rmanagent/bin/ddbmcon.orig "$@"

Copy ddbmcon.bash to ddbmcon

Change the owner on the file ddbmcon to oracle:oinstall or the appropriate [oracle owner]:[oracle install group]

The substituted ddbmcon script sets the LDD_LIBRARY_PATH and then calls the original ddbmcon using the command string that was passed by the calling application.