Avamar Plug-in for Oracle: Avamar Rman plugin backups may fail with avoracle_util:get_oracle_version get_oracle_version_by_CLI() failed.

Avamar Rman plugin backups may fail with avoracle_util:get_oracle_version    get_oracle_version_by_CLI() failed.

Error:   

2017/03/21-19:55:47.02547 [avoracle_assist]  avoracle_util::get_oracle_version_by_select   Entering.
2017/03/21-19:55:47.02746 [avoracle_assist]  avoracle_util::get_oracle_version_by_select   The Oracle user name is
 not set. Exiting.
2017/03/21-19:55:47.02949 [avoracle_assist]  avoracle_util:get_oracle_version    get_oracle_version_by_select() fa
iled.  Trying get_oracle_version_by_CLI().
2017/03/21-19:55:47.03134 [avoracle_assist]  avoracle_util::get_oracle_version_by_CLI   Entering.
2017-03-21 15:55:47 avoracle Info <14190>: sqlplus cmd:<$ORACLE_HOME/bin>/sqlplus var:/opt/AVMRclnt/var/clientlogs
2017-03-21 15:55:47 avoracle Info <7783>: Setting spawn user/group (<default>)
2017-03-21 15:55:47 avoracle Info <7953>: Username set to "<oracle_username>"
2017/03/21-19:55:47.05429 [avoracle_assist]  avspawn::spawn command_line_args:sqlplus -V priority background
2017/03/21-19:55:47.05521 [avoracle_assist]  avspawn::spawn executable_path '<$ORACLE_HOME/bin>/sqlplus'
2017/03/21-19:55:47.05612 [avoracle_assist]  avspawn::spawn execargs[0] '<$ORACLE_HOME/bin>/sqlplus' (arg0)
2017/03/21-19:55:47.05702 [avoracle_assist]  avspawn::spawn execargs[1] '-V' (arg1)
2017/03/21-19:55:47.05791 [avoracle_assist]  avspawn::spawn new command line 'sqlplus -V'
2017/03/21-19:55:47.07329 [avoracle_assist]  avspawn::avspawn(14, stdout) fd:15
2017/03/21-19:55:47.07626 [stdout]  Startup, pid=9 tid=39 stack=fe5fbe37 td=a67180
2017/03/21-19:55:47.07779 [stdout]  avspawnpipe::body stdout
2017/03/21-19:55:47.07787 [avoracle_assist]  avspawn::avspawn(16, stderr) fd:17
2017/03/21-19:55:47.08071 [stderr]  Startup, pid=10 tid=40 stack=fe4dbe37 td=a98f40
2017/03/21-19:55:47.08232 [stderr]  avspawnpipe::body stderr
2017/03/21-19:55:47.16012 [stdout]  avspawnpipe::body EOF  stdout
2017/03/21-19:55:47.16289 [stdout]  avspawnpipe::body stdout exiting
2017/03/21-19:55:47.16389 [stdout]  Thread finished stdout
2017/03/21-19:55:47.16014 [stderr]  avspawnpipe::body EOF  stderr
2017/03/21-19:55:47.16637 [stderr]  avspawnpipe::body stderr exiting
2017/03/21-19:55:47.16744 [stderr]  Thread finished stderr
2017/03/21-19:55:47.16862 [avoracle_assist]  avspawn::run_and_wait calling wait
2017/03/21-19:55:47.16996 [avoracle_assist]  avspawn::wait setting error_num from 0 to 0
2017/03/21-19:55:47.17086 [avoracle_assist]  avspawn::run_and_wait back from wait error_num:0 wait_error:65545
2017/03/21-19:55:47.17175 [avoracle_assist]  called avspawnpipe::~avspawnpipe
2017/03/21-19:55:47.17264 [avoracle_assist]  avspawnpipe::~avspawnpipe done with avspawnpipe::wait stdout
2017/03/21-19:55:47.17364 [avoracle_assist]  called avspawnpipe::~avspawnpipe
2017/03/21-19:55:47.17457 [avoracle_assist]  avspawnpipe::~avspawnpipe done with avspawnpipe::wait stderr
2017/03/21-19:55:47.17612 [avoracle_assist]  avoracle_util::get_oracle_version_by_CLI   The sqlplus command return
ed no output. Exiting.
2017/03/21-19:55:47.17806 [avoracle_assist]  avoracle_util:get_oracle_version    get_oracle_version_by_CLI() faile
d. Exiting.

This article applies if tracing avagent shows the OS is not able to find: libsqlplus.so:    

7744:   stat("/opt/AVMRclnt/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0)  Err#2 ENOENT
7744:   stat("/usr/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/share/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/ccs/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/openwin/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/sadm/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/sfw/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/openwin/sfw/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/dt/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/X11/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/j2se/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/j2se/jre/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/openv/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/openv/java/jre/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/etc/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/etc/security/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFFFFFFFF7F200000
7744:   stat("/opt/SUNWcluster/lib/sparcv9/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/opt/oracle/product/12cR1_tptqa1/lib/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/opt/ORCLcluster/lib//libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/lib/64/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   stat("/usr/lib/64/libsqlplus.so", 0xFFFFFFFF7FFFE7F0) Err#2 ENOENT
7744:   write(2, " l d . s o . 1 :   s q l".., 79)      = 79

How to trace avagent:     

On Solaris:     

• truss -a -vall -e -f -p <avagent_pid> > /tmp/avagent.trc 2>&1

On Linux:     

• strace -a -vall -e -f -p <avagent_pid> > /tmp/avagent.trc 2>&1

Avamar Rman plugin backups are failing because sqlplus is not finding libsqlplus.so which is required for the binary to run.

Workaround:     

1. vi avagent.d
2. Look for line: LD_LIBRARY_PATH

3. Replace this line with:     

• Linux: /usr/local/avamar/etc

• Solaris: /opt/AVMRclnt/etc

4. avagent.d restart

Resolution:     
Contact Dell EMC Avamar support referring to this KB article to obtain a fix for this problem.