AppSync: Oracle Databases Running on ASM Disk are Detected as Offline

A user sees the database status as "OK" under AppSync GUI -> Copy management -> Copies -> Oracle Application.
But the service plan protecting this database fails on the discovery phase. A user sees the below errors in event logs.

<DATE & TIME>,Application discovery,21cc64f3-795a-42d0-aab5-818615d0e645,MILE_000005,<AppSync Server>,Application discovery phase for <Database Name> completed with errors
<DATE & TIME>,Application discovery,21cc64f3-795a-42d0-aab5-818615d0e645,ORCL_000104,<AppSync Server>,During discovery, AppSync detected that the following database(s) were offline: <Database Name>. As a result, they will not be available for protection.

In server logs, AppSync sends the checkdatabase command and then failing:

<Date  TIME> INFO [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.service.eventservice.EventServiceBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> EVENT [ORCL_000064]: Protecting the Oracle database <Database Name> with hot backup mode disabled.(METADATA: TYPE-INFO, TIME-2022-03-03 10:34:10.227-0600NATIVETIME-2022-03-03 10:34:10.227-0600, HOST-<AppSync Server>, PHASE-Application discovery, THREAD=EE-ManagedExecutorService-default-Thread-5, USER-admin, CATEGORY-GENERIC, SESSIONID-)
<Date  TIME> DEBUG [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.commands.application.oracle.DiscoverOracleDatasetsCommandBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> Started Executing isASMRequired 
<Date  TIME> DEBUG [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.commands.application.oracle.DiscoverOracleDatasetsCommandBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> DB Name:  <Database Name>
<Date  TIME> DEBUG [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.commands.application.oracle.DiscoverOracleDatasetsCommandBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> Completed Executing isASMRequired....
<Date  TIME> DEBUG [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.commands.application.oracle.DiscoverOracleDatasetsCommandBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>>  Is ASM required for OracleAPP? : true
<Date  TIME> INFO [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.commands.application.oracle.DiscoverOracleDatasetsCommandBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> ASM Rediscover Dataset Required? true
<Date  TIME> INFO [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.commands.application.oracle.DiscoverOracleDatasetsCommandBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> Checking ASM and Database instances on host: 192.168.8.206
<Date  TIME> INFO [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.service.oracleservice.OracleServiceBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> Beginning task for checkDatabase operation on <source host> task=1f5795f8-1932-4eec-8c0b-40a7e74601dc

<Date  TIME> INFO [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.service.sshservice.SshServiceBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> Copying AppSync request...<ir_message ir_requestId="00000" ir_sessionId="00000" ir_status="0" ir_type="ArchwayRequest" master_id="0" phaseName="" pollMode="SYNC" taskId="0" timeToWait="0"><operation><agent>Oracle</agent><command>checkDatabase</command><parameters><parameter><paramName>oracleSid</paramName><paramValue><Database Name>1</paramValue></parameter><parameter><paramName>oracleHome</paramName><paramValue>/oracle/app/oracle/product/19.0.0.0/dbhome_1</paramValue></parameter></parameters></operation><version>1.0</version></ir_message>

<Date  TIME> INFO [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.services.agent.AgentServiceBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> String response: <ir_message ir_requestId="00000" ir_sessionId="00000" ir_status="0" ir_type="ArchwayRequest" master_id="0" phaseName="" pollMode="SYNC" taskId="0" timeToWait="0"><operation><agent>Oracle</agent><command>checkDatabase</command><parameters><parameter><paramName>oracleSid</paramName><paramValue><Database Name>1</paramValue></parameter><parameter><paramName>oracleHome</paramName><paramValue>/oracle/app/oracle/product/19.0.0.0/dbhome_1</paramValue></parameter></parameters><result return="0"/><responses><response><![CDATA[<oracleStatus><status>true</status><state>OPEN</state></oracleStatus>]]></response></responses></operation><version>1.0</version></ir_message>

<Date  TIME> INFO [EE-ManagedExecutorService-default-Thread-5] [com.emc.archway.service.eventservice.EventServiceBean] [<AppSync Server>] [] 0fc79d48-60a7-4960-a9dd-558fd6e4e4c8->>> EVENT [ORCL_000104]: During discovery, AppSync detected that the following database(s) were offline: <Database Name>. As a result, they will not be available for protection.

In the agent logs, the database is online as the state is as OPEN.

<Date  TIME> acp.py[116]:main() Command: checkDatabase
<Date  TIME> oracle.py[1727]:__init__() Info:Setting LD_LIBRARY_PATH=/oracle/app/oracle/product/19.0.0.0/dbhome_1/lib
<Date  TIME> oracle.py[1955]:runProgram() Running ASM command  ::: ['/oracle/app/oracle/product/19.0.0.0/dbhome_1/bin/orabase']
<Date  TIME> oracle.py[5215]:_sqlQuery() SQL*Plus Query: select 'AppSync_entry:'|| status from v$instance;
<Date  TIME> oracle.py[5243]:_sqlQuery() SQL*plus > SQL> SQL> SQL> SQL> SQL> 
<Date  TIME> oracle.py[5243]:_sqlQuery() SQL*plus > 'APPSYNC_ENTRY:'||STATUS
<Date  TIME> oracle.py[5243]:_sqlQuery() SQL*plus > --------------------------------------------
<Date  TIME> oracle.py[5243]:_sqlQuery() SQL*plus > AppSync_entry:OPEN

The database running on ASM disk and appsync is not able to find ASM home since it queries the oratab file for oracle home and ASM home. Whereas the oratab file did not have an entry for ASM. It only has an entry for the database.

Add the entry into the oratab file for both DB and ASM instance.

Format: <ASM SID>:<grid_home>:Y|N

Sample entry from our lab setup:

DB1:/u01/app/oracle/product/19c/db_home:N # entry for DB
+ASM:/u01/app/grid/product/19c/grid_home:N # entry for ASM instance