AppSync: failed to disable ASM rebalancing on host

After expanding the space or increasing the available disk group space by adding devices to an ASM disk group in oracle there is a process of ASM Rebalancing in Oracle.

If the time it takes Oracle to rebalance exceeds the AppSync SFTP timeout period, the following error will be seen in the Service plan events:
 

Server log:

 DATE> <TIME>ERROR [Thread-336979 (HornetQ-client-global-threads-2040611065)]
[com.emc.archway.service.sshservice.SecureShellSession] [] [] Error while getting file over SFTP: session is down
DATE> <TIME>ERROR [Thread-336979 (HornetQ-client-global-threads-2040611065)] [com.emc.archway.service.sshservice.SecureShellSession] [] [] Error while getting file over SFTP: No such file
DATE> <TIME>ERROR INFO [Thread-336979 (HornetQ-client-global-threads-2040611065)] [com.emc.archway.service.sshservice.SshServiceBean] [] [] Retrying get of file..0f6bc923-7c8c-474d-bb38-ac531ba21xxx.rsp

DATE> <TIME>ERROR INFO [Thread-336979 (HornetQ-client-global-threads-2040611065)] [com.emc.archway.service.eventservice.EventServiceBean] [] [] EVENT [GENL_000023]: An error occurred while communicating with the AppSync Host Plug-in.  Please check to see if agent service is running, and restart if necessary. (METADATA: TYPE-ERROR, TIME-2016-12-20 00:40:05.474+0100NATIVETIME-2016-12-20 00:40:05.474+0100, HOST-sa5emp01, PHASE-Create local copy, THREAD=Thread-336979 (HornetQ-client-global-threads-2040611065), USER-null, CATEGORY-GENERIC, SESSIONID-null)
DATE> <TIME>ERROR INFO [Thread-337534 (HornetQ-client-global-threads-2040611065)] [com.emc.archway.commands.event.ArchwayEventListenerHelperCommandBean] [] []  Event listener received: GENL_000023
DATE> <TIME>ERROR ERROR [Thread-336979 (HornetQ-client-global-threads-2040611065)] [com.emc.archway.commands.application.oracle.QuiesceOracleCommandBean] [] [] Failed to disabe ASM rebalancing on host: ip-10-237-0-xxx.env. Copy may not be consistent. 

 

The Cause of this issue for Versions of AppSync prior to 3.0.2 is because the time it took for Oracle to perform rebalance exceeded the default AppSync SFTP timeout period and AppSync will not try to stop the rebalance before taking a copy.

A new feature is added to AppSync 3.0.2 which will switches off ASM rebalancing before taking a snapshot or clone of the underlying disks and turns it back on, after the copy is created.
Upgrade to AppSync Version 3.0.2 or higher to resolve this issue.

It is still possible the AppSync SFTP timeout will be exceeded should the stopping or restarting of the ASM rebalance not happen quickly. In such cases, increasing the SFTP timeout may be necessary which can be done by Customer Support. The default SFTP timeout is 3600 seconds.
Please call in a service request and quote this article if you feel as though the SFTP timeout needs to be increased.

 

It is important to know why the rebalance is running. Rebalancing can happen for different reasons: if there is failure in the disk, database is out of sync or no available space.

If the rebalancing is happening not because of adding disk to available disk group, this article cannot be referred to, hence there might be possible hang in the database or DU, and the customer needs to engage the DBA team to fix it.

Please refer https://docs.oracle.com/cd/B28359_01/server.111/b31107/asmdiskgrps.htm#OSTMG10040 for more information on adding and managing ASM disk group and rebalancing.

If there is a hang after adding devices to the ASM disk group please refer to article 000359509.