AppSync: When the maximum number of vCenter (VC) Connections are in use, subsequent connection requests will wait and may timeout while waiting.

More than the allowed 4 VC connections are needed simultaneously.
This can be observed by finding the similar messages to the following in the AppSync Server log:

->>> getURLConnection::https://vcenter-IP/sdk reached maximum connection limit. Wait for sometime to get a free connection
->>> EJB default - 1Unlocking entityID:https://vcenter-IP /sdk for update
->>> --getConnection::Waiting on BUSY connection for entity:https://vcenter-IP/sdk

If the service plan that is waiting for a VC connection times out, then the service plan may become stuck in the “in-progress” state.

The timeout of a service plan can be observed in the AppSync Server log by the following:

WARN [Transaction Reaper] [com.arjuna.ats.arjuna] [] [] ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0ae651aa:1c135b84:60be4e22:1f2b2c in state  RUN
WARN [Transaction Reaper Worker 0] [org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorTrackingImpl] [] [] HHH000451: Transaction afterCompletion called by a background thread; delaying afterCompletion processing until the original thread can handle it. [status=4]
WARN [Transaction Reaper Worker 0] [com.arjuna.ats.arjuna] [] [] ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0ae651aa:1c135b84:60be4e22:1f2b05
WARN [Transaction Reaper Worker 0] [com.arjuna.ats.arjuna] [] [] ARJUNA012095: Abort of action id 0:ffff0ae651aa:1c135b84:60be4e22:1f2b2c invoked while multiple threads active within it.
WARN [Transaction Reaper Worker 0] [com.arjuna.ats.arjuna] [] [] ARJUNA012108: CheckedAction::check - atomic action 0:ffff0ae651aa:1c135b84:60be4e22:1f2b2c aborting with 1 threads active!
WARN [Transaction Reaper Worker 0] [com.arjuna.ats.arjuna] [] [] ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0ae651aa:1c135b84:60be4e22:1f2b2c

Subsequent runs of the service plan that is stuck in “in-progress” fails, stating the service plan is already running.

Increasing the transaction timeout which is specified in seconds may resolve the timeout issue.
The default is 5 minutes, and it can be increased incrementally if need be. The example below increases the timeout to 20 minutes.

To increase the transaction timeout in the standalone.xml, which is located in C:\EMC\AppSync\jboss\standalone\configuration on the AppSync Server:

• Make a backup of the standalone.xml file.
• Edit the standalone.xml file and increase:

<coordinator-environment default-timeout="300"/>

<coordinator-environment default-timeout="1200"/>

• Restart the three AppSync Server services.

Scheduling vCenter related Service Plans to reduce the simultaneous interaction with vCenter may resolve the issue and eliminate the need to increase the transaction timeout.