AppSync: "VPLEXClientFactory - could not obtain lock within 5000/10000MILLISECONDS"
Summary: A service plan in which VPLEX can fail with the error "VPLEXClientFactory - could not obtain lock within 5000MILLISECONDS/10000MILLISECONDS." Also, while editing the service plan and in the create-copy phase, selecting the VPLEX cluster, the error appears as above. ...
Symptoms
Service plan fails, or errors appears while editing the service plan settings. The error is as below:
When on AppSync 4.1.0.x, the error is:
concurrent access timeout on VPLEXClientFactory - could not obtain lock within 5000MILLISECONDS
When on AppSync 4.2.0.x, the error is:
concurrent access timeout on VPLEXClientFactory - could not obtain lock within 10000MILLISECONDS
AppSync Server logs show the following error:
<DATE & TIME> ERROR [default task-38] [org.jboss.as.ejb3.invocation] [AppSync Server] [] WFLYEJB0034: EJB Invocation failed on component VPLEXClientFactory for method public com.emc.archway.service.vplex.VPLEXClient com.emc.archway.service.vplex.VPLEXClientFactory.getClient(com.emc.archway.context.ExecutionContext,com.emc.archway.objectmodel.storage.vplex.VPLEXManagementServer) throws com.emc.archway.errorMessages.ArchwayException: javax.ejb.ConcurrentAccessTimeoutException: WFLYEJB0241: EJB 3.1 PFD2 4.8.5.5.1 concurrent access timeout on VPLEXClientFactory - could not obtain lock within 10000MILLISECONDS
Cause
The VPLEX response is not received within the AppSync hard coded timeout.
This seems to be a VPLEX performance issue. VPLEX getting slow to respond and AppSync is not able to fetch the VPLEX client within the hard coded 5s or 10s timeout window. Hence, this is causing the service plan failure.
This error occurs when AppSync attempts to rediscover VPLEX information before displaying it in the UI. Due to VPLEX performance issues, the discovery process times out, resulting in the error.
This timeout exists in AppSync 4.1 and 4.2.
AppSync 4.1.0.x has a timeout value of 5000 MILLISECONDS (5 seconds).
AppSync 4.2.0.x has a timeout value of 10000 MILLISECONDS (10 seconds).
Resolution
Upgrade to AppSync 4.3 or later where the timeout check is eliminated.