AppSync Mount Copy Phase Failure with VMAX Operation Failed Error Caused by GenericJDBCException
Summary: Mount phase failing due to an insufficient memory issue for Postgres with the error in server logs "ERROR: out of memory Detail: Failed on request of size"
Symptoms
Service Plan is failing in mount phase while running an internal query to get the VMAX device details and copy details for linking. For the specific issue this article addresses, the following query is seen in server logs just before the issue:
Query string is:SELECT n FROM VMAXDeviceEntity n left JOIN FETCH n.snapVxCopyDevice WHERE n.storageArrayId = :storageArrayId AND n.inUseForReplication = 'false' AND n.enabledForReplication = 'true' AND n.capacity = :capacity
AppSync Event Log error shows:
<DATE>, Mount copy, VMAX_000002,<appsync-server>,VMAX operation failed. Error: Exception thrown while linking the copy org.hibernate.exception.GenericJDBCException: could not extract ResultSet.
AppSync Server Log error shows:
<DATE>INFO [EJB default - 10] [com.emc.archway.commands.storage.vmax.GetStorageForSnapVXCommandBean] [appsync-server] [] b8c58a7f-f1c4-493a-8ca2-e0354b9b44b3->>> Unlinking the previous relationship of device(if any) :000E5 <DATE> INFO [EJB default - 10] [com.emc.archway.service.vmax.VmaxServiceBean] [appsync-server] [] b8c58a7f-f1c4-493a-8ca2-e0354b9b44b3->>> The device 000E5 is not used as target in the array <Array S/N> <DATE> ERROR [EJB default - 10] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] [appsync-server] [] ERROR: out of memory Detail: Failed on request of size 17. <DATE> INFO [EJB default - 10] [org.hibernate.event.internal.DefaultLoadEventListener] [appsync-server] [] HHH000327: Error performing load command : org.hibernate.exception.GenericJDBCException: could not extract ResultSet
Cause
Error occurs when PostgreSQL is unable to allocate the memory required for a query to run or any other internal operation to complete.
Resolution
-
Create a backup of the AppSync postgres database by opening a DOS window as Administrator, cd to C:\EMC\AppSync\backup-assistant and run appsync-backup.bat
-
Ensure a copy of C:\EMC\AppSync\jboss\datastore\data\postgresql.conf is made and stored in a safe location.
-
Edit C:\EMC\AppSync\jboss\datastore\data\postgresql.conf and set work_mem to 128MB (uncomment if necessary).
-
Restart the three AppSync services. The best order to start them is:
EMC AppSync Datastore ServiceAppSync Security ServiceAppSync Server Service
Additional Information
Many other AppSync functions may incur the similar errors for which the same resolution can be used:
The similar errors being:
Event errors that contain:
could not extract ResultSet
Along with the following in the AppSync Server log: c:\EMC\AppSync\jboss\logs\server.log
ERROR: out of memory Detail: Failed on request of size