FPClip_Open deadlock

Question

Hi, We are experiencing some issues using the SDK via the Java FPLibary. After around 100,000 documents read from the CAS, all threads become blocked waiting on FPClip_Open requiring us to restart the JVM.     public byte[] readFile(CasConfig.POOL pool, String casId) {         FPPool connection = null;         FPClip clip = null;         FPTag tag = null;         ByteArrayOutputStream outputStream = null;         try {             connection = getConnect(pool);             outputStream = new ByteArrayOutputStream();             clip = openFpClip(connection, casId);             tag = clip.getTopTag();             tag.BlobRead(outputStream);             return outputStream.toByteArray();         } catch (Exception e) {             log.error('Error from CAS. Reading CAS ID {}', casId, e);             try {                 connection.Close();             } catch (FPLibraryException e1) {             }             connectionPool.remove(pool);             throw new RuntimeException('Could not read from CAS: ' + e.getMessage(), e);         } finally {             try {                 if (outputStream != null) {                     outputStream.close();                 }                 if (tag != null) {                     tag.Close();                 }                 if (clip != null) {                     clip.Close();                 }             } catch (FPLibraryException | IOException e) {                 log.error('Failed to close CAS connection', e);             }         }     }       protected FPPool getConnect(CasConfig.POOL pool) throws FPLibraryException {         if (!connectionPool.containsKey(pool)) {             FPPool connection = new FPPool(casConfig.getPool().get(pool));             log.debug('Connected to CAS POOL 
 {}', connection.getPoolInfo().toString());             connectionPool.put(pool, connection);         }         return connectionPool.get(pool);     }     protected FPClip openFpClip(FPPool pool, String casId) throws FPLibraryException {         return new FPClip(pool, casId, FPLibraryConstants.FP_OPEN_FLAT);     } Options used:     nodeselection: 1     retrysleep: 10     retrycount: 3     maxconnections: 500     embedding_threshold: 30720     openstrategy: 1 Opening with FPLibraryConstants.FP_OPEN_FLAT. Files vary in size but are generally less than 1MB. Thanks.

mfh2 · Answer

Hi Dan -

Is there a reason you are invalidating the FPPool instance if you get an error? I'm pretty sure that connection.Close() call will fail if there are any open clips at the point of that call, which may be related to your issue if you are hitting errors with any frequency.

Best choice would be to simply open a single FPPool and leave it open for the complete lifetime of the system; our migration system often sees singleton FPPool lifetimes measured in weeks, with billions of successful SDK transactions across dozens of concurrent threads.

Keep us posted on your findings.

Best Regards,

Mike Horgan

dan_blackhurst · Answer

Thanks for the reply. I'm not seeing that error in the logs. The exception handler is a precaution to reconnect after a temporary network drop. Can I ask what settings you are using with the SDK?

mfh2 · Answer

Hi Dan -

Sorry for the delayed response. . Nothing particularly special in our configuration:

set FP_OPTION_OPENSTRATEGY_OVERRIDE=0
set FP_OPTION_ENABLE_MULTICLUSTER_FAILOVER_OVERRIDE=1
set FP_OPTION_MAXCONNECTIONS=300

Is that what you were asking?

Regards,

Mike Horgan

Centera

Was this post helpful?