Unsolved
This post is more than 5 years old
2 Posts
0
1894
FPClip_Open deadlock
Hi,
We are experiencing some issues using the SDK via the Java FPLibary. After around 100,000 documents read from the CAS, all threads become blocked waiting on FPClip_Open requiring us to restart the JVM.
public byte[] readFile(CasConfig.POOL pool, String casId) {
FPPool connection = null;
FPClip clip = null;
FPTag tag = null;
ByteArrayOutputStream outputStream = null;
try {
connection = getConnect(pool);
outputStream = new ByteArrayOutputStream();
clip = openFpClip(connection, casId);
tag = clip.getTopTag();
tag.BlobRead(outputStream);
return outputStream.toByteArray();
} catch (Exception e) {
log.error("Error from CAS. Reading CAS ID {}", casId, e);
try {
connection.Close();
} catch (FPLibraryException e1) {
}
connectionPool.remove(pool);
throw new RuntimeException("Could not read from CAS: " + e.getMessage(), e);
} finally {
try {
if (outputStream != null) {
outputStream.close();
}
if (tag != null) {
tag.Close();
}
if (clip != null) {
clip.Close();
}
} catch (FPLibraryException | IOException e) {
log.error("Failed to close CAS connection", e);
}
}
}
protected FPPool getConnect(CasConfig.POOL pool) throws FPLibraryException {
if (!connectionPool.containsKey(pool)) {
FPPool connection = new FPPool(casConfig.getPool().get(pool));
log.debug("Connected to CAS POOL \n {}", connection.getPoolInfo().toString());
connectionPool.put(pool, connection);
}
return connectionPool.get(pool);
}
protected FPClip openFpClip(FPPool pool, String casId) throws FPLibraryException {
return new FPClip(pool, casId, FPLibraryConstants.FP_OPEN_FLAT);
}
Options used:
nodeselection: 1
retrysleep: 10
retrycount: 3
maxconnections: 500
embedding_threshold: 30720
openstrategy: 1
Opening with FPLibraryConstants.FP_OPEN_FLAT.
Files vary in size but are generally less than 1MB.
Thanks.
mfh2
208 Posts
0
December 15th, 2017 06:00
Hi Dan -
Is there a reason you are invalidating the FPPool instance if you get an error? I'm pretty sure that connection.Close() call will fail if there are any open clips at the point of that call, which may be related to your issue if you are hitting errors with any frequency.
Best choice would be to simply open a single FPPool and leave it open for the complete lifetime of the system; our migration system often sees singleton FPPool lifetimes measured in weeks, with billions of successful SDK transactions across dozens of concurrent threads.
Keep us posted on your findings.
Best Regards,
Mike Horgan
dan_blackhurst
2 Posts
0
December 15th, 2017 06:00
Thanks for the reply. I'm not seeing that error in the logs. The exception handler is a precaution to reconnect after a temporary network drop.
Can I ask what settings you are using with the SDK?
mfh2
208 Posts
0
December 19th, 2017 10:00
Hi Dan -
Sorry for the delayed response. . Nothing particularly special in our configuration:
Is that what you were asking?
Regards,
Mike Horgan