JVM Crash Error while reading blobs (clips)

Firstly, welcome to the forums, and above all, thank you for being an EMC customer.

Please consider moving this question as-is (no need to recreate) to the proper forum for maximum visibility.  Questions written to the users' own "Discussions" space don't get the same amount of attention and can go unanswered for a long time.

You can do so by selecting "Move" under ACTIONS along the upper-right.  Then search for and select: "Centera Support Forum"

Centera Support Forum

Hello ChronHermann -

It appears to be crashing inside the native DLL. Which SDK version and which platform are you running on?

Regards,

Mike Horgan

Hello Mike,

my SDK Version:

libFPCore64.so.3.3.719

libFPLibrary64.so.3.3.719

libFPParser64.so.3.3.50

libFPStreams64.so.3.3.719

libFPUtils64.so.3.3.719

libFPXML64.so.3.3.719

libPAI_module64.so.3.3.100

Platform:

Red Hat Enterprise Linux Server release 6.4 (Santiago)

Linux localhost 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 12:08:01 PDT 2013 x86_64 x86_64 x86_64 GNU/Linux

Regards,

Korbinian Hermann

Hello Korbinian -

That all looks good to me. Your best bet for finding a cause would be to enable SDK logging and to wait for the crash to occur again. Since you say it takes hours for the crash to reoccur you may find the log gets very large (many GB) so make sure there is plenty of disk available.  You can enable SDK logging by adding this to your programs environment:

FP_LOGPATH=/path/to/where/he/has/enuf/space/filename.log

If you manage to catch the crash in the log, I'd say the first couple thousand lines and the last couple thousand would be the most important to the solution; you could post those here for input if you wish.

Good Luck,

Mike Horgan

Hello Mike,

thanks for your answer. I enabled the SDK logging and found some errors:

...

1385971200873   2013-12-02 08:00:00.873         [debug]         27481.277350144 [TRANSACTION]   Probe Reply localhost/6/PROBE

1385971201862   2013-12-02 08:00:01.862         [debug]         27481.276297472 [EXCEPTION]     In 'Connection.cpp' at line 994: Exception

error=-10101

syserror=0

message=receive: waitForReadingData(1000) returned zero

trace=No trace available

1385971201862   2013-12-02 08:00:01.862         [debug]         27481.276297472 [POOL]  Close FPSocket (mSocket=106) Connection open,unlocked marked(GOOD)

1385971201862   2013-12-02 08:00:01.862         [debug]         27481.276297472 [RETRY] AN(168.159.214.21) Unavailable (for 15 secs)

1385971201862   2013-12-02 08:00:01.862         [debug]         27481.276297472 [POOL]  Close connections for addr=168.159.214.21 (num=0,max=0)

1385971201862   2013-12-02 08:00:01.862         [debug]         27481.276297472 [POOL]  End close connections (num=0,max=0) ----

1385971201862   2013-12-02 08:00:01.862         [debug]         27481.276297472 [RETRY] retry (0) Probe because Exception

error=-10101

syserror=0

message=receive: waitForReadingData(1000) returned zero

trace=FPDatagramSocket.receive (timeout=1000)

transid=localhost/4/PROBE

1385971201862   2013-12-02 08:00:01.862         [debug]         27481.276297472 [RETRY] Cluster::selectAccessNodeWithoutProbe(true) -> node #0=168.159.214.21, load=99-99

1385971201862   2013-12-02 08:00:01.862         [debug]         27481.276297472 [POOL]  Open FPSocket (mSocket=106) Connection open,unlocked marked(GOOD) Type(2) for 168.159.214.21

1385971201862   2013-12-02 08:00:01.862         [debug]         27481.276297472 [TRANSACTION]   Probe Request localhost/7/PROBE

1385971201863   2013-12-02 08:00:01.863         [debug]         27481.276297472 [PACKET]        send SmartPacket

  NET_SYSTEMID type=string value=localhost

  NET_TRANSACTIONID type=string value=localhost/7/PROBE

  NET_VERSION type=integer value=3  HPP_CLIENT_VERSION type=integer value=197376  POOLSERVER_VERSION type=integer value=1  NET_MESSAGEID type=integer value=36  POOLSERVER_OPCODE type=integer value=1

1385971201863   2013-12-02 08:00:01.863         [debug]         27481.276297472 [TRANSACTION]   Probe Reply localhost/7/PROBE

1385971201874   2013-12-02 08:00:01.874         [debug]         27481.283666176 [EXCEPTION]     In 'Connection.cpp' at line 994: Exception

error=-10101

syserror=0

message=receive: waitForReadingData(1000) returned zero

trace=No trace available

1385971201874   2013-12-02 08:00:01.874         [debug]         27481.283666176 [POOL]  Close FPSocket (mSocket=107) Connection open,unlocked marked(GOOD)

1385971201874   2013-12-02 08:00:01.874         [debug]         27481.283666176 [RETRY] AN(168.159.214.21) Unavailable (for 15 secs)

1385971201874   2013-12-02 08:00:01.874         [debug]         27481.283666176 [POOL]  Close connections for addr=168.159.214.21 (num=0,max=0)

1385971201874   2013-12-02 08:00:01.874         [debug]         27481.283666176 [POOL]  End close connections (num=0,max=0) ----

1385971201874   2013-12-02 08:00:01.874         [debug]         27481.283666176 [RETRY] retry (0) Probe because Exception

error=-10101

syserror=0

message=receive: waitForReadingData(1000) returned zero

trace=FPDatagramSocket.receive (timeout=1000)

transid=localhost/5/PROBE

...

1385971208857   2013-12-02 08:00:08.857         [debug]         27481.276297472 [API]   PoolOption 'prefetchsize' = '32768'

1385971208857   2013-12-02 08:00:08.857         [log]           27481.276297472 [API]   End FPPool_Open8(168.159.214.20,168.159.214.21?/home/transfer/c1armtesting.pea)

1385971208857   2013-12-02 08:00:08.857         [debug]         27481.283666176 [EXCEPTION]     In 'FPClip.cpp' at line 3793: Exception

error=-10018

syserror=0

message=

trace=FPClip.GetMetaAttribute(retention.class)

1385971208858   2013-12-02 08:00:08.858         [debug]         27481.283666176 [API]   GlobalOption 'retrycount' = '6'

1385971208858   2013-12-02 08:00:08.858         [debug]         27481.283666176 [API]   GlobalOption 'retrysleep' = '-1'

...

1385973098643   2013-12-02 08:31:38.643         [log]           27481.274192128 [API]   Start FPPool_GetLastError()

1385973098643   2013-12-02 08:31:38.643         [log]           27481.274192128 [API]   End FPPool_GetLastError() --> [0]

1385973098643   2013-12-02 08:31:38.643         [log]           27481.274192128 [API]   Start FPPool_GetLastError()

1385973098643   2013-12-02 08:31:38.643         [log]           27481.274192128 [API]   End FPPool_GetLastError() --> [0]

1385973098643   2013-12-02 08:31:38.643         [log]           27481.274192128 [API]   Start FPTag_GetTagName8(-,-,256)

1385973098643   2013-12-02 08:31:38.643         [debug]         27481.274192128 [EXCEPTION]     In 'FPTag.cpp' at line 1530: Exception

error=-10012

syserror=0

message=

trace=No trace available

1385973098643   2013-12-02 08:31:38.643         [error]         27481.274192128 [API]   End FPTag_GetTagName8(-,-,0): Error -10012

1385973098643   2013-12-02 08:31:38.643         [log]           27481.274192128 [API]   Start FPPool_GetLastError()

1385973098643   2013-12-02 08:31:38.643         [log]           27481.274192128 [API]   End FPPool_GetLastError() --> [-10012]

1385973098643   2013-12-02 08:31:38.643         [log]           27481.274192128 [API]   Start FPPool_GetLastError()

1385973098643   2013-12-02 08:31:38.643         [log]           27481.274192128 [API]   End FPPool_GetLastError() --> [-10012]

...

Hi Korbinian -

There seem to be a number of things going on here. The -10101 errors indicate a networking problem, possibly with that particular access node (is it down?). This isn't of great concern, although it may slow you down a bit.

The -10018 is an FP_ATTR_NOT_FOUND error that occurs when your code is checking to see if a clip has a retention class defined, not a problem as long as you handle this return code appropriately.

The -10012 looks like an actual problem. The SDK is having some sort of issue related to how you are trying to traverse a clip that was opened in FP_FLAT mode (which does not support parent/child/sibling type traversal).  You should probably look into this and see if it is preventing you from releasing all of your SDK resources correctly. Leaked SDK handles will eventually case problems, possibly along the lines of what you are experiencing after a few hours of runtime.

Good Luck,

Mike Horgan

Hi Mike,

thank you for your answer! I tried to find clips that were opened in FP_FLAT mode but all the clips are opened in FP_OPEN_ASTREE mode. I checked all my SDK handles too and the handles and clips are closed correctly. I tried to "null" the handles and clips after closing - The same problem.

Can i discover which clips are openend in FP_FLAT mode?

Regards,

Korbinian Hermann

Hi Korbinian -

The SDK log contains every FPClip_Open call with clipid and open_mode in the second and third parameters. From the docs:

That seems to be your best bet. Happy Spelunking!

Mike Horgan

Hello Mike,

i searched in my sdk-logfile but i found only FPClip_Open calls with FP_OPEN_ASTREE mode..

It seems that my problem is on another cause.

Regards,

Korbinian Hermann

I agree, this path of investigation has reached a dead end. But hopefully you have learned quite a bit about Centera  SDK logs and they may help you find the root cause.

Best Regards,

Mike Horgan

Hello togehter,

i'm looking still after a root cause and so i added to my java-execution call the parameter "-Xcheck:jni". Now i receive after every BlobRead-Call the following exception in my console:

WARNING in native method: JNI call made with exception pending

        at com.filepool.natives.FPLibraryNative.FPTag_BlobReadPartial(Native Method)

        at com.filepool.fplibrary.FPTag.BlobReadPartial(Unknown Source)

        at com.filepool.fplibrary.FPTag.BlobRead(Unknown Source)

Has anyone an idea why this error is shown?

Regards,

Korbinian Hermann

Sorry Korbinian, I have no clue on this one.

Perhaps someone from EMC support will chime in.

Mike Horgan

ChronHermann wrote: Hello togehter, i'm looking still after a root cause and so i added to my java-execution call the parameter "-Xcheck:jni". Now i receive after every BlobRead-Call the following exception in my console: WARNING in native method: JNI call made with exception pending         at com.filepool.natives.FPLibraryNative.FPTag_BlobReadPartial(Native Method)         at com.filepool.fplibrary.FPTag.BlobReadPartial(Unknown Source)         at com.filepool.fplibrary.FPTag.BlobRead(Unknown Source) Has anyone an idea why this error is shown? Regards, Korbinian Hermann

Yesterday, i added the parameter to the "EMC Console". The exception came also there, as i downloaded a clip. So i think this is a main-problem from the centera sdk, but not the cause of my problem.

Regards,

Korbinian Hermann