Don't know if this can be the reason, but are you sure you're not bumping into some kind of memory issue? Reading a 900Mo file in a byte array might put some strain on the java heap space.
It might be that 2Gb is enough to store the file once, but if you read into a ByteArrayOutputStream, you allocate truckload of byte[] because the buffer has to grow constantly (it will double in size every time it has to grow):
So it might be that there is too much garbage created, and that the SDK receives an OOM internally. The difference with a standalone program where you read and write locally is that the SDK generates some load and also uses internal buffers, which might push the JVM right over the edge.
Don't know if this is the problem, but it might not hurt either using an OutputStream that just throws away all data, just to check that this is indeed the problem, or use something like new BytearrayOutputStream(1024 * 1024 * 1024L) to avoid all the array copies and garbage (but it's a bit decadent :-)).
It is true that 900Mo file is big but I have specified a 2Go JVM for runtime, also I can read and write the file locally without problem.
In the scenario I have described it seems that the problem is really on the Centera side only for reading because it looks like writing works well for the same file (at least I do not have errors but I cannot check if the file has been stored without corruption because I cannot read it back...)
I finally managed to solve my initial problem by using the BlobReadPartial method along with multiple threading. I have read somewhere in the Centera documentation that the reading on a single stream cannot last more than 60s and this is maybe what was causing the error.
To solve my problem, I used the attached sample that was posted by Graham Stuart on another thread.
kim_marivoet
41 Posts
0
July 8th, 2011 11:00
Don't know if this can be the reason, but are you sure you're not bumping into some kind of memory issue? Reading a 900Mo file in a byte array might put some strain on the java heap space.
regards,
Kim
kim_marivoet
41 Posts
0
July 11th, 2011 08:00
It might be that 2Gb is enough to store the file once, but if you read into a ByteArrayOutputStream, you allocate truckload of byte[] because the buffer has to grow constantly (it will double in size every time it has to grow):
So it might be that there is too much garbage created, and that the SDK receives an OOM internally. The difference with a standalone program where you read and write locally is that the SDK generates some load and also uses internal buffers, which might push the JVM right over the edge.
Don't know if this is the problem, but it might not hurt either using an OutputStream that just throws away all data, just to check that this is indeed the problem, or use something like new BytearrayOutputStream(1024 * 1024 * 1024L) to avoid all the array copies and garbage (but it's a bit decadent :-)).
lmichel
1 Rookie
•
3 Posts
0
July 11th, 2011 08:00
Hello Kim,
It is true that 900Mo file is big but I have specified a 2Go JVM for runtime, also I can read and write the file locally without problem.
In the scenario I have described it seems that the problem is really on the Centera side only for reading because it looks like writing works well for the same file (at least I do not have errors but I cannot check if the file has been stored without corruption because I cannot read it back...)
Regards,
Laurent
lmichel
1 Rookie
•
3 Posts
0
July 25th, 2011 09:00
Hello,
I finally managed to solve my initial problem by using the BlobReadPartial method along with multiple threading. I have read somewhere in the Centera documentation that the reading on a single stream cannot last more than 60s and this is maybe what was causing the error.
To solve my problem, I used the attached sample that was posted by Graham Stuart on another thread.
Regards,
Laurent
1 Attachment
BlobReadPartialExample(2).java