Highlighted
2 Bronze

AtmosApi.readObjectStream returns empty stream.

Jump to solution

Hello,

Please do let me know if this is not the right forum to ask this question.

That being said, I'm having an issue with using the Java Atmos Client API (ver 2.1.5). I want to to stream the content to the Servlet response instead of first reading the byte array because that loads all the content of the file to memory.

Here is the code so far:

ReadObjectResponse<java.io.InputStream> response = this.esuAPI.readObjectStream(new ObjectId(atmosObjectId), null);

after this executes the response.getContentLength() method prints 0.

I have verified following details before posting:

1. Connection credentials are correct (host, port, uid and secret key)

2. File does have proper data on the server. I'm able to download the file via AtmosChrome extension.

I'm not sure why the response would be empty?

Labels (1)
Reply
1 Solution

Accepted Solutions
Highlighted
3 Argentum

You cannot pass null to the second argument of readObject.  This is the class type you want back (I'm not sure how it even compiled).  You should pass InputStream.class if you want an InputStream, e.g.

InputStream stream = this.atmosAPI.readObject(new ObjectId(atmosObjectId), InputStream.class);

What is the size of the object if you getSystemMetadata() ?

View solution in original post

0 Kudos
Reply
16 Replies
Highlighted
3 Argentum

getContentLength() will return zero for objects over 1MB in size.  This is because for objects greater than 1MB Atmos uses Transfer-Encoding: chunked to "stream" the response and does not send a Content-Length header in the response.  You should simply read the stream until you reach EOF.  If you need to know the size of the object before reading it, we suggest you call getSystemMetadata() first to read the value of the "size" field.

Reply
Highlighted
2 Bronze

Thank you for the response Jason!

Initially I had used the InputStream returned to create a file and the file was of 0kb, and for that reason I wanted to print the content length.

Here is what my code looks like for creating a file (this code writes out 0kb file as the control would not execute the while loop even once):

FileOutputStream output = new FileOutputStream(new File("C:/Sample.pdf"));

  byte[] buff = new byte[1024];

  int bytesRead;

  boolean contentRead = false;

  while ((bytesRead = inputStream.read(buff)) > 0) {

  output.write(buff, 0, bytesRead);

  contentRead = true;

  }

  output.flush();

If I use readObject method instead of readObjectStream:

InputStream stream = this.atmosAPI.readObject(new ObjectId(atmosObjectId), null);

I'm getting null pointer exception shown below. I cant seem to get any download working with the Atmos Client 2.1.5 jar.

java.lang.NullPointerException

  at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.isReadable(AbstractRootElementProvider.java:92)

  at com.sun.jersey.core.spi.factory.MessageBodyFactory._getMessageBodyReader(MessageBodyFactory.java:345)

  at com.sun.jersey.core.spi.factory.MessageBodyFactory._getMessageBodyReader(MessageBodyFactory.java:315)

  at com.sun.jersey.core.spi.factory.MessageBodyFactory.getMessageBodyReader(MessageBodyFactory.java:294)

  at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:617)

  at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:586)

  at com.emc.atmos.api.jersey.AtmosApiClient.readObject(AtmosApiClient.java:196)

  at com.emc.atmos.api.AbstractAtmosApi.readObject(AbstractAtmosApi.java:53)

...

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

  at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

  at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)

  at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:82)

  at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)

  at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:240)

  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)

  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)

  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)

  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)

  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)

  at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)

  at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)

  at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

  at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:180)

  at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)

  at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)

  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)

  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)

  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

0 Kudos
Reply
Highlighted
3 Argentum

You cannot pass null to the second argument of readObject.  This is the class type you want back (I'm not sure how it even compiled).  You should pass InputStream.class if you want an InputStream, e.g.

InputStream stream = this.atmosAPI.readObject(new ObjectId(atmosObjectId), InputStream.class);

What is the size of the object if you getSystemMetadata() ?

View solution in original post

0 Kudos
Reply
Highlighted
2 Bronze

The getSystemMetadata() returns the size 21122.

If I use the InputStream.class parameter, I get the stream object back. But when I try to read content from Stream - I get Stream Closed error:

java.io.IOException: Stream closed

  at java.io.BufferedInputStream.getBufIfOpen(Unknown Source)

  at java.io.BufferedInputStream.read(Unknown Source)

  at java.io.FilterInputStream.read(Unknown Source)

  at java.io.FilterInputStream.read(Unknown Source)

Another stupid mistake I had done is to use different credentials, basically the object Id I was looking to download existed in a different subtenant. I am surprised why I didn't receive errorCode=1003 and HTTP Status Code 404. Anyways once the issue with credentials is corrected, I used readObjectStream() again and this time it worked like a charm! . I can see this method returns ByteArrayInputStream by default.

Surprising thing is the readObject() method still has issues. For now since I got readObjectStream() working, I'm going to continue using that. However I would need to circle back to this if I run into trouble handling large files (about 200 MB size) with out-of-memory issues.

Thank you once again for your time!

0 Kudos
Reply
Highlighted
3 Argentum

Glad you're up and running.  I'm not sure what the issue is that you're seeing with readObject, but I would recommend upgrading to the latest atmos-client (2.2.2).  If you're using a Maven-compatible system, the artifact is:

  <dependency> 

    <groupId>com.emc.vipr</groupId> 

    <artifactId>atmos-client</artifactId> 

    <version>2.2.2</version> 

  </dependency> 

Otherwise, you can download the latest from this page.

0 Kudos
Reply
Highlighted
2 Bronze

We did try to use the atmos-client jar 2.2.2 version, but the distributed jar requires Java 7+ but our systems are still using Java 6 and it involves spending additional effort (and money ) trying to upgrade our systems at this point, for that reason we resorted to version 2.1.5 which requires Java 6 as minimum requirement. I believe from version 2.1.6 the minimum requirement is Java 7+.

I had run into same issue of empty stream again yesterday with both readObject and readObjectStream methods. This time when I verified the file via Chrome extension I could see that the Object was created but there was no content in it. Content-length was 0. I also confirmed this by calling getSystemMetadata() and that prints size 0 too.

When I went through the log entries when upload is done I can see following message in the logs:

Content request with input stream and zero-length will not send any data

So I believe the content was never transferred to Cloud. I'm using below Code to upload the file:

FileInputStream inputStream = new FileInputStream(new File("C:/SampleFile.docx"));

CreateObjectRequest req = new CreateObjectRequest();

req.content(inputStream);

req.setAcl(new Acl()); //This may not be required

CreateObjectResponse resp = this.atmosAPI.createObject(req);

ObjectId objectId = resp.getObjectId();

Do I have to use code similar to what we have in com.emc.esu.api.rest.UploadHelper.createObject(InputStream, Acl, MetadataList, boolean) as this uploads the file content as small chunks?

0 Kudos
Reply
Highlighted
3 Argentum

If you're uploading a file, you can use the File object as the content.  That should solve you're problem.

Whenever you upload a stream, you must specify a content-length as well.  The Atmos API requires a content-length header and, aside from buffering the entire stream in memory, there is no other way for the client to know what that is.

Reply
Highlighted
2 Bronze

Thank you for the response! you were spot on the issue.

Jason's response to this post was the hint to get me to the solution.

It was easy for us to find out the content-length because our source of data was SQL Server (files are stored in a column with image data type). I could run a simple SQL query to get the length of the image column in bytes and set that to the CreateObjectRequest-object.

We did not use any temporary storage and were able to implement a 'pure streaming' from SQL Server to Atmos and successfully transferred few thousand files in our testing. Looking forward to migrate all the data on production (it has about 1.5 million files of various sizes)

It would not have been possible without all of your help! thank you!

0 Kudos
Reply
Highlighted
2 Bronze

SNShiva and Stu Arnett,

I am in the same situation where I have to write a large file onto Atmos.

The solution explained here is what I am looking for. I just need one more bit of detail.

How did you implement the "pure streaming" from your source location to Atmos? I have questions about how to write the content in a loop where we will stream only certain number of bytes. And make the receiving side receive it the same way and return only after the whole content is read.

Could you please explain that part?

Thanks,
Vandan

0 Kudos
Reply