Unable to read the contents of the HTTP body

Just to clarify, are you uploading to Atmos hardware or are you using the Atmos API on ECS?

It would also help to see the code that is doing the upload.  I'm specifically interested in the content-type being set and the data object (is it a stream, byte array, or.. ?)

Have you tried upgrading to the most recent version of the atmos-client (3.0.1)?  It looks like you're using 2.1.4 (from that log excerpt).

1. We are uploading to an ATMOS appliance (not a VM and not ECS).

2. The content-type is set to: application/pdf

3. We are using the 2.1.4 client API. We have not upgraded to 3.0.1 yet.

Code Fragment:

AtmosApi atmos = getAtmosApi(request.getConnectionFrom());

String md5ChecksumBefore = PersisterUtils.getFileMD5Checksum(request.getContent());

String uid = PersisterUtils.getUID();

Metadata md1 = new Metadata("FileName", request.getFileName(), true);

Metadata md2 = new Metadata("MD5CheckSum", md5ChecksumBefore, true);

CreateObjectRequest coRequest = new CreateObjectRequest();

coRequest.content(request.getContent());

coRequest.contentType(request.getContentType());

coRequest.identifier(new ObjectPath(uid));

coRequest.userMetadata(md1, md2);

CreateObjectResponse coResponse = atmos.createObject(coRequest);

What type of object is "request" and what type of object does getContent() return?  If it's an InputStream, you likely exhausted the stream when computing the checksum.

Note that the Atmos API has its own streaming checksumming algorithm you can use (aka "wschecksum") that will also allow Atmos to verify the upload body.

getContent returns a byte[]

The file sizes we have attempted to upload are 85kb, 500kb, 60mb, etc. all fail.

We have other applications that use this same shared library and are able to load files successfully without fail, albeit the files are generally less than 20kb. We are pointed to the same backend Atmos appliance that makes this even more strange.

As a test, we even loaded the failing WAR onto the same VM as the applications that are working, but this particular application still failed.

The failing application is working in a multitude of other environments just fine. It is only failing in a single environment, and fails on every attempt.

"Unable to read body" vs  "Broken pipe"

The error message randomly changes to the following in some executions:

com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Broken pipe (Write failed)

        at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:192) ~[jersey-apache-httpclient4-1.16-CUSTOM.jar:?]

        at com.emc.atmos.api.jersey.ErrorFilter.handle(ErrorFilter.java:32) ~[atmos-client-2.1.4.jar:?]

        at com.emc.atmos.api.jersey.RetryFilter.handle(RetryFilter.java:51) ~[atmos-client-2.1.4.jar:?]

        at com.emc.atmos.api.jersey.AuthFilter.handle(AuthFilter.java:41) ~[atmos-client-2.1.4.jar:?]

        at com.sun.jersey.api.client.Client.handle(Client.java:648) ~[jersey-client-1.16.jar:1.16]

        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) ~[jersey-client-1.16.jar:1.16]

        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.16.jar:1.16]

        at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568) ~[jersey-client-1.16.jar:1.16]

        at com.emc.atmos.api.jersey.AtmosApiClient.createObject(AtmosApiClient.java:181) ~[atmos-client-2.1.4.jar:?]

        at com.bkfs.dp.intf.impl.AtmosPersister.upload(AtmosPersister.java:131) [document-persister-1.0.2.1.jar:?]

        at com.bkfs.dp.FilePersisterWrapper.atmosContentUpload(FilePersisterWrapper.java:300) [classes/:?]

        at com.bkfs.dp.FilePersisterWrapper.uploadContent(FilePersisterWrapper.java:376) [classes/:?]

        at org.apache.jsp.LOS.Application.UploadFiles.UploadFile_jsp._jspService(UploadFile_jsp.java:433) [SRVLPSUPLOAD/:?]

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) [jasper.jar:8.5.4]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet-api.jar:?]

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:443) [jasper.jar:8.5.4]

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385) [jasper.jar:8.5.4]

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329) [jasper.jar:8.5.4]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet-api.jar:?]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [catalina.jar:8.5.4]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [catalina.jar:8.5.4]

        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.4]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [catalina.jar:8.5.4]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [catalina.jar:8.5.4]

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [catalina.jar:8.5.4]

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) [catalina.jar:8.5.4]

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:522) [catalina.jar:8.5.4]

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.4]

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.5.4]

        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620) [catalina.jar:8.5.4]

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.4]

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) [catalina.jar:8.5.4]

        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:780) [tomcat-coyote.jar:8.5.4]

        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.4]

        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785) [tomcat-coyote.jar:8.5.4]

        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425) [tomcat-coyote.jar:8.5.4]

        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.4]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.4]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

A broken pipe usually indicates some kind of network problem.  Are you going through a load balancer or proxy?  Are the NICs on the machine using jumbo packets by chance?  Does an upload from the same machine using a different application work?

I noticed that you have an SR opened for this problem.  Can you note a specific namespace path and the UTC time the errors happened or reproduce them, providing that information to support?  That will allow them to examine the errors in the logs on the Atmos nodes.

We believe it's a VIP issue. We are able to complete uploads when directly routing to the servers. We will continue to work with NOC resources to resolve.