naveen_bijalwan

updated

8 years ago

N

naveen_bijalwan

1 Rookie

•

11 Posts

0

13928

May 18th, 2018 04:00

getting Caused by: java.net.SocketException: Connection reset by peer: socket write error

I am getting Caused by: java.net.SocketException: Connection reset by peer: socket write error

when using multi-part upload with SSECustomerKey using Amazon s3 client sdk

                // Create the request to upload a part.

                UploadPartRequest uploadRequest = new UploadPartRequest()//.withSSECustomerKey(SSE_KEY)

                        .withBucketName(bucketName)

                        .withKey(keyName)

                        .withUploadId(initResponse.getUploadId())

                        .withPartNumber(i)

                        .withInputStream(is)

                        .withPartSize(partSize)

                        .withSSECustomerKey(SSE_KEY);

                UploadPartResult uploadResult = S3_CLIENT.uploadPart(uploadRequest);

while upload works fine with single part upload

PutObjectRequest putRequest = new PutObjectRequest(bucketName, keyName, is, metadata).withSSECustomerKey(SSE_KEY);

S3_CLIENT.putObject(putRequest);

why is dell emc ecs throwing java.net.SocketException: Connection reset by peer: socket write error

  • chris_arnett

    1 Rookie

    •

    110 Posts

    11264

    1

    Posted May 23rd, 2018 06:00

    Thanks for posting that code.  This is exactly what we needed to see.  The problem is that you are specifying SSE (with server-managed key) in the init request, but specifying SSE-C (customer-managed key) in the part uploads.  If you want to use SSE-C (where you manage your own keys), you need to change your init call to the following:

    // DELETE THIS LINE

    //objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);

    // Initiate the multipart upload.

    InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, keyName,

            objectMetadata);

    initRequest.setSSECustomerKey(SSE_KEY); // ADD THIS LINE

    InitiateMultipartUploadResult initResponse = S3_CLIENT.initiateMultipartUpload(initRequest);

    If you want to use SSE (with server-managed keys), then remove the customer managed key assignment in the part uploads (it will inherit the setting from the init request).

  • kastr1

    16 Posts

    11264

    1

    Posted May 18th, 2018 09:00

    If you are using ECS 3.2.0.0, this is a known bug with Multi-part uploads.

    The fix will be in v3.2.0.1, which should be released 22-May.

    Normal uploads will work as will multi-part uploads if the URL is not percent-encoded. For 3.2.0.0, the work-around is to avoid using percent-encoding in multi-part uploads.

  • naveen_bijalwan

    1 Rookie

    •

    11 Posts

    11267

    0

    Posted May 18th, 2018 10:00

    ok thanks. I'll wait for v3.2.0.1.


    "For 3.2.0.0, the work-around is to avoid using percent-encoding in multi-part uploads."

    -- But I am using ip and port directly like to create AmazonS3 client.

    EndpointConfiguration endpointConfiguration = new EndpointConfiguration("*.*.*.*:9021", "US_WEST_2");

    Which URL should be percent-encoded?

  • kastr1

    16 Posts

    11267

    0

    Posted May 21st, 2018 12:00

    I'm talking about the URL that is the MPU request. For instance:

    POST /Bucket1/object.name?uploads

    It is common to percent-encode the 'dot' in the URL above:

    POST /Bucket1/object%2Ename?uploads

    If the URL does contain a percent-encoded character (such as the %2E above), the MPU will fail until you upgrade to ECS 3.2.0.1.

  • naveen_bijalwan

    1 Rookie

    •

    11 Posts

    11267

    0

    Posted May 22nd, 2018 05:00

    Yes but I am doing using aws-java-sdk-s3  and not through rest url and my keyname and bucketName doesn't contain any special characters. As said before that It works for me without .withSSECustomerKey(SSE_KEY) but with server side encryption S3_CLIENT.uploadPart fails 

  • kastr1

    16 Posts

    11267

    0

    Posted May 22nd, 2018 06:00

    Your original note stated that the upload failed as a multi-part, but worked as a single upload. In BOTH cases you were using withSSECustomerKey(SSE_KEY). It never states that you were successful without "withSSECustomerKey(SSE_KEY)"


    From reading the bug report, there should be no association of this error with SSE. Please verify this behavior, and be careful that any request that has no percent-encoded will cause the multi-part upload to succeed.

  • naveen_bijalwan

    1 Rookie

    •

    11 Posts

    11267

    0

    Posted May 22nd, 2018 09:00

    my bad if post is misinterpreted . I meant uploadpart is successful without "withSSECustomerKey(SSE_KEY)". I have tried it number of times but upload part is continuously failing with withSSECustomerKey(SSE_KEY). I am not sure whether withSSECustomerKey(SSE_KEY) introduces any percent-encoded due to encryption but same request is passing without withSSECustomerKey(SSE_KEY).

  • chris_arnett

    1 Rookie

    •

    110 Posts

    11267

    0

    Posted May 22nd, 2018 15:00

    Since you are using the AWS SDK, you have access to the TransferManager class.  Using this to upload MPU with SSE-C against ECS Test Drive (ECS v3.2.0) works for me.  Here is my code excerpt:

            // Create an encryption key.

            KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");

            keyGenerator.init(256, new SecureRandom());

            SSECustomerKey encKey = new SSECustomerKey(keyGenerator.generateKey());

            ObjectMetadata metadata = new ObjectMetadata();

            metadata.setContentLength(data.length);

            // only way to use SSE-C is to use a POR:

            PutObjectRequest putRequest = new PutObjectRequest(testBucket, key, dataStream, metadata);

            putRequest.withSSECustomerKey(encKey);

            // set MPU threshold to 1 to make sure we use MPU

            TransferManager tm = TransferManagerBuilder.standard().withS3Client(client).withMultipartUploadThreshold(1L).build();

            tm.upload(putRequest).waitForUploadResult();

  • naveen_bijalwan

    1 Rookie

    •

    11 Posts

    11267

    0

    Posted May 23rd, 2018 02:00

    But how will TransferManager work with UploadPartRequest object for I don't have single stream but each part is coming as seperate stream of length say 5MB each. Is there something we can do similar to com.emc.vipr.services.s3.model.AppendObjectRequest or com.emc.vipr.services.s3.model.UpdateObjectRequest in com.emc.vipr s3-client package for aws-s3-sdk as well for ViPRS3Client doesn't work with  withSSECustomerKey


    The current code is

    long partSize = 5 * 1024 * 1024; // Set part size to 5 MB.

    // Create a list of ETag objects. You retrieve ETags for each object part

    // uploaded,

    // then, after each individual part has been uploaded, pass the list of ETags to

    // the request to complete the upload.

    List partETags = new ArrayList ();


    ObjectMetadata objectMetadata = new ObjectMetadata();

    // objectMetadata.setContentType("application/x-www-form-urlencoded;

    // charset=utf-8");

    objectMetadata.setContentLength(contentLength);


    objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);


    // Initiate the multipart upload.

    InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, keyName,

    objectMetadata);

    InitiateMultipartUploadResult initResponse = S3_CLIENT.initiateMultipartUpload(initRequest);


    // Upload the file parts.

    long filePosition = 0;

    for (int i = 1; filePosition < contentLength; i++) {

    // Because the last part could be less than 5 MB, adjust the part size as

    // needed.

    partSize = Math.min(partSize, (contentLength - filePosition));


    byte[] subArray = Arrays.copyOfRange(contentBytes, (int) filePosition, (int) (filePosition + partSize));


    // Create the request to upload a part.

    UploadPartRequest uploadRequest = new UploadPartRequest()// .withSSECustomerKey(SSE_KEY)

    .withBucketName(bucketName).withKey(keyName).withUploadId(initResponse.getUploadId())

    .withPartNumber(i).withInputStream(new ByteArrayInputStream(subArray)).withPartSize(partSize)

    .withSSECustomerKey(SSE_KEY);//works without withSSECustomerKey

    //but fails with Unable to execute HTTP request: Connection reset by peer: socket write error for withSSECustomerKey

    boolean isError = false;

    while (!isError) {

    try {

    // Upload the part and add the response's ETag to our list.

    UploadPartResult uploadResult = S3_CLIENT.uploadPart(uploadRequest);

    partETags.add(uploadResult.getPartETag());

    } catch (Exception e) {

    System.out.println(e.getMessage());

    continue;

    }

    System.out.println("done upload part");

    isError = true;

    }

    filePosition += partSize;

    }


    // Complete the multipart upload.

    CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName, keyName,

    initResponse.getUploadId(), partETags);

    S3_CLIENT.completeMultipartUpload(compRequest);

  • naveen_bijalwan

    1 Rookie

    •

    11 Posts

    2363

    0

    Posted May 23rd, 2018 07:00

    Thanks for pointing out the mistake that I was doing. I was doing objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION); and then passing it to initRequest but going by suggestion to use

    1. initRequest.setSSECustomerKey(SSE_KEY); // ADD THIS LINE and removing objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);

    worked for me and I am able to upload file successfully using UploadPartRequest with .withSSECustomerKey(SSE_KEY) attached screenshot. thanks for quick suggestions.dellemc.png