ECS: CopyObject timeout 500 during high copy concurrency of large remote objects

Summary: A user may receive 500 timeout errors in response to one or more of the CopyObject requests if 1. If there are many concurrent CopyObject requests received with the same x-amz-copy-source object 2. The source object is larger than 1GB and the chunks holding the source object data are not replicated to the requested site. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

When the issue occurs, the user sees 500 error responses to CopyObject operation with the cause ServerTimeout. The response body contains additional information as follows. 
<Error>
    <Code>ServerTimeout</Code>
    <Message>We encountered an internal timeout error. Please try again.</Message>
    <Resource>bucket/object</Resource>
    <RequestId>0aece26b:18c57b06ec7:e3f:5a41d</RequestId>
</Error>

Cause

A portion of the CopyObject request is internally synchronized on the source object using a locking mechanism. The time required for the synchronized portion is proportional to the size of the object. When there are many concurrent CopyObject requests with the same large source object, the queue of requests waiting for the lock grows. The problem is exacerbated when the chunks holding the source object data are not replicated to the site in which the CopyObject request is being served, leading to a timeout when the queue grows too large.

Resolution

  • Retry the request that failed.
  • Slow the rate of CopyObject requests with the same x-amz-source-object.
  • If the above steps do not solve the issue, contact Dell EMC technical support for assistance.
  • Validate that the CopyObject requests that previously failed due to 500 ServerTimeout are now successful.

Affected Products

ECS, ECS Appliance, ECS Appliance Software with Encryption, ECS Appliance Software without Encryption
Article Properties
Article Number: 000223406
Article Type: Solution
Last Modified: 22 Mar 2024
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.