Unsolved
This post is more than 5 years old
1 Message
0
580
Atmos with JetS3t question
Hi,
We are migration from S3 to Atmos and I was looking to leverage JetS3t again for this. The problem I am running into is on the listObjectsChunked method of JetS3t. When making this request the call hangs and I eventually get a timeout error.
I am able to get the bucket, put an object into the bucket and get an object out of the bucket, I am just unable to get the list of objects in a bucket.
What I am trying to figure out is if this is an issue with our Atmos configuration or if this method is not supported.
JetS3t version = 0.7.1
Thank you,
Chris
chris_arnett
110 Posts
0
December 3rd, 2013 09:00
Chris,
This is a known issue in the GET bucket call. A hot-fix is currently in the works and I can update this thread as more information is available.
Atmos S3 buckets can operate in two modes. In what we call “flat-mode”, a bucket can store billions of keys with no hierarchy, but listing the bucket will be extremely slow due to the sharding distribution scheme used under the covers. Flat-mode is currently the default mode for new buckets. In “one-to-one” mode, a bucket can only store about 100k keys per prefix (directory), but listing is faster.
If your Atmos environment is at version 2.1.4 or newer, you may be able to improve performance by creating a new bucket and changing the bucket mode to one-to-one. That procedure is documented in the latest Programmer's Guide on pg. 44.
In future versions (and after the hot-fix is applied), one-to-one will be the default mode for new buckets and all GET bucket calls will be much faster. However, any existing buckets will still be in flat-mode. If you use the GET bucket call, we highly recommend you move your keys into a new one-to-one mode bucket.
- Stu