PowerScale: Listing S3 Bucket Objects Loops Infinitely when Using API ListObjects or ListObjectsV2
Summary: The S3 App or S3 command uses S3 API listobject or listobjectv2 to query objects from the S3 bucket. In some instances, PowerScale OneFS S3 returns the wrong result if dirCache is there but it does not fulfill the request. The S3 App stops responding due to list objects experiencing an infinite loop. ...
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
S3 App stops responding due to listing a bucket which does not complete.
S3 command
S3 command
ls bucket repeatedly gets the same objects returned in a paginated list.
Cause
This may experience an S3 cache issue.
If the client requests
ListObjects with max entry = 1000, internally we may get 10,000 entries (if directory is wide) but only return 1000 of them to client, then reusing the remaining 9000 for a future request. On the next request, if there is a cache, and there are still files under the directory which are not yet returned to the client, and the current cache does not have enough entries to fulfill the request, we experience the issue and the wrong results are returned.
Resolution
Dell Engineering is working on a permanent resolution for this issue. A roll-up patch (RUP) patch is expected to be released.
The current workaround is to configure S3
To manual
The current workaround is to configure S3
ls DIR_CACHE_TTL to 0 temporarily:
# isi_for_array -X "/usr/likewise/bin/lwsm inject s3 s3_ls_set_dir_cache_ttl on 0"If the S3 process restarts or a node reboots, apply the workaround again on the node since it has reverted back to the default 1 hour when S3 restarted.
To manual
DIR_CACHE_TTL:
#isi_for_array -X "/usr/likewise/bin/lwsm inject s3 s3_ls_set_dir_cache_ttl off"
Affected Products
Isilon, PowerScaleArticle Properties
Article Number: 000223936
Article Type: Solution
Last Modified: 11 Jun 2024
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.