PowerScale: Listing S3 Bucket Objects Loops Infinitely when Using API ListObjects or ListObjectsV2

摘要: The S3 App or S3 command uses S3 API listobject or listobjectv2 to query objects from the S3 bucket. In some instances, PowerScale OneFS S3 returns the wrong result if dirCache is there but it does not fulfill the request. The S3 App stops responding due to list objects experiencing an infinite loop. ...

本文适用于 本文不适用于 本文并非针对某种特定的产品。 本文并非包含所有产品版本。
查看其他资源

症状

S3 App stops responding due to listing a bucket which does not complete.

S3 command ls bucket repeatedly gets the same objects returned in a paginated list.

原因

This may experience an S3 cache issue. 
 
If the client requests ListObjects with max entry = 1000, internally we may get 10,000 entries (if directory is wide) but only return 1000 of them to client, then reusing the remaining 9000 for a future request. On the next request, if there is a cache, and there are still files under the directory which are not yet returned to the client, and the current cache does not have enough entries to fulfill the request, we experience the issue and the wrong results are returned.

解决方案

Dell Engineering is working on a permanent resolution for this issue. A roll-up patch (RUP) patch is expected to be released.

The current workaround is to configure S3 ls DIR_CACHE_TTL to 0 temporarily: 
# isi_for_array -X  "/usr/likewise/bin/lwsm inject s3 s3_ls_set_dir_cache_ttl on 0"
If the S3 process restarts or a node reboots, apply the workaround again on the node since it has reverted back to the default 1 hour when S3 restarted.

To manual DIR_CACHE_TTL: 
#isi_for_array -X  "/usr/likewise/bin/lwsm inject s3 s3_ls_set_dir_cache_ttl off"

受影响的产品

Isilon, PowerScale
文章属性
文章编号: 000223936
文章类型: Solution
上次修改时间: 11 6月 2024
版本:  3
从其他戴尔用户那里查找问题的答案
支持服务
检查您的设备是否在支持服务涵盖的范围内。