ECS: S3 Bucket Listing Using Pagination To Improve Performance

Dell Elastic Cloud Storage (ECS) does not impose a hard limit on the number of subfolders or objects within a bucket or prefix. However, performance degradation can occur when listing many objects (such as, thousands) under a single prefix or "folder"—especially when using S3-compatible clients.

The issue with the S3 Bucket with 570,909 objects listed can take several minutes. This makes it unmanageable comparing with a NAS with the same structure listing is instantaneous.  
This issue may be seen as performance impacting with listings appearing to timeout.

Difference in how NAS and S3 handle files

NAS has folders in the file system as a structure. Example structure layout:  

ROOT /
-> Subfolder l1
-> Subfolder l2  ... lX
-> Files. 

When opening a Subfolder containing for example 8000 subfolders inside, means to open the metadata for the 8000 subfolders, returning an instant list. 

S3 the concept of folders does not exist which means, instead of having a tree:

 ROOT /
-> Subfolder l1
-> Subfolder l2  ... lX
-> Files.

Only have a PATH:-

PATH:bucket/subfolder-l1/subfolder-l2/FILE

Now the performance is more affected by the way that the folders are listed. Example of 20 folders with 8000 subfolders with 10,000 objects each:
 

get "/bucket?prefix=/subfolder-l1/subfolder-2/&delimter=/"

Since limiting down the list all objects indicated with a common string will not have to do any grouping and the list of 10,000 items return faster.

GET "bucket?prefix=/subfolder-l1&delimiter=/"

In this case, the ECS must now run over all 8000*10000 objects to group the "common prefixes" in other words the "subfolders-l2"
So instead of checking 10,000 objects, now have to wait for the ECS to verify 80 Million objects.

Improve listing performance use pagination, example S3Broswer Enable Bucket Pagination (enable similar features in other client applications)

Enable the following option in S3Browser -> Tools -> Options -> Interface -> Enable bucket pagination with page size:

 The process of sending subsequent requests to continue where a previous request left off is called pagination. The pagination feature helps you manage and view large bucket listings by breaking them into more manageable pages. It allows the specific directory path to be listed rather than all objects in a bucket.