aronb1
1 Copper

Scalability - Is Atmos capable of storing more than 5 billion objects ?

According to the Atmos user guide, the number of unique listable tags should be limited to 50,000, and the number of objects associated with a single tag should not exceed 100,000.

According to my understanding this means that in the best case (fully balanced tree) no more than 5 billion objects can be stored in a single Atmos and accessible using listable tags. Hardly an astronomical figure these days.

Furthermore, the same limit seems to apply to the namespace API as well. Meaning that when using the namespace API I cannot store more than 5 billion objects.

The matters get much worse when the logical division of objects is not perfectly balanced.

Is there any way to overcome this limitation?

Labels (1)
0 Kudos
2 Replies
Highlighted
David_Oswill
1 Copper

Re: Scalability - Is Atmos capable of storing more than 5 billion objects ?

The limits you have mentioned are recommendations provided to application developers to help ensure their applications perform optimally when using listable tags. These guidelines apply for a given Atmos subtenant and are not a limit across the entire system. When an application uses non-listable tags there are no such stipulations.

We are working on a release that addresses the listable tag limit, but I don't have any further information at the moment.

As for namespace, the recommendations are given for the same reasons as you would see similar recommendations for a traditional storage system. Supporting 100,000+ files in a directory can be burdensome for an application trying to enumerate the directory.

If you don't need the traditional path/filename reference to your objects, I would suggest you use non-listable tags for your meta data and manage the objects using their object ID references.

aronb1
1 Copper

Re: Scalability - Is Atmos capable of storing more than 5 billion objects ?

Thank you for your reply.

It is enouraging to hear that you are working on a release that addresses the listable tag limit.

The issue is that I am trying to port an application from a competing cloud storage vendor to support Atmos. Our existing vendor did not impose such limitations when using the filesystem-like API, thus making our migration very tricky ....

Also see the second question that I posted today, which is complicating migration to OID mode .

0 Kudos