Highlighted
3 Silver

How many buckets does CloudBoost create ?

Jump to solution

Hi, everyone. I know CloudBoost create two buckets by defualt. But can I create more bucket for CloudBoost to use?

0 Kudos
1 Solution

Accepted Solutions
2 Bronze

Re: How many buckets does CloudBoost create ?

Jump to solution

CloudBoost will never create additional buckets, except as I mentioned already for Atmos or OpenStack Swift.

Also the largest CloudBoost object is less than a megabyte because incoming data is deduplicated, and compressed.


I guess that on Atmos or Swift if the incoming save set was very large CloudBoost would be forced to create a new directory or container to accommodate every new batch of 100,000 or million segments respectively.

View solution in original post

0 Kudos
10 Replies
Highlighted
2 Bronze

Re: How many buckets does CloudBoost create ?

Jump to solution

For object stores that have per bucket performance limitations CloudBoost does have the ability to do sharding across multiple buckets but there is no way to manually set the number of buckets used for this purpose.

Highlighted
3 Silver

Re: How many buckets does CloudBoost create ?

Jump to solution

Do you mean CloudBoost will create multiple buckets? If so, how many buckets CloudBoost could create?

0 Kudos
Highlighted
2 Bronze

Re: How many buckets does CloudBoost create ?

Jump to solution

It is not something that you need to know about as an administrator. It is just one of the many things the engineers need to deal with when using different object stores. CloudBoost will create however many are needed to hold the data on the object storage being used. Unless you are using an object storage provider that has per bucket limits CloudBoost will only create the two buckets, one for the deduped chunks, and one for the automatic backups it does of itself for DR recoveries.

For example Atmos has a limit of 100,000 blobs per directory (similar to a bucket) and OpenStack Swift starts slowing down at about 1 million blobs per container. The public object stores on the other hand do this load-balancing at a lower layer than the bucket level so CloudBoost does not have to do it itself.

Highlighted
3 Silver

Re: How many buckets does CloudBoost create ?

Jump to solution

Thanks a lot for your reply. One more question, will CloudBoost create multiple buckets when storing a big object or storing multiple objects simultaneously?

0 Kudos
2 Bronze

Re: How many buckets does CloudBoost create ?

Jump to solution

CloudBoost will never create additional buckets, except as I mentioned already for Atmos or OpenStack Swift.

Also the largest CloudBoost object is less than a megabyte because incoming data is deduplicated, and compressed.


I guess that on Atmos or Swift if the incoming save set was very large CloudBoost would be forced to create a new directory or container to accommodate every new batch of 100,000 or million segments respectively.

View solution in original post

0 Kudos
Highlighted
3 Silver

Re: How many buckets does CloudBoost create ?

Jump to solution

If the object storage provider with S3 API has per bucket limit, CloudBoost will create the third bucket or not?

0 Kudos
Highlighted
2 Bronze

Re: How many buckets does CloudBoost create ?

Jump to solution

CloudBoost automatically creates whatever buckets it needs. Although I should add none of the supported object stores that use an S3 API would require it.

0 Kudos
Highlighted
3 Silver

Re: How many buckets does CloudBoost create ?

Jump to solution

So do you mean that CB would create another bucket if the bucket has reached the limit of objects when the object storage is using S3 API?

0 Kudos
Highlighted
2 Bronze

Re: How many buckets does CloudBoost create ?

Jump to solution

No I do not mean that. There are no object stores that use the S3 API that have a limit.

0 Kudos