Highlighted
2 Bronze

What are optimal file sizes?

Jump to solution

We are currently using the Centera, and we have need to zip files for two cases:

  • Combine many very small files (it is our understanding that files under 50KB are inefficient)
  • Break apart very large files (to stay under the 100MB limit of Centera)

Moving forward to Atmos, what strategies should we consider for optimally storing files?  Is it fine to store many small files?  What is the maximum file size?

Labels (1)
0 Kudos
Reply
1 Solution

Accepted Solutions
Highlighted
3 Argentum

Re: What are optimal file sizes?

Jump to solution

Theoretically, there is no maximum object size in Atmos.  Overhead is a little less in Atmos, but not much.  If you already have bundling baked into your application, I would keep it around to be on the safe side.  If your objects vary greatly in size, you can also establish policies based on object size to help efficiency.  For example, you can have objects larger than 512k use erasure coding (parity) to reduce overhead.  This is covered on page 195 of the Administrator's Guide.

FYI, when ViPR Data Services is available (which supports the Atmos API), it automatically bundles small objects on the server side, completely transparent to the user.

View solution in original post

0 Kudos
Reply
1 Reply
Highlighted
3 Argentum

Re: What are optimal file sizes?

Jump to solution

Theoretically, there is no maximum object size in Atmos.  Overhead is a little less in Atmos, but not much.  If you already have bundling baked into your application, I would keep it around to be on the safe side.  If your objects vary greatly in size, you can also establish policies based on object size to help efficiency.  For example, you can have objects larger than 512k use erasure coding (parity) to reduce overhead.  This is covered on page 195 of the Administrator's Guide.

FYI, when ViPR Data Services is available (which supports the Atmos API), it automatically bundles small objects on the server side, completely transparent to the user.

View solution in original post

0 Kudos
Reply