This post is more than 5 years old
4 Posts
0
622
What are optimal file sizes?
We are currently using the Centera, and we have need to zip files for two cases:
- Combine many very small files (it is our understanding that files under 50KB are inefficient)
- Break apart very large files (to stay under the 100MB limit of Centera)
Moving forward to Atmos, what strategies should we consider for optimally storing files? Is it fine to store many small files? What is the maximum file size?
chris_arnett
110 Posts
0
September 26th, 2013 11:00
Theoretically, there is no maximum object size in Atmos. Overhead is a little less in Atmos, but not much. If you already have bundling baked into your application, I would keep it around to be on the safe side. If your objects vary greatly in size, you can also establish policies based on object size to help efficiency. For example, you can have objects larger than 512k use erasure coding (parity) to reduce overhead. This is covered on page 195 of the Administrator's Guide.
FYI, when ViPR Data Services is available (which supports the Atmos API), it automatically bundles small objects on the server side, completely transparent to the user.