Reply to Message

Reply to Message

View discussion in a popup

Replying to:
6 Indium

Re: Optimal Write/Object Size in Atmos


This is a great question and one which needs a little background on the Atmos architecture. The Atmos onLine service comprises a massively scalable, parallel architecture which adds some level of overhead to transactions but which can also be leveraged to provide for concurrent write and read transactions. The service comprises a load-balanced front end with numerous back end access nodes which can be leveraged for multiple parallel transactions.

As you have pointed out there are essentially three ways to go about writing objects onto the system. 1) Writing a large object in its entirety 2) Writing a large object using multiple PUTs and 3) breaking up a large object into smaller objects for writing. Any of these methods are valid with the following caveats.

1) Writing a single large object to Atmos or Atmos onLine will effectively minimize the overhead on a per transaction basis but will not take advantage of the parallelism or concurrency of the system. You may also be exposed to network induced failures of the connection due to the longer transaction time of writing a single large object. A broken connection and subsequent retry can lead to what may look as very long latency.

2) Writing a single large object using multiple PUTs reduces the chances of network induced retries (such as the broken connection example above) but still operates as a single transaction and again does not leverage the concurrency of the system.

3) Breaking the large object up into multiple small objects will not only provide increased protection against network induced issues, but by writing multiple objects in parallel will allow accesses to be made to multiple access nodes in parallel, greatly speeding up the effective creation of the large object onto the Atmos system. By spreading these objects across the system, they can also be retrieved in parallel to increase the access time. The tradeoff is some added complexity in your application to create and then aggregate the object chunks.

For either 2 or 3, the recommended object size would be roughly 4MB so as to minimize the overhead associated with each object's creation.

As for network level parameters, these will depend largely on the local network as well as the internet characteristics and we would welcome any empirical data that you may be able to provide during your testing of the service.


0 Kudos