Multiple ecs-sync server support

Hello,

There a couple of different ways to approach the question of scaling.

• For a single set of objects, such as a Centera pool, split up the object list, such as a CLIP list, then copy each split to the different ECS Sync VMs.
• For multiple sets of objects, such as multiple Centera pools, copy each separate object list to individual ECS Sync VMs.

Proper consideration needs to focus on network bandwidth of the aggregate ECS Sync traffic as well as VM resources.

Cheers,

/Chris

Currently, there is no orchestration component built in to ecs-sync, so managing a cluster is done manually.  This is accomplished via list files (clip lists and object lists).  To split a bucket across multiple jobs would require generating a complete list of keys from the bucket and then splitting that list just as Chris mentions above.

However, S3 scales a bit differently than other storage plugins and you can use a much higher thread count.  I recommend trying that before adding more VMs.

Note that the typical problem with large S3 buckets is that listing the bucket can be quite slow.  If ecs-sync performs the list operation, it will only go as fast as results are returned.  Please make sure your ECS is on the latest software revision, as this may come with a performance boost.  3.1 was just released, so speak with your account rep to schedule an upgrade.

Also note that a file system (NFS) bucket will list slower than a standard bucket, since there is a hierarchy that must be traversed and aggregated.

Appreciate the information, thank you!