This post is more than 5 years old
6 Posts
0
1424
Writing objects directly to the ECS store
Hi all,
In the ScaleIO subforum, I've been pointed to see ECS for object
storage. From what I read, it is API-compatible with S3, Swift, Atmos
etc. Is there any tool, however, that can be used to read/write objects
directly to the ECS store, without the HTTP overhead, like Ceph does
with librados?
Thanks,
Alex
JasonCwik
281 Posts
0
August 18th, 2015 14:00
Hi Alex,
ECS does not currently export any access to its internal protocols. Can you expand upon your use case for direct access? What are you trying to achieve?
Alex_Pyrgiotis
6 Posts
0
August 19th, 2015 02:00
Hi Jason,
You're right, I should have provided more context for our use case. You
can take a look at the thread in ScaleIO that was the basis for my
question.
You'll see that in both of my questions, I'm searching for a librados
equivalent tool in ScaleIO or ECS. In case you don't know what librados
does, it's a library that implements all the low-level operations on a
RADOS cluster. It also provides an API that is the building block for
most of Ceph's interfaces: Ceph Object Gateway and RBD.
Now, why would a third party need such a tool? The reason is that such a
client tool enables one to write a custom interface (business logic) over the
underlying object store. That's what we need and currently do with Ceph.
So, I'll reiterate my question. Given that ECS has two interfaces and
one on the making (Object, HDFS and NFS), is there any client tool that
provides the basis for the above and, most importantly, performs object
requests directly to the ECS store?
JasonCwik
281 Posts
0
August 19th, 2015 13:00
Yes, there is an underlying protocol inside ECS, but it is currently not documented and supported for customer use. I'll forward your request on to product management; this request has come up once before.
Are you simply looking to reduce your latency vs. HTTP-based protocols? What kinds of latency requirements are you looking at? What I/O patterns does your application perform?
Alex_Pyrgiotis
6 Posts
0
August 20th, 2015 02:00
Yes, this is probably what we're looking for. We'd be very interested to know
more.
The benefits of using a librados-equivalent tool are two-fold:
a) First, we reduce the I/O latency with the removal of the HTTP
protocol but, most importantly, we can do I/O asynchronously with callbacks.
b) Second, we can use features of the object store that are not exposed
by the Object interface, such as locking, snapshotting, etc.
As for the latency requirements, we're hoping for at most 500us added latency
to the native speeds of our storage devices. This should roughly cover the cost
of the network latency and internal ECS replication. Also, our hard I/O pattern
consists of an equal amount of reads/writes to multiple new and old objects at
the same time.