This post is more than 5 years old
12 Posts
0
3690
Atmos API for listing objects w/o giving any tags
So there are APIs to list objects but we need to give a tag. I was wondering if there is an API or an option to just list the objects belonging to a uid. The goal is to have a way to list all the objects and their (size, ctime system metadata values).
I'm trying using the Rest WS API to do this with tags value of "size" and get no objects found (1003).
From the DOCS>>>
This operation retrieves all object IDs indexed by a given tag. To
specify the tag, use the x-emc-tags header; only one tag
name/hierarchy may be included. Listable tags are created in a user’s
own namespace; hence, they are private to that user. Only objects
belonging to the requesting UID are returned.
Object
Interface—without
Metadata
Request The x-emc-include-meta header, set to 0, indicates that only object IDs
should be returned.
GET /rest/objects HTTP/1.1
accept: */*
date: Wed, 18 Feb 2009 16:39:49 GMT
content-type: application/octet-stream
x-emc-date: Wed, 18 Feb 2009 16:39:49 GMT
x-emc-tags: size
host: 168.159.116.96
x-emc-uid: 33115732f3b7455d9d2344ddd235f4b9/user1
x-emc-signature: ZllFtIyYe6kvqibS9eqcIBpiQ7I=
x-emc-include-meta: 1
<<
rbala1
222 Posts
0
February 4th, 2011 05:00
That's correct -- your application would create a series of listable tags to group objects together by some convention known and understood by the application.
Unfortunately, there's not a way to automatically have Atmos itself add the tags. We expose this level of functionality in the API to give the developer maximum flexibility in determining the optimal way to categorize the objects.
I'm not certain as to future functionality where one can list objects by system tags, but it's a good suggestion and I can raise it as an enhancement with the PM team.
ragr1
12 Posts
0
February 2nd, 2011 21:00
I figured some of this, but still looking for a solution.
It seems I can list objects if I use listable tags (user tags) and ask for system metadata like size and ctime as well. But in our case not all object have or will have user metadata tags, so in that case how would I be able to list all the objects.
All I'm looking for is to list objects with specific system metadata like size and ctime w/o specifying any user metadata tags.
Any suggestions?
rbala1
222 Posts
1
February 3rd, 2011 07:00
Rag,
There are a few questions here. Let me know if this answers them.
1. It's not possible to make a single request to return all objects belonging to a given UID. One would have to iterate through either the metadata or directory hierarchy (depending on whether namespace and/or object interface was used) to build a list. As you can imagine this isn't an inexpensive operation.
2. To list objects without specifying a listable metadata tag you would need to use the namespace interface and make GET requests to a given directory. We don't currently support the x-emc-include-meta header when using the namespace interface, but that support is forthcoming.
3. Error 1003 refers to "object not found." Are you certain that the object exists? I just tried to get system metadata on a given object and filtered the results using x-emc-tags and it returns the expected result. Here's an example of the headers that I used:
GET
application/octet-stream
Thu, 03 Feb 2011 15:08:05 GMT
/rest/objects/4980cdb2a910109804bd104db3990904d2365dae4aa1?metadata/system
x-emc-date:Thu, 03 Feb 2011 15:08:05 GMT
x-emc-tags:size
x-emc-uid:d131c2805d244b1a9e91aefb4b545ebc/NA000FBBFE5BE6CEBFBE
x-emc-signature:7t/R7n4Nw+bQfvZZVsvOzYaz4vg=
Returns: {'size': '4362018468'}
Raj
ragr1
12 Posts
0
February 3rd, 2011 09:00
Hi Raj,
Thank for the details. What would you recommend I do if I wanted to get a list of objects and some of their system metadata like ctime, and size. I don't need to get all the objects if it will be expensive operations, I can restrict to a list of objects created/updated within a time window like 12 hrs or 24hrs.
Thank you.
Rag
rbala1
222 Posts
0
February 3rd, 2011 12:00
How many objects do you expect to be created within that window of 12 - 24 hours? Are the object creates spread out evenly during that time?
ragr1
12 Posts
0
February 3rd, 2011 15:00
That could be anything. I'm wondering if you have different approaches based on the expected numbers. For discussions sake, lets say 0 to 1000 for now. I'm sure someone could say the upper limit is 1million objects
rbala1
222 Posts
1
February 3rd, 2011 18:00
My suggestion would be to partition the objects into a reasonable, but relatively evenly distributed series of metadata tags or directories depending on whether you're using the namespace or object interface. Also, consideration should be taken as to only store a reasonable number of objects for each given tag or directory for performance reasons related to enumerating large numbers of objects.
Your application could then call list objects or list directory on each tag or directory for the specified period and then combine the results.
Here's a simple example of a structure where an application could store objects using the hour with which they were created in the tag or directory.
YYYY/MM/DD/HH
So all objects created at 6AM on Feb 4, 2011 would be in
2011/02/04/06
ragr1
12 Posts
0
February 3rd, 2011 20:00
Does it mean one has to create a listable user tag with date or some other "partioning" data value so that can be used to list objects?
Is there a way to configure Atmos so some kind of user tag is automatically added? Otherwise, we have to enforce each client and API (or wrap your API) to insert specific user tags so we can use them to list objects.
Best would be to allow system tags to be used for listing - is that a considertion on the roadmap?
Of course, one other alternative inovling more development is to use Atmos log files to report on object level activity.
Thank you for all the help.