Ask the Expert: Object Storage into the Cloud

This Ask the Expert session is now open for questions. For the next couple of weeks our Subject Matter Experts will be around to reply to your questions, comments or inquiries about our topic.

Let’s make this conversation useful, respectful and entertaining for all. Enjoy!

Robert, thanks for setting this up.

Can someone outline the Best Practices for Centera to ECS Migrations?  Also, is there a formula for calculating how quickly the migration will complete?

Thanks,

Mike Horgan

I will certainly contact Engineering and see if any test data can be made available.

This may just take a little time so please bear with me for a few days.

For Ecs transformation (so all reads and writes are just to ECS), the indexing side of things is quite quick and only needs to populate tables with queries to Centera.

For actually migrating the data I will revert as soon as I can

regards

Dean

Excellent, thanks Dean. I'd be very interested to know a baseline ECS Transformation duration expectation for something like 100M (100,000,000) small clips, say 50KB each from an imaging application.  Lets assume the Centera has 16 nodes and 4 access nodes which would seem pretty typical.

Best Regards,

Mike Horgan

Hello Mike

There are a number of tools to migrate Centera to ECS but ECS Transformation has the most advantages.

With ECS Transformation, (built into ECS from V2.2) Centera data can be indexed within ECS before the data itself is actually migrated.

Transformation is initiated from ECS and once the data to be migrated is indexed, the application can cutover at that point.

The application would read and write directly to ECS. ECS can retrieve indexed data from Centera with minimal performance cost.

At any time the migration of the data itself can then be started in the background.

There are no official figures as to how long a migration would take as there are so many variables. We can certainly ask to see if the Engineering test figures can be made.

Dean

Timely thread for me! 

Can someone explain the technical difference between Atmos and ECS or Point me to a detailed comparision?  I have looked at a number of "technical" presentations and everything I see suggests that ECS is an update/rewrite of Atmos with a few  changes such as containerization, xfs and a more modern stoage algorithm.  Conceptually, I see no major difference and the HW is basically identical. 

Thanks.

Simon

Hi Simon,

Though the hardware of ECS and Atmos can be similar and both have support multi protocol access (Note: ECS also has support for Hadoop and OpenStack Swift, refer to the ECS 2.2.1 Data Access Guide for details on ECS API support) , the underlying Architecture layers in each product work very differently. Comparing the documents below should help you here I think. Let me know if you need anything clarified.

ATMOS

Atmos 2.4 Administrators Guide -> https://support.emc.com/docu68366_Atmos-2.4-Administrator%E2%80%99s-Guide.pdf?language=en_US

Atm,os 2.4.1 Relesae Notes -> https://support.emc.com/docu70762_Atmos-2.4.1-Release-Notes.pdf?language=en_US

ECS

ECS 2.2.1 Planning Guide -> https://support.emc.com/docu70102_ECS_2.2.1_Planning_Guide.pdf?language=en_US

ECS 2.2.1 Data Access Guide -> https://support.emc.com/docu70101_ECS_2.2.1_Data_Access_Guide.pdf?language=en_US

ECS 2.2.1 Administrators Guide ->

https://support.emc.com/docu70100_ECS-2.2.1-Administrator's-Guide.pdf?language=en_US

Good morning guys,

I have currently set up an ECS 2.2 Single Node Docker.  I have been using this for archiving data to this endpoint, my problem is, I have used up all the space on the endpoint, after seeing this I then deleted the archives that I had sent to this endpoint but I dont see the available space increasing.  I use the EMC2 Dashboard.to monitor the space and efficiency of the endpoint.

Is there a way to reclaim the space?

I appreciate any help that can be given.

Thanks

Emma

Hi Mike,

While Dean looks into your query there is a Centera to ECS migration White Paper which you may find useful available @ http://www.emc.com/collateral/white-papers/h15115-centera-to-ecs-migration.pdf

Hi Emma

Thanks for your question.

When data is written to ECS, it gets appended into chunks (like containers)

When the whole chunk is empty the space is eligible for space reclamation by background Garbage Collection tasks.

How long ago was the data deleted and what would be the average file size?

regards

Dean

The biggest difference is the policy engine on ATMOS.

If application simply need to set/update the proper metadata of an object. With the corresponding policy definition on ATMOS, these user- or system metadata are used as trigger for different protection/distribution of objects. This also covers retention enforcement, erasure service or migration to “public” cloud (if this is an ATMOS or ECS system).

--

With kind regards/Mit freundlichen Grüßen

Helmut Gotsche

Advisory Systems Engineer, EMEA

Object Storage

Software Defined Solutions@Emerging Technology Division

EMC² Computer Systems Austria GmbH

(Firmenbuch Gericht: HG Wien, FN 160018x, VATno: ATU43027201)

Am Euro Platz 1/4,

A-1120 Wien / Austria

Phone: +43 1 599 52 490

Fax: +43 1 253 67 22 1830

Mobile: +43 664 411 7466

e-Mail: Helmut.Gotsche@emc.com

More about ViPR: http://www.emc.com/cloud/vipr/index.htm

More about Atmos: http://www.emc.com/storage/atmos/atmos.htm

More about Centera: http://www.emc.com/archiving/centera.htm

For Developers: https://community.emc.com/community/edn/

The information contained in this communication is confidential.

It is intended solely for the person(s) to whom this email message is addressed.

If you have received this email message in error, please notify the sender immediately by telephone or email

and destroy the original message without making a copy.

P Please don't print this e-mail unless you really need to.

Thank you.

Thanks Michael. I browsed the paper and it looks very well done, kudos to the author.

Best Regards,

Mike Horgan

Hi Dean,

The data was deleted 2 days ago (Tues 21st).  Hard to say what the average size was to be honest.  We were archiving projects that were between 2GB and 240GB.

I was testing archiving a project with 0.5TB to the endpoint that only had 300GB space available, I wanted to see what would happen and how it would handle the large file.  It failed at 240GB and the archive was deleted (if an archive fails with our software, the archive is then deleted from the destination) however it still registered on the dashboard that the space was used and we were unable to archive large projects.

I hope this helps.

Thanks

Emma

Hi Emma,

ECS Free and Frictionless is unfortunately outside the scope of our expertise here, and I was going to repost your issue on the main ViprECS community, but I see you have already done so here

Best regards

Dean

https://community.emc.com/thread/230083

A question concerning XOR recovery.

Assume that we have 8 DataCenters/VDCs (as shown in the ECS Storage Overhead illustrations in various places),  how many sites can we lose and still be able to recover all data? 

Thanks.

Simon