Ask the Expert, EMC Business Data Lake: Fully Engineered, Simple to Configure and Run at Scale

Welcome everyone, this ATE discussion has began. Feel free to post your questions and our SMEs will be sure to answer them before the end of our event. Let's keep this discussion respectful and informative. Thanks!

Which applications do you regularly use for big data analytics for which you’d see high value in a validated, simplified deployment onto the Data Lake? 

Please describe the use case and any pain points in ease of deployment and usage today

Deduplication needs to be addressed at some point to prevent the bloating of the Lake with redundant, raw, data entries.

That said, it needs to be done intelligently so that even though the raw data is not duplicated, relevant unique metadata surrounding that raw data is persisted to support subsequent analytics and search.

In your sample, the sensor raw data may be the same, but also as important is to "which sensor" captured the data "when" from "where" - as examples.

Arguably similar to unstructured content in an enterprise. The content store, url, and acls are sometimes as interesting as the content itself (re: data governance).

A Data Catalogue can be defined to have entries for each item in the Lake, each with its own specific metadata, but also with a many-to-one relationship, pointing to a single, raw data artifact.

What value do you see in pre-filtering the data lake before the data is pooled - when dealing with massive lists of repeated (but not redundant, necessarily - such as many sensors' reporting of the same data many times) data, is deduplication a possibility to retain the uniqueness of the points in time but eliminate the repetition?

Excellent suggestion. The dedupe algorithm should work equally well at ingest as when the data is at rest.

I like the "configurable approach" very much as the "knowledge of the characteristics of the data" should guide the most suitable best practice configuration.

Ted Bardasz

Have you considered virtualizing your Hadoop environment? How about in conjunction with a shared storage model? What benefits or drawbacks have you seen?

Good one

Hi

It would be perfect to have a section in Business Datalake on ECN that explains how to make the app, i saw that Cloudera was on the stage and a partner, but it would be nice to see  som use case exampels on how they did it for at EMC Business Datalake customer.

we see most common sensor and statistics data from engineering companies. Seiscmic and Nautic Data live Sea Sounds for anaylyzing movement of ships and their impact on the Sea is a Big Lake i have currently seen, since that data is also kept to analize movements of foreig ships . . .

When it comes to data protection of data lake nowadays and trends, are we talking about:

a) data protection (and archiving) at levels of data lake components

b) data protection (and archiving) at levels of data lake manager

c) hybrid

d) something else

I believe latest mantra is that that we no longer talk about backup, but rather management of copies, but I still miss that plugin or product which enables integration with this new modern view (not to mention legacy environment and compliance management).

Are Tier aproach valid to Data Lakes? Like Tier 1 Isilon S or X Series, Tier 2 NL or HD Series and Tier 3 LTFS (Linear Tape Filesystem). Are any integration or solution with this type of aproach?

Regarding Data Protection,

Does EMC has any specific product to use with data lake? We have great products, but not sure they will fit for this role.

Regards.

Hello Petter,

This is a great suggestion.  Please follow our community Everything Big Data and we will plan on posting an example of making an app.

Everything Big Data at EMC

Thanks,

Mona Patel

This Ask the Expert has concluded. Many thanks to those who participating with their questions, but especial thanks to our SMEs who loan their time to answer questions.