Ask the Expert: Are you ready to manage deep archiving workloads with Isilon’s HD400 node and OneFS 7.2.0? Find out more about the Data Lake Foundation products

Can we check on which user is using a specfic file type? pictures or video or .bak files?  Users exceeding advisory quotas and soft quotas?

IIQ 3.1 has new reports on Smartquota. Are there any specific reports you are looking for?

George 

We heard a lot about HD400 and OneFS 7.2. Can you tell us more about how ECS fits as a data lake foundation?

Thank you George, this is interesting stuff. If you don't mind my asking; what new possibilities does the inclusion of ECS as a Data Lake Foundation enable?

Sure! And thanks for asking Niki. As a data lake foundation, ECS fits when the customer desires object storage as the foundation for their storage. ECS is object based and provides a very low cost, highly dense storage platform for Hadoop. ECS consists of low cost commodity components, architected and designed to deliver an enterprise class storage platform that can scale to hundreds of petabytes, even exabytes. The benefit for customers is that they get the economics of commodity but also get the reliability, availability and serviceability (RAS) typical of integrated storage platforms; fully supported by EMC.

EMC has a couple customers using ECS as a large, multi-site archive for unstructured content. The HDFS data service on ECS allows them to bring analytics to those deep archives. Metadata querying is an example. They can also support Web, mobile and cloud apps written to object storage APIs such as Amazon S3, OpenStack Swift and EMC Atmos.  ECS addresses some of the limitations of traditional HDFS by featuring a hybrid distributed erasure coding mechanism that protects data across multiple sites with very low overhead. And ECS can maintain that low overhead without sacrificing accessibility of the data. In fact, data can be read from and written to any site in a multi-site environment – even in the event of a site failure. ECS also supports multi-tenancy so an enterprise or a service provider can offer Hadoop-as-a-Service (HaaS).

We do have reports on quota capacity. There isn't a report looking at users access to a particular file type. But with the audit capabilities on OneFS 7.2, there are other ways to track this information.

Having ECS as part of the data lake foundations gives customers the choice of file with Isilon and object with ECS for their workloads. We have number of future projects with Isilon and ECS together that will further enhance the data lake foundation products.

There are several new possibilities that ECS brings as a data lake foundation.

First, it’s an object storage platform. In addition to the object APIs, ECS also provides HDFS access to that same object data. HDFS is presented by ECS as an access API on top of the object storage engine. So organizations can bring analytics to their object data without have to move or copy the data or subject it to an ETL process. As an example, one customer is using ECS as a global archive growing to hundreds of petabytes. They can now bring analytics- such as metadata queries to their large, global archive to capture new business insights.

Secondly, ECS also opens new possibilities for efficiently ingesting and storing both very large or very small files. Object has always been good at handling large files, but small files were problematic. ECS has a feature called box-carting which it enables it to efficiently process a large volume of files with high IO – like sensor or telemetry data. ECS can package multiple writes in memory and execute them as a single write to the underlying storage. This improves the performance of the system. It’s also very efficient for large files since ECS will split the large write into multiple, independent pieces which are then processed in parallel across multiple nodes. In that way, ECS takes advantage of all spindles and NICs in a cluster, which enables applications to use the full bandwidth of every node in the system to achieve maximum throughput. So whatever size file, ECS can handle it efficiently.

Lastly, ECS is a multi-tenant cloud storage platform so both enterprises abnd service providers can offer Hadoop-as-a-Service (HaaS)

Are there any new enhancements or features planned to better manage the complexities associated with large clusters containing mostly small files?  For example, combining or consolidating jobs that reference or require the same information, or perform redundant tasks.

This ATE events has ended. We would like to thank all those who participated in this discussion, but special thanks to our experts who took their time from their busy schedule to answers our user's questions.

Cheers.

Brian, we've made a number of enhancements in the snapshot delete performance in OneFS 7.2. This is applicable to both small file and large file environments in large clusters. There isn't a specific feature to small file efficiency that I would call out. We've made quite a lot of improvements for small file efficiency in 7.1.1 and continue to do so in future OneFS releases.

Hi Julien,

the event is over but the discussion thread will stay open for ever. You are welcome to post your question and we will endeavor to get the SME's to answer as soon as possible.

Hello,

Hope this finds you well.

This looks great as a scale out solution.  I was wondering if you could point me to some documentation/white papers on best practices for integrating this solution in with OpenStack.  Either Swift or Cinder...or both =)

Thank you so much for your assistance,

Jason

Jason,

OneFS 7.2 supports Openstack Swift - You can find the paper here:

https://www.emc.com/collateral/TechnicalDocument/docu56064_onefs-swift-technote-7-2.pdf

thanks,

- KR