Unsolved
This post is more than 5 years old
2 Intern
•
718 Posts
9
22994
Ask the Expert: Are you ready to manage deep archiving workloads with Isilon’s HD400 node and OneFS 7.2.0? Find out more about the Data Lake Foundation products
Welcome to the EMC Isilon Community Ask the Expert conversation.
|
|
Ask The Expert – Isilon’s New Releases: IsilonSD Edge, OneFS.NEXT and CloudPools Ask the Experts: EMC Isilon technical content and documentation |
Our Experts today are here to answer any and all questions you may have related to the Data Lake Foundation product launches from earlier this month. As a quick recap, our experts are here to talk about the latest Isilon OneFS operating system (version 7.2.0), Insight IQ 3.1 and the newest storage platform for high density, deep archiving workloads, called HD400. Additionally we have a guest speaker George Hamilton to answer any questions about the Elastic Cloud Storage solutions and how it works as part of your Data Lake Foundation. This discussion is now open for questions. If you missed the virtual launch event, you can view the video on demand here.
Meet Your Experts:
Moderator: Niki Vecsei
This discussion takes place from Feb. 19th - March 8th. Get ready by bookmarking this page or signing up for e-mail notifications.
Share this event on Twitter or LinkedIn:
>> Announcing #DataLake Foundation with #EMCIsilon and #EMCECS - Join the Ask the Expert discussion! http://bit.ly/1KpJxLQ #EMCATE <<
Bhakti1
23 Posts
0
March 2nd, 2015 12:00
Can we check on which user is using a specfic file type? pictures or video or .bak files? Users exceeding advisory quotas and soft quotas?
karthikrm1
19 Posts
0
March 2nd, 2015 12:00
IIQ 3.1 has new reports on Smartquota. Are there any specific reports you are looking for?
Nikschen
179 Posts
0
March 3rd, 2015 10:00
George
We heard a lot about HD400 and OneFS 7.2. Can you tell us more about how ECS fits as a data lake foundation?
Nikschen
179 Posts
0
March 3rd, 2015 11:00
Thank you George, this is interesting stuff. If you don't mind my asking; what new possibilities does the inclusion of ECS as a Data Lake Foundation enable?
eghamilton
28 Posts
0
March 3rd, 2015 11:00
Sure! And thanks for asking Niki. As a data lake foundation, ECS fits when the customer desires object storage as the foundation for their storage. ECS is object based and provides a very low cost, highly dense storage platform for Hadoop. ECS consists of low cost commodity components, architected and designed to deliver an enterprise class storage platform that can scale to hundreds of petabytes, even exabytes. The benefit for customers is that they get the economics of commodity but also get the reliability, availability and serviceability (RAS) typical of integrated storage platforms; fully supported by EMC.
EMC has a couple customers using ECS as a large, multi-site archive for unstructured content. The HDFS data service on ECS allows them to bring analytics to those deep archives. Metadata querying is an example. They can also support Web, mobile and cloud apps written to object storage APIs such as Amazon S3, OpenStack Swift and EMC Atmos. ECS addresses some of the limitations of traditional HDFS by featuring a hybrid distributed erasure coding mechanism that protects data across multiple sites with very low overhead. And ECS can maintain that low overhead without sacrificing accessibility of the data. In fact, data can be read from and written to any site in a multi-site environment – even in the event of a site failure. ECS also supports multi-tenancy so an enterprise or a service provider can offer Hadoop-as-a-Service (HaaS).
karthikrm1
19 Posts
0
March 4th, 2015 08:00
We do have reports on quota capacity. There isn't a report looking at users access to a particular file type. But with the audit capabilities on OneFS 7.2, there are other ways to track this information.
karthikrm1
19 Posts
0
March 4th, 2015 08:00
Having ECS as part of the data lake foundations gives customers the choice of file with Isilon and object with ECS for their workloads. We have number of future projects with Isilon and ECS together that will further enhance the data lake foundation products.
eghamilton
28 Posts
1
March 4th, 2015 09:00
There are several new possibilities that ECS brings as a data lake foundation.
First, it’s an object storage platform. In addition to the object APIs, ECS also provides HDFS access to that same object data. HDFS is presented by ECS as an access API on top of the object storage engine. So organizations can bring analytics to their object data without have to move or copy the data or subject it to an ETL process. As an example, one customer is using ECS as a global archive growing to hundreds of petabytes. They can now bring analytics- such as metadata queries to their large, global archive to capture new business insights.
Secondly, ECS also opens new possibilities for efficiently ingesting and storing both very large or very small files. Object has always been good at handling large files, but small files were problematic. ECS has a feature called box-carting which it enables it to efficiently process a large volume of files with high IO – like sensor or telemetry data. ECS can package multiple writes in memory and execute them as a single write to the underlying storage. This improves the performance of the system. It’s also very efficient for large files since ECS will split the large write into multiple, independent pieces which are then processed in parallel across multiple nodes. In that way, ECS takes advantage of all spindles and NICs in a cluster, which enables applications to use the full bandwidth of every node in the system to achieve maximum throughput. So whatever size file, ECS can handle it efficiently.
Lastly, ECS is a multi-tenant cloud storage platform so both enterprises abnd service providers can offer Hadoop-as-a-Service (HaaS)
geekfyi
3 Posts
1
March 5th, 2015 18:00
Are there any new enhancements or features planned to better manage the complexities associated with large clusters containing mostly small files? For example, combining or consolidating jobs that reference or require the same information, or perform redundant tasks.
RobertoAraujo1
2 Intern
2 Intern
•
718 Posts
0
March 9th, 2015 06:00
This ATE events has ended. We would like to thank all those who participated in this discussion, but special thanks to our experts who took their time from their busy schedule to answers our user's questions.
Cheers.
karthikrm1
19 Posts
0
March 9th, 2015 09:00
Brian, we've made a number of enhancements in the snapshot delete performance in OneFS 7.2. This is applicable to both small file and large file environments in large clusters. There isn't a specific feature to small file efficiency that I would call out. We've made quite a lot of improvements for small file efficiency in 7.1.1 and continue to do so in future OneFS releases.
Julien_Fontaine
951 Posts
0
March 10th, 2015 03:00
Is the ask the expert session still open ?
Nikschen
179 Posts
0
March 10th, 2015 09:00
Hi Julien,
the event is over but the discussion thread will stay open for ever. You are welcome to post your question and we will endeavor to get the SME's to answer as soon as possible.
JGroce213
16 Posts
0
March 11th, 2015 14:00
Hello,
Hope this finds you well.
This looks great as a scale out solution. I was wondering if you could point me to some documentation/white papers on best practices for integrating this solution in with OpenStack. Either Swift or Cinder...or both =)
Thank you so much for your assistance,
Jason
karthikrm1
19 Posts
1
March 11th, 2015 15:00
Jason,
OneFS 7.2 supports Openstack Swift - You can find the paper here:
https://www.emc.com/collateral/TechnicalDocument/docu56064_onefs-swift-technote-7-2.pdf
thanks,
- KR