The Dell AI Data Platform (AIDP) provides an abundance of value. In this blog, I describe the different areas. My growing list includes:

• Consolidation
• Reduction of legacy costs of one-time data sources
• Support for both analytics and AI on the same platform
• Flexible Open Table Formats (OTF) storage
• Support for the medallion architecture
• A high-performance query layer 
• Separate specialty engines for analytics, data processing, and search
• A choice of either migrating or keeping data sources as-is
• The opportunity to measure and improve data quality
• The ability to create curated data products
• A choice of object or file storage
• Support for either ETL or ELT

Consolidation is a simple concept yet often overlooked. Fewer data sources and platforms reduce complexity that can often be one of the biggest detractors from agility. Consider taking an inventory of all data sources and applications and then rationalize which to keep and which to retire.

One-time data sources can complicate support costs, agility, and compliance. For each source, consider performing an assessment by using all the dimensions and metrics that matter to your organization. Consider a simple spreadsheet exercise to help you understand quantitatively the burdened costs of these systems and data sources.

Data platforms generally support either analytics or AI. Few data platforms support both. AIDP is one of the few data platforms that can address both needs. This coverage is most readily accomplished by using a series of OTF tables that might inherit data from multiple other sources, not all of which must be in the data lakehouse. Data structures that support analytics and AI tend to be different. Analytics might have an overabundance of rollups and aggregates while AI might have a subset of the required data to use in a model to perform training.

AIDP supports OTFs such as Iceberg, Delta Lake, and Hudi. Each has its’ own advantages and disadvantages. For simplicity, only one format is typically chosen. A distinctive feature of AIDP is that you have a choice of formats.

The medallion architecture has been available for data warehouses for many years. It is a solid architecture, allowing for raw data to be stored in bronze tables, refined data to be stored in silver tables, and consumable data to be stored in gold tables. I would argue that it is even more useful in a data lakehouse than in a data warehouse. For more information about medallion architecture, see What is a Medallion Architecture?.

A high-performance query layer is especially important for the user experience. Slow queries can be the bane of existence for a legacy data platform such as Hadoop with Hive, Pig, and Impala. The AIDP includes Starburst software, which includes an MPP query engine that is similar to Greenplum and Teradata.  

The Starburst software included in the AIDP architecturally supports different node types. Node types can be used for analytics and queries, data processing with Spark, or search using elastic search. Each node type is optimized for a specified role. We have configurations for each to help with optimization.

For each data source in your data landscape, you have a choice of either migrating the data and then retiring it or leaving it undisturbed and querying it remotely. This option is extremely powerful as it allows you to retire uncomplicated data sources first and leave those tricky, complex interconnected data sources for migration or consideration later. The resulting data lakehouse is often a combination of local and federated data sources that you might query separately or by using joins.

Data quality in most data sources is lacking and is a key contributor to delays and even project failure. You can assess data quality by using open-source tools or a series of queries. Improvement of a select 10 percent of your most important data might make a huge difference.

The ability to create curated data products is a key functionality included in AIDP. A data product is an amalgamation of various data attributes into a well-understood collection of data that might support a single-use case. Typically, it is derived data from upstream sources and tables. It is manifested as an OTF table.

The storage for your data lakehouse in AIDP can be either object or file although not bothin the same cluster. For Dell Technologies, this means use of ObjectScale or PowerScale appliances respectively.

Like data warehouses in the past, AIDP data lakehouses can support either ELT or ETL. For some, this choice is integral or historical in that it has been used for years. With AIDP, we offer support for both processes; you can use both in the same cluster.  

If you want to talk about these subjects or delve deeper into your data platform needs, I am happy to schedule a call to discuss. Send me an email at:

 Mike.King2@Dell.com
#IWorkforDell