Organizations must find ways to fully use their data in today’s data-driven environment while assuring scalability, security, and agility. The modern data stack architecture described in this white paper combines Dell infrastructure, Red Hat OpenShift Container Platform, Starburst query engine, and open-source technologies like Delta Lake and Apache Iceberg. This architecture enables data management transformation and gives businesses access to unmatched data capabilities.

The expansion of data sources and formats has led to complex data ecosystems, making it challenging to extract insights to make informed business decisions effectively. Traditional data warehouses were optimized for BI and decision support but with only well-defined and structured data. Furthermore, data warehouses require the data to be curated using the extract-transform-load (ETL) process, which can be time-consuming, before users can draw insights from it.

Dell’s end-to-end validated design brings together the platforms (servers, storage, networking, and software), services, OPEX pricing options, and on-demand and self-service capabilities. The Dell Validated Design for Analytics - Modern Data Stack includes Red Hat OpenShift Container Platform – the cloud platform layer for the modern data stack. It also includes Starburst Enterprise – a fully supported, enterprise-grade query engine; Apache Spark; and open table format technologies such as Delta Lake and Apache Iceberg.

This modern data stack addresses a new generation of analytics challenges that arise from extracting actionable data from massive datasets. It delivers a unified repository, eliminating data silos and simplifying access while providing data quality, security, and governance, ensuring compliance. It puts data scientists and data engineers in real-time control of the design and the deployment of workloads.

This document discusses a cutting-edge data management solution. It helps personnel who are involved in analytics, data-center modernization, or designing service offerings for data lakes and data warehouses to better serve their customers. Modernized infrastructure and data transformation initiatives help IT shops, analytics providers, and data engineers deploy and use modern data stacks. Data analytics becomes more powerful, capable, and relevant to the goals of the organization.

This document is intended for enterprises with data lakes or a data lake strategy interested in empowering their organizations to act more quickly, effectively, and efficiently on their data. Audience roles include:

• Data and application administrators
• Data engineers
• Data scientists
• Hadoop administrators
• IT decision-makers

A modern data stack can assist more traditional analytics customers looking to modernize their data collection. It can also help analytics systems to get more value from their data or standardize their data for modern analytics workloads.

Rapid advancements in artificial intelligence (AI) and machine learning (ML) have transformed the business landscape. Given the growing popularity of hybrid cloud computing, businesses face the challenges of establishing an enterprise data platform that seamlessly integrates with both on-premises and public cloud providers. They also seek self-service analytics and data mesh principles in a cost-effective way. While these technologies offer exciting opportunities for innovation and data-driven decision-making, they also introduce complexities in storage, integration, and processing.

The types of data generated and collected by businesses have expanded significantly. Organizations have historically dealt with structured data from traditional databases. Today, multinational organizations now deal with semistructured and unstructured data, such as text, logs, images, audio, video, social media content, and sensor data that spans across borders. These types of data cannot be placed in the same repository as structured data or in the same vicinity due to sovereignty regulations. Managing and analyzing this diverse data requires a modern data stack.

As data volumes and processing demands increase, traditional data management systems may struggle to scale efficiently to meet the growing needs of businesses. It is evident that the need for an enterprise-wide data management solution is a must have. However, integrating these disparate approaches can be complex and impeded by siloed data environments. These challenges require adopting new scalable and high-performance data management solutions that complement the traditional data stack. Technologies like data mesh are new data architectures that treat data as a product and each domain or business unit becomes responsible for managing and owning its data.

By implementing a data mesh architecture, organizations can associate distinct data architecture designs across various hyperscalers or on-premises. Doing so enables each team to leverage the best technology for their domain while ensuring compatibility and cohesiveness at the company level. This modern data stack addresses the issues of data silos, streamlines the data sharing process, and enables the organization to fully use its data assets. It encourages cooperation, innovation within diverse functional areas, and minimize costs.

Addressing these challenges requires adopting modern data management strategies, embracing innovative data technologies, and empowering organizations with the necessary tools and skills for effective data utilization. Businesses that can navigate these complexities and demands successfully are better positioned to leverage data-driven insights and gain a competitive advantage in their respective markets.

Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) are two common techniques used in data analytics. They prepare and move data from source systems to a target data warehouse or data storage for analysis.

The ETL process is typically used in a data warehouse architecture. The data is first extracted from source systems and then goes through the transformation process to cleanse the data to prepare for analysis. Once the data has been transformed, it is loaded into a target data warehouse or a data mart. This load process typically maps the transformed data into tables and structures in the target systems.

The ELT process, typically used in a modern data stack, starts with data extraction from source systems. This technique works well in a modern data stack architecture. Data is extracted and loaded into the target storage as-is, with little transformation. The transformation step is performed directly on the target systems using integrated processing capabilities without cleansing the data first. This approach takes advantage of the modern data stack’s compute power to transform the data as needed during the analysis phase.

ELT compared to ETL shows the differences between the ETL and ELT processes.

Both ETL and ELT methods can be used simultaneously, and each has its own benefits. The choice between them depends on factors such as data volume, data transformation complexities, and capabilities of the data warehouse or storage system being used for analysis.

In today’s data-driven world, organizations face the challenge of efficiently managing and delivering value from vast and diverse datasets. To address this challenge, a robust data management solution is essential.

This validated design offers an end-to-end on-premises or co-located modern data stack; an alternative to cloud-based modern data stacks. It includes the partnership between Dell Technologies and Red Hat to deliver a modern data platform powered by Dell infrastructure. This collaboration combines the latest generation Dell PowerEdge servers and software-defined storage with Red Hat OpenShift Container Platform, a cloud platform layer for the modern data stack. It also includes a suite of cutting-edge data management technologies like Delta Lake, Apache Iceberg, Delta Sharing, Data Mesh, and the Starburst query engine.

These integrated technologies enable customers to process, store, and analyze large-scale data workloads efficiently and cost-effectively. It empowers organizations to perform advanced analytics, including machine learning and artificial intelligence to get valuable insights and make sound business decisions.

The Dell Validated Design for Analytics — Modern Data Stack solution combines the strengths of PowerEdge servers, Dell object storage such as PowerScale, ECS, and ObjectScale, Red Hat OpenShift Container Platform, and advanced data technologies that include:

• Delta Lake and Apache Iceberg—Delta Lake and Apache Iceberg are ideal table formats for a modern data management platform. Each format has its own strengths and weaknesses. Delta Lake is best suited for data lakes and data pipelines. Iceberg is ideal for data warehousing and analytics with large tables in tens of petabytes of data.
• Delta Sharing—The industry’s first fully secured and compliant open-source protocol for data sharing. It enables organizations to seamlessly share existing large-scale datasets based on open-source formats Apache Parquet and Delta Lake without moving data while maintaining data consistency, security, and governance. Dell has successfully validated Delta Sharing with a Databricks instance on AWS. More information about how Dell validated Delta Sharing can be found in the Dell white paper Power Multicloud Data Analytics using Dell ECS and Databricks.
• Data mesh—A new data architecture that treats data as a product and each domain or business unit is responsible for managing and owning its data. By implementing a data mesh architecture, organizations can associate distinct data architecture designs across various hyperscalers or on-premises. These designs enable each team to leverage the best technology for their domain while ensuring compatibility and cohesiveness at the company level.
• Starburst query engine—Starburst is a fully supported and enterprise-grade distributed SQL query engine designed for high-performance analytics. It allows users to query large amounts of data stored in various data sources throughout an organization using standard SQL syntax. One of the key features of Starburst is its ability to run queries across different data sources simultaneously, in the same query. These sources include relational databases, NoSQL databases, object storage systems, and more. Response times are fast enough to support real-time analysis.

Some key benefits of this solution include:

• Red Hat OpenShift Container Platform—The industry-leading hybrid cloud application platform powered by containers and Kubernetes. It enables a cloud-native development environment together with a cloud operations experience, giving you the ability to choose where you build, deploy, and run applications, all through a consistent interface.
• Unified data repository—Centralize data from diverse sources into a single, scalable repository, and simplifying data access while supporting open table formats.
• Scalable and performant—Dell PowerEdge servers and object storage provide the scalability and performance needed to handle growing data volumes and analytics workloads.
• Advanced analytics—Enable data-driven decision-making with efficient analytics powered by the Starburst query engine.
• Time travel and schema evolution—Leverage time travel and schema evolution capabilities of Delta Lake and Apache Iceberg for agile data management.

The following figure shows a modern data stack architecture on Dell infrastructure.

The Dell Validated Design for Analytics — Modern Data Stack can run critical analytics projects featuring a faster time-to-insight cycle that results in better data quality and control. The net result for your organization is faster business intelligence and reporting.

Data engineers and data scientists can be empowered through the solution’s self-service, on-demand, tools and frameworks that can either be run on-premises or in a co-located facility.

Faster interactive queries, coupled with better data availability, facilitates more informed decision-making. And modern data stack operations like caching, indexing, and data compaction help to increase performance, enabling organizations to access and process the data to quickly drive outcomes.

With a modern data stack, your data management is simplified since, as stated above, structured, semistructured, and unstructured data can all be stored in the data lake. This design reduces the need for chasing, copying, or moving data between architectures. It gives data scientists and data engineers the ability to access all types of data from a single repository.

With support for ACID transactions, the Dell solution helps ensure consistency, even as multiple parties concurrently read, or write data. Complexity and guesswork are eliminated by leveraging the Dell Validated Design for Analytics — Modern Data Stack through Dell’s thorough validation processes.

Red Hat OpenShift Container Platform is the industry-leading hybrid cloud application platform powered by containers and Kubernetes. It enables a cloud-native development environment together with a cloud operations experience, giving you the ability to choose where you build, deploy, and run applications, all through a consistent interface.

OpenShift’s full-stack automated operations and self-managed clusters enable teams to adopt agile DevOps practices for building cloud-native containerized applications. They enable teams to work together more efficiently to move ideas from development to production.

Dell Technologies, together with Red Hat, delivers optimized, validated, and documented designs, with configuration and deployment guidance to help you rapidly implement Red Hat OpenShift Container Platform on Dell infrastructure.

Red Hat OpenShift Container Platform is popular with organizations looking to leverage containerization and Kubernetes-based orchestration for their application deployment and management. Some of the key benefits include:

• Scalability and high availability—Applications can scale to thousands of instances across hundreds of nodes and maintain high availability through automatic load balancing and failover mechanisms.
• Flexibility—Simplifies deployment and management of a hybrid infrastructure providing the flexibility to have a self-managed or fully managed service running on-premises or in the cloud and hybrid environments.
• Container portability—Container images built on the Open Container Initiative (OCI) industry standard ensure portability between developer workstations and OpenShift environments.
• Operator framework—Simplifies complex applications management by encapsulating operational knowledge and automating application life cycle management.

Starburst Enterprise is a fully supported and enterprise-grade distributed SQL query engine designed for high-performance analytics. It is built on top of the open-source Trino project (formerly Presto® SQL) and provides enterprise-grade security and governance features. It delivers data virtualization and federation at enterprise scale.

Starburst Enterprise allows users to query large amounts of data stored in various data sources throughout an organization using standard SQL syntax. One of the key features of Starburst is its ability to run queries across different data sources simultaneously, in the same query. These sources include relational databases, NoSQL databases, object storage systems, and more. Response times are fast enough to support real-time analysis. This capability is achieved using catalogs and connectors. Starburst Enterprise shows a Starburst cluster between users’ SQL client and the existing data sources that they want to query.

A Linux Foundation project since 2019, Delta Lake by the Linux Foundation is an independent project controlled by a development community rather than any single technology vendor. The Dell Validated Design for Analytics — Modern Data Stack enables reliable deployment and operation of Delta Lake in the solution. More than 150 developers from over 50 different organizations associated with Delta Lake working on multiple storage repositories are all engaged daily to help push the program goals forward. Key Delta Lake capabilities include:

• ACID transactions—Ensure data consistency and reliability and isolating it at the strongest isolation level, the serializable level.
• Time travel and data versioning—Each data write to a Delta table creates a version number. This feature enables users to query a Delta Lake table as of a specific time. Users can view and revert to previous versions of the data using a timestamp or a version number.
• Scalable metadata—Leverages Spark’s distributed processing power to easily handle all the metadata for petabyte-scale tables with billions of files.
• Schema evolution and enforcement—Perform automatic schema validation by checking against a set of rules to determine the compatibility of a write from a DataFrame to a table.
• DML operations—Support Data Manipulation Language (DML) operations like updates, deletes, and merges by using transaction logs. They enable easy handling of complex use cases like change-data-capture, slowly changing dimension (SCD) operations and streaming upserts.

Delta Lake 3.0 is the latest major release of the Linux Foundation open-source Delta Lake project, announced on June 28, 2023. This release introduces powerful features such as:

• Delta Universal Format (UniForm)—Enables reading Delta in the format needed by the application, improving compatibility and expanding the ecosystem. Delta will automatically generate metadata needed for Apache Iceberg or Apache Hudi, so users do not have to choose or do manual conversions between formats. With UniForm, Delta is the universal format that works across ecosystems.
• Delta Kernel—Simplifies building Delta connectors by providing simple, narrow programmatic APIs that hide all the complex details of the Delta protocol specification.
• Delta Liquid Clustering—Simplifies getting the best query performance with cost-efficient clustering as the data grows.

Delta UniForm automatically unifies table formats without creating additional copies of data by incrementally generating a layer of metadata to spec for Hudi, Iceberg, and Delta. Data users who use query engines designed to work with Iceberg or Hudi data can read Delta tables without copying or converting it. Delta Lake 3.0 shows the concept of Delta Lake with UniForm.

Apache Iceberg is a new open-source, high-performance data table format designed for large-scale data platforms. Its primary goal is to bring the reliability and simplicity of SQL tables to big data while providing a scalable and efficient way to store and manage data. This performance is especially important in the context of big data workloads. Iceberg makes it possible for engines such as Spark, Trino, Flink, Presto, Hive, and Impala to safely work with the same tables, simultaneously. It addresses some of the limitations of traditional data storage formats such as Apache Parquet and ORC.

Some key Apache Iceberg features and concepts include:

• ACID transactions—Iceberg tables guarantee atomic, consistent, isolated, and durable (ACID) transactions for write operations, ensuring data integrity.
• Catalog—A mechanism that houses metadata pointers for Iceberg tables.
• Table metadata—Maintains comprehensive metadata information about tables, including schema details, partitioning information, and datafile locations. The metadata layer consists of three categories:
  • Metadata files that define the table
  • Manifest lists that define a snapshot of the table
  • Manifests that define groups of datafiles that may be part of one or more snapshots
• Time travel—Supports the ability to query data at different points in time, enabling historical analysis and data rollback to specific versions.
• Schema evolution—Enables schema evolution by adding, renaming, or deleting columns without requiring expensive and time-consuming data migrations.
• Partitioning—Enables data to be partitioned into logical segments based on specific columns, such as date or region. Partitioning helps improve query performance by enabling efficient data pruning and reducing the amount of data scanned during queries.

Apache Iceberg table architecture depicts the Iceberg table architecture.

Designing a robust modern data stack requires careful planning and a well-structured architecture. In general, a modern data stack consists of five layers:

• Ingestion layer—This layer pulls data from various sources, including databases, IoT devices, applications, logs, and many more, and deliver it to the storage layer. There are tools to import data from RDBMS and NoSQL databases while streaming data can be imported using Apache Kafka or other open-source tools.
• Storage layer—In a modern data stack design, the storage layer is meant to keep all types of data in inexpensive object storage. This capability enables the client tools to work with these objects directly using open file formats.
• Metadata layer—The foundation of a modern data stack is the metadata layer. Its purpose is to provide a catalog to store metadata of the data and provides a data lineage which tracks data origin and transformation history of the data. This layer makes it possible to implement data management such as ACID compliance, data versioning, caching, indexing, and more.
• API layer—This layer hosts various types of APIs for data analytics. It also hosts machine learning libraries like TensorFlow and Spark MLlib which can read open file formats such as Parquet and query the metadata layer directly.
• Consumption layer—This layer hosts different tools and applications such as Tableau, Power BI, and others to access data stored in the data lake. This layer is where the users perform analytics activities such as running SQL queries, BI, and data visualization.

Modern data stack layers illustrates the layers of a modern data stack.

The end-to-end modern data stack requires careful consideration of various components and their capabilities. Dell Technologies has worked closely with its software partners to develop a solution that addresses the growing analytics and machine learning challenges. These challenges are the results of architectural differences between data warehouses and data lakes. The Dell Validated Design for Analytics - Modern Data Stack addresses the needs for organizations already down the path of deploying data warehouses and data lakes today. It also addresses the needs of organizations that have not deployed yet but recognize the value that advanced analytics and machine learning can deliver to their data-driven decision-making process.

The Dell Technologies strategy combines compute and storage into a seamless solution. It gives developers, data scientists, and data engineers access to a self-service, one-click deployment of advanced analytic and AI and ML pipelines, with no need for extensive infrastructure interaction. See Modern data stack.

This solution is designed for analytics, AI, and ML deployments across a wide range of environments. The validation process assures that the solution provides flexibility for organizations to configure according to their needs.

The Dell Validated Design for Analytics — Modern Data Stack consists of the finest hardware and software from Dell Technologies and its partners. These platforms have been carefully chosen for their ability to handle compute-intensive workloads and scale to the demanding requirements of modern data stack environments for data analytics, AI, and ML.

Hardware components of the Dell Validated Design for Analytics — Modern Data Stack system include:

• The Dell PowerEdge R660 server is perfectly matched for compute-intensive workloads while also minimizing the data center footprint through its 1U form factor. The PowerEdge R660 design enables business to easily scale while still handling challenging and emerging workloads. This dual-socket compute platform is ideal for the Red Hat OpenShift control nodes. Control nodes handle the resource allocation, scheduling, system monitoring while maintaining the state of the cluster.
• The worker nodes handle all the heavy lifting for the solution. The PowerEdge R660 server model was chosen for this purpose due to its ample performance and flexibility to efficiently handle containerized workloads. This server supports PCIe Gen5 for faster access and transport of data, optimizing application output. Also, this server has enough capacity to support GPUs and I/O bandwidth.
• The Dell PowerSwitch S5248F-ON delivers the top of rack switching. It is a 25 GbE/100 GbE open networking switch that provides state-of-the-art, high-density switching. The open networking capability provides extra flexibility for ever-changing network configurations. The software-defined networking enables a communications fabric that can change and adapt to the containerized compute cluster. The fabric optimizes communications paths when the compute demands as workloads are deployed and shifted across the worker nodes.
• The Dell PowerScale H7000 hybrid storage platform delivers a versatile yet simple scale-out storage architecture accessing massive amounts of data. With scalability of up to 1.28 PB per chassis and up to 8 GB/s of data bandwidth, the PowerScale H7000 can support demanding modern data stack environments. The OneFS-powered scalable storage speeds access to massive unstructured data stores that can help feed data-hungry applications and analytics. The PowerScale H7000 achieves up to 80% storage utilization compared with the 50% of traditional storage solutions. It is better suited for the demands of modern data stacks and continually changing underlying compute models.
• Some modern data stacks include workloads that demand both unstructured data storage and object-based storage. The ECS EX500 delivers the perfect blend of economy and density for modern applications or deep archive environments. With scalability of up to 16 nodes, the ECS EX500 can handle up to 6,144 TB of unstructured storage per rack. Dell ECS enables enterprise-grade cloud scale storage for unstructured (object and file) storage while maintaining control, in a private cloud environment. The software-defined storage is layered to help enable a limitless scalability while maintaining the abstraction that promotes high availability.

Modern data stack hardware list shows the hardware components that are used for the modern data stack solution.

Software components of the Dell Validated Design for Analytics — Modern Data Stack system include:

• Red Hat OpenShift Container Platform is the industry-leading hybrid cloud application platform powered by containers and Kubernetes. It enables a cloud-native development environment together with a cloud operations experience, giving you the ability to choose where you build, deploy, and run applications, all through a consistent interface.
• Starburst Enterprise is a fully supported and enterprise-grade distributed SQL query engine designed for high-performance analytics. It provides data virtualization and query federation across the enterprise, making it possible for all types of users to derive insights from various data sources.
• Delta Lake is an open table format that provides high-performance ACID table storage over object stores. Delta Lake leverages transaction logs and versioned Parquet files to enable update, insert, and delete operations on its tables.
• Apache Iceberg is an open table format designed for large, petabyte-scale tables. It uses SQL tables for big data, while making it possible for engines such as Spark or Trino to safely work with the same tables simultaneously.
• Apache Spark is an open-source, multipurpose unified analytics engine for large-scale data processing. It can be used for several use cases, but the most popular use case is to process streaming data in real time.
• Apache Kafka is a data processing platform that allows multiple client applications to publish and subscribe to real-time information with a scalable, distributed message broker architecture.

Partner software details shows the software components used for the modern data stack.

The Dell Technologies Customer Solution Center (CSC) is an important supporting organization for your analytics implementation. The CSC uses the Dell Validated Design for Analytics — Modern Data Stack and many other solutions to accelerate achievement of your goals and realize your digital future:

• Proof of Concept—Validate that your preferred solution meets your needs with a custom Proof of Concept. Dell Technologies solution architects enable practical, firsthand implementation based on your test cases.
• Design Session—Collaborate with Dell Technologies experts to design a solution framework. Brainstorm with our experts to explore your current IT environment, your future objectives, and potential solutions.
• Technical Deep Dive—Dive into the technical solution details that you are considering for your organization. Learn from live product demonstrations and solution-focused discussions with Dell Technologies subject matter experts.

Contact your Dell Technologies sales representative today to schedule a customized briefing or solutions engagement for this or any other Dell Validated Design.

Dell Technologies can provide a broad set of capabilities for implementing and maintaining solutions, linking people, processes, and technology to accelerate innovation and enable optimal business outcomes.

• Consulting Services help you create a competitive advantage for your business. Our expert consultants work with companies at all stages of data analytics. They can help you plan, implement, and optimize solutions that enable you to unlock your data capital and support advanced techniques, such as AI, ML, and DL.
• Deployment Services help you streamline complexity and bring new IT investments online as quickly as possible. Leverage our 30 plus years of experience for efficient and reliable solution deployment to accelerate adoption and return on investment (ROI) while freeing IT staff for more strategic work.
• Support Services driven by AI and DL will change the way you think about support with smart, groundbreaking technology backed by experts to help you maximize productivity, uptime, and convenience. Experience more than fast problem resolution—our AI engine proactively detects and prevents issues before they impact performance. Select ProSupport Plus for a single point of contact for hardware support.
• Payment Solutions from Dell Financial Services help you maximize your IT budget and get the technology you need today. Our portfolio includes traditional leasing and financing options, and advanced flexible consumption products to let you leverage OPEX instead of CAPEX if that suits your business requirements.
• Dell Technologies APEX offers a simple approach that gives you a wide range of consumption models, payment solutions, and services so you can optimize for various factors while realizing more predictable outcomes.
• Residency Services provide the expertise needed to drive effective IT transformation and keep IT infrastructure running at its peak. Resident experts work tirelessly to address challenges and requirements, with the ability to adjust as priorities shift.

In a world driven by data, the choice of infrastructure and platform for a modern data stack architecture is pivotal. Dell Technologies combined the power of Red Hat OpenShift, Starburst Enterprise, and cutting-edge technologies like Delta Lake and Apache Iceberg, to offer a transformative modern data stack solution. This stack empowers organizations to manage, analyze, and derive actionable insights from their data assets while ensuring reliability, flexibility, scalability, and security.

The Dell Validated Design for Analytics — Modern Data Stack delivers the power of a modern data stack with confidence to businesses. Dell Validated Designs are tested and proven configurations, designed from the start to dynamically fit the needs based on specific use cases. These integrated solutions help speed and simplify advanced analytics deployments.

Dell Technologies and the authors of this document welcome your feedback on the solution and the solution documentation. Contact the Dell Technologies Solutions team by email.

Authors: Dell Technologies Integrated Solutions Engineering, Technical Marketing, and Content Engineering and Translation teams

This document may contain language from third-party content that is not under Dell's control and is not consistent with Dell's current guidelines for Dell's own content. When such third-party content is updated by the relevant third parties, this document will be revised accordingly.

The following Dell Technologies documentation provides additional and relevant information. Access to these documents depends on your login credentials. If you do not have access to a document, contact your Dell Technologies representative.

• Dell Technologies Info Hub for Artificial Intelligence
• Dell Technologies Info Hub for Data Analytics

The following Red Hat documentation provides additional and relevant information.

• Red Hat documentation website

The following Delta Lake documentation provides additional and relevant information.

• Delta Lake Community Website

The following Starburst documentation provides additional and relevant information.

• Starburst Enterprise documentation website

The following Apache Iceberg documentation provides additional and relevant information.

• Apache Iceberg Documentation Website