A modern data stack is an open data management architecture that combines the best aspects of a data lake and a data warehouse in a single platform.

A traditional data lake tends to be flexible and cost-effective by storing data in its raw or natural form typically unstructured or semistructured. A data warehouse is a more advanced repository of data for reporting and analysis that tends to store data that is more structured. The data has typically been cleansed or operationalized for better data quality, often the result of extract-transform-load (ETL) or extract-load-transform (ELT) operations.

Data analytics usage is increasingly widespread and changing in nature. Those trends, coupled with the requirement to access large amounts of disparate data by many different users, require a modern, integrated approach to data access. A modern data stack combines the best of data lakes and data warehouses, supporting business intelligence and machine learning technologies in one platform. The platform can store all types of data and provide it with a cloud-like, multiresource, and self-service interface for data scientists and other users.

The Dell Validated Design for Analytics — Modern Data Stack has been developed to address the needs of organizations deploying advanced analytics. It incorporates the concepts of a modern data stack architecture together with a container platform using decoupled compute and storage.

This document provides design guidance for data analytics infrastructure managers and architects by describing a predesigned, validated, and scalable architecture for advanced analytics on Dell hardware infrastructure.

This document describes a validated design and a reference architecture for a modern data stack platform that combines Dell infrastructure, Red Hat® OpenShift® Container Platform, Starburst query engine, and open-source technologies: Delta Lake and Apache Iceberg. Together they address the needs of organizations deploying advanced analytics and AI workloads.

This design guide can be read alongside the associated white paper Dell Validated Design for Analytics – Modern Data Stack with Red Hat OpenShift Container Platform. The white paper discusses the benefits of a modern data stack compared to data lakes and data warehouses. It also provides a more general overview of a modern data stack concept and its components.

This document is intended for enterprises with data lakes or a data lake strategy interested in empowering their organizations to act more quickly, effectively, and efficiently on their data. Audience roles include:

• Data and application administrators
• Data engineers
• Data scientists
• Hadoop administrators
• IT decision-makers

A modern data stack can assist more traditional analytics customers looking to modernize their data collection. It can also help analytics systems to get more value from their data or standardize their data for modern analytics workloads.

This document presents the key concepts of the solution, namely the modern data stack architecture with open table formats, the container platform, and the use of decoupled compute and storage. It describes the solution architecture and components, including Dell infrastructure and node configurations that support the functions of the modern data stack. The Dell storage configuration options for the modern data stack are described, including PowerStore for container runtime storage, PowerScale for HDFS object storage, and ECS and ObjectScale for S3 object storage.

In addition to the modern data stack, the container platform architecture and components are described. The container platform in this validated design is Red Hat OpenShift – the industry-leading hybrid cloud application platform powered by containers and Kubernetes. Other industry-leading container platforms, such as SUSE Rancher or Symcloud, could also be options.

Going beyond the Dell hardware infrastructure and Red Hat OpenShift Container Platform, other software components that were validated include the Starburst query engine, Apache Spark, Apache Kafka, and open table formats like Delta Lake and Apache Iceberg.

Lastly, guidelines are presented for sizing and scaling the solution based on various workload requirements.

This document contains CLI command examples. Many PDF viewers add a line break to the end of each line of text in a PDF. Adobe Acrobat (Reader, Standard, and Pro) and other common PDF viewers, including the Google Chrome and Microsoft Edge web browsers, insert these line breaks.

As a result, when you copy commands that wrap across multiple lines in a PDF, the copied commands contain erroneous line breaks. If you copy and paste commands, the line breaks cause issues when you run the commands.

To address this known PDF limitation and ensure that copied commands do not contain unintentional line breaks, do one of the following:

• Paste the copied commands into a text editor, and remove the line breaks.
• Use the HTML version of this document when you copy commands.

The Dell Validated Design for Analytics — Modern Data Stack incorporates the concepts of a modern data stack architecture together with a container platform using decoupled compute and storage.

Modern data stack architecture gained popularity in the late 2010s as an evolution of the well-established data warehouse and data lake architectures. The architecture provides the most significant capabilities of both data warehouses and data lakes in a single system, reducing cost and complexity without compromising functionality.

There is no formal definition of modern data stack architecture. It primarily describes a system that combines the open file formats and cost-effective scalable storage of data lakes with the ACID transactions and table-oriented schema definitions of data warehouses.

Modern data stack architecture is often based on a modern table format such as Delta Lake or Apache Iceberg. This format provides a table abstraction above the underlying storage layer in the modern data stack. The other key features of a modern data stack are:

• ACID transaction support for inserts, updates, and deletes
• Scalable metadata
• Schema enforcement and evolution
• Support for diverse datatypes ranging from unstructured to structured data
• Data versioning or time travel capability
• Support for SQL access
• Support for direct table access through APIs such as DataFrames
• Support for scalable storage using open file formats

The Dell Validated Design for Analytics — Modern Data Stack is built on a container platform. This approach provides agility, flexibility, and scalability while supporting diverse analytics workloads.

The container platform abstracts the machine level details and host operating system dependencies, exposing them as a pool of compute, storage, and communications resources. The platform also provides application orchestration capabilities to streamline the deployment and management of analytics workloads.

The container platform used in this validated design is Red Hat OpenShift. The implementation details are described in Container platform implementation. Analytics workloads are packaged and distributed using standard Kubernetes methods including images, Helm charts, and operators.

This validated design for data analytics decouples compute and storage resources. This approach provides increased resource utilization, increased flexibility, and lower costs.

The system supports independent provisioning of storage and compute and enables the use of heterogenous storage and compute resources. This design provides a better balance between storage and compute for varying workloads.

Decoupling also simplifies the life cycle and management of the system by allowing independent management, scaling, and upgrades of storage and compute resources.

The design also separates runtime storage from data lake storage. PowerStore storage with the Dell Container Storage Interface (CSI) drivers provides runtime storage. PowerScale with the HDFS protocol, or ECS or ObjectScale with the S3 protocol, provides data lake storage.

Data has always been a vital part of every business. It is therefore valuable for businesses to be able to integrate, manage, access, and aggregate data from different sources regardless of their physical location or real-time structure.

Distributed databases and heterogeneous data stores can be viewed as a single database by virtue of data virtualization technologies. ETL on data is virtualized through data virtualization servers rather than physically performed with transformation engines. With this technology, users can access enterprise-wide data - including big data sources, traditional databases, cloud, and IoT - in a fraction of the time and cost required for physical warehousing.

In contrast to traditional data warehouses, data virtualization often does not require a comprehensive infrastructure since data are often unbroken in their data supply systems. Usually the cost of this approach is lower than for normal ETL, which requires that data be transformed into specific formats before being physically stored.

Dell Technologies has partnered with Starburst to offer its customers the ability to use data virtualization across multicloud environments. For data-driven organizations, Starburst's capability of querying data across any database makes the solution instantly actionable. With Starburst, you can run rapid analytic queries against data sources ranging from gigabytes (GBs) to petabytes (PBs) by using Trino, an open-source, high-performance distributed SQL engine.

The Dell Validated Design for Analytics — Modern Data Stack has been developed to address the needs of organizations deploying advanced analytics. It incorporates the concepts of a modern data stack architecture along with a container platform using decoupled compute and storage.

The architecture has been designed to support the requirements of modern data analytics. It provides a scalable, flexible platform for multiple analytics workloads, and supports the use of a modern data stack architecture. The figure below illustrates the overall modern data stack platform architecture.

Dell infrastructure

The infrastructure layer provides the core compute, storage, and network resources for the platform. For details about the recommended infrastructure, see Dell infrastructure. For additional sizing and scaling guidance, see Sizing the solution.

Red Hat OpenShift Container Platform

The OpenShift cloud-native platform, which is based on Kubernetes, is designed to enable developers to build, deploy, and manage applications across multiple environments consistently and securely. It provides developers with tools for building, deploying, and managing containers. Some of the key benefits of OpenShift include simplified application development, consistent deployment, management, security and compliance, and scalability and performance.

Applications and workloads

Applications and workloads are the components that provide the customer specified data analytics functions. For details about core applications used for modern data stack architecture, see Workload design. Design guides for other solutions running on this platform can be found on the Dell Info Hub for Data Analytics.

The architecture provides scalability of compute, storage, and networking at the infrastructure layer, scalability of the control plane and runtime layers, and scalability of application workloads. Storage and compute are decoupled to support independent scaling, contingent upon application requirements. The network layer scales with infrastructure growth.

The architecture provides the flexibility to run multiple workloads on a single platform with resource management and operations management. Workloads can be containerized cloud native applications or can run in virtual machines.

The architecture also implements high-availability features at all levels. Redundant networking is used for the core platform, Elastic Cloud Storage (ECS), PowerScale storage, and ObjectScale storage. Cloud native workloads that are deployed on Kubernetes are replicated automatically when a node fails to match the original replica set requirement.

Overall, the architecture enables data analytics on structured, semistructured, and unstructured data on a single platform. It gives developers, data scientists, and data engineers a single interface for deployment of advanced analytic pipelines. It provides administrators with a production-grade platform that is based on the de facto standard Kubernetes ecosystem.

Dell infrastructure provides the compute, memory, storage, and network resources for the platform. Modern data stack on Dell infrastructure illustrates the required infrastructure components and their roles in the cluster.

Control plane nodes

Cluster control plane components make global decisions such as scheduling, and detecting and responding to cluster events. Such events can include starting a new pod when replicas are unsatisfied in a deployment.

Worker nodes

These nodes are used to run containerized workloads. The Kubernetes control plane controls each node through the kubelet agent. The kubelet connects to the control plane and synchronizes the node state with the cluster.

Storage infrastructure

Storage infrastructure provides the modern data stack storage layer. Dell ObjectScale, PowerScale, and ECS configurations are provided. You can use any of them as modern data stack storage. The CSI driver for Dell PowerStore has been incorporated into Kubernetes to provide persistent storage.

Network infrastructure

Network infrastructure provides the required connectivity between the server and storage infrastructure and the on-premises network.

The architecture is flexible and supports a wide range of infrastructure configurations. Dell Technologies recommends the following configurations for a general-purpose modern data stack environment running analytics workloads. Individual node configurations include node-specific guidance. More guidance for cluster level sizing and scaling is provided in Sizing the solution. Design guides for individual solutions on the platform may include more specific recommendations for those solutions.

The server infrastructure provides compute, memory, and some of the storage resources that are required to run customer workloads. A wide variety of PowerEdge server configurations are possible. The recommendations here support a wide variety of workloads typical in a modern data stack architecture implementation.

Modern data stack control plane nodes support the core platform services that are defined in the Red Hat OpenShift Container Platform. Dell recommends the configuration in Modern data stack control plane node configuration as a starting point for these nodes.

Dell Technologies recommends the disk volume layout for this set of machines that are listed in Modern data stack control plane node volumes.

Three control plane nodes are required for production clusters to provide high availability for the control plane. For pilot testing, the control plane services that are defined in the Red Hat OpenShift Container Platform can be assigned to worker nodes instead of using dedicated control plane nodes.

Memory, storage, and processors have been sized to support all the required services in a production deployment.

The configuration includes two network ports to support high-availability (HA) networking. These ports can be from a single network card, or a pair of network cards for additional adapter level HA.

The operating system is installed on a 960 GB boot-optimized storage (BOSS) card.

Modern data stack worker nodes support the platform runtime services and customer workloads. The runtime services are defined in the Red Hat OpenShift Container Platform. User deployments define the workloads. Dell Technologies recommends the configuration in General-purpose modern data stack worker node configuration as a starting point for general-purpose workloads.

Dell Technologies recommends the disk volume layout for this set of machines that are listed in General-purpose modern data stack worker node volumes.

Volumes 1 and 2 provide node local storage, and can be used with the OpenShift Local Storage Operator to provide persistent storage through local volumes.

Dell Technologies recommends four worker nodes for a minimum deployment.

The configuration includes two network ports to support high-availability (HA) networking. These ports can be from a single network card, or a pair of network cards for additional adapter level HA.

Two M.2 960 GB SSDs in a RAID 1 configuration are used for the operating system volume.

The servers are not configured with a large amount of storage because the Dell CSI drivers provide runtime storage on PowerStore.

• See CSI Drivers for more information about Dell CSI drivers.
• See PowerStore infrastructure for more information about PowerStore storage.

System memory has been sized to support all required services and common workloads. If the anticipated workloads are larger, memory can be increased.

The processors have been chosen to support compute intensive AI and ML workloads and include dual Intel Advanced Vector Extensions (AVX) units for maximum compute speed. Other processor choices are possible but should be made with memory requirements and overall power consumption in mind.

Modern data stack storage can use the options that are shown in the following table:

You can use any combination of those systems depending on workload requirements.

Dell PowerStore provides application runtime storage. Dell Container Storage Module (CSM) is used for integrating PowerStore storage within the Red Hat OpenShift Container Platform environment.

Dell ObjectScale, ECS, and PowerScale are deployed as cluster-level systems. The node recommendations here can be used as guidance for new clusters, verification of compatibility with existing clusters, or expansion of existing clusters.

Dell Technologies recommends the configuration in PowerScale configuration for storage in clusters using PowerScale for their primary modern data stack storage using HDFS.

The recommended configuration is sized for typical usage as modern data stack HDFS storage.

Two Ethernet network ports per node are included for connection to the Cluster data network or a PowerScale storage network. Two additional network ports are included for connection to the PowerScale back-end network. These additional ports can be either InfiniBand QDR or 40 GbE, depending upon on-site preferences.

One PowerScale H7000 chassis supports four PowerScale H7000 nodes. This configuration provides approximately 720 TB of usable storage. At 85% utilization, 600 TB of HDFS storage is a good guideline for available storage per chassis.

This configuration assumes that the PowerScale nodes are primarily used for HDFS storage. If the PowerScale nodes are used for other storage applications or clusters, you must account for it in the overall cluster sizing. You can also use other PowerScale H7000 drive configurations.

Dell Technologies recommends the configurations in ECS EX500 node configuration or ECS EXF900 node configuration for storage in clusters using ECS for their primary modern data stack storage using the s3a:// protocol.

The ECS EX500 configuration provides a good balance of storage density and performance for modern data stack usage.

Two Ethernet network ports per node are included for connection to the Cluster data network or an ECS storage network. Two additional network ports are included for connection to the ECS back-end network.

The ECS EXF900 configuration is an all-flash configuration and provides the highest performance for modern data stack usage.

Two Ethernet network ports per node are included for connection to the Cluster data network or an ECS storage network. Two additional network ports are included for connection to the ECS back-end network.

Dell Technologies recommends the configuration in ObjectScale all flash configuration for storage clusters that use ObjectScale for primary modern data stack storage using the S3:// protocol.

ObjectScale is high-performance containerized object storage that is built for the toughest applications and workloads-Generative AI, analytics, and more. ObjectScale is the evolution of Dell ECS enterprise object storage architecture. ObjectScale provides increased flexibility in infrastructure choices.

Customers can use either of the following options to deploy ObjectScale:

• Software-defined storage to deploy ObjectScale software on Red Hat OpenShift
• An XF960 appliance that is based on the latest generation of Dell PowerEdge servers

This advanced system is designed as a distributed architecture, establishing a robust foundation for scalability across all nodes and ensuring unparalleled availability. The software component of ObjectScale offers an array of features that are engineered for optimal protection of data. It incorporates geo-replication capabilities contributing to enhanced data availability. Secure data access is another pivotal feature, ensuring stringent security measures are in place for data retrieval and modification.

The ObjectScale configuration is an all-flash configuration and provides the highest performance for modern data stack usage.

Two Ethernet network ports per node are included for connection to the Cluster data network or an ECS storage network.

Dell PowerStore uses a single architecture for block, file, and Virtual Volumes (vVols) storage. It uses the latest technologies to achieve these disparate objectives without sacrificing the cost-effective nature of midrange storage. Dell Technologies recommends the configuration in PowerStore configuration for containers in clusters for their runtime storage Persistent Volume Claim (PVC).

The Dell PowerStore 5200T is used for provisioning dynamic persistent volumes. The services to support this storage are deployed with the Dell Container Storage Module (CSM) operator available within the Red Hat Marketplace. This storage is exposed to workloads running on the cluster through the Kubernetes CSI interface. Workloads that require PVCs use this storage. PVCs enable users to consume abstract storage resources. Users often require Persistent Volumes with different properties, such as performance. In a modern data stack environment, most of the data is stored on external storage. The external storage can be either HDFS provided by Dell PowerScale, or object storage provided by ECS or ObjectScale.

The Dell Technologies-recommended configuration is sized for running various workloads which are hosted on Red Hat OpenShift Container Platform to provision persistent volumes dynamically using Persistent Volume Claims. The Dell PowerStore CSM driver supports the three access modes which are listed below. These access modes have different capabilities.

The access modes are:

• ReadWriteOnce—The volume can be mounted as read/write by a single pod.
• ReadOnlyMany—The volume can be mounted read-only by many pods.
• ReadWriteMany—The volume can be mounted as read/write by many pods.

See the Dell Technologies Container Storage Modules document PowerStore for more information about the Dell PowerStore Container Storage Module (CSM).

The network is designed to meet the needs of a high performance and scalable cluster, while providing redundancy and access to management capabilities. The architecture is a leaf and spine model that is based on Ethernet networking technologies. It uses PowerSwitch S5248F-ON switches for the leaves and PowerSwitch Z9432F-ON switches for the spine.

Physical networking in this architecture is straightforward since most of the advanced capabilities of the system are implemented using software defined networking. The logical network is described in Container platform implementation. This architecture has three physical networks, as shown in Physical network infrastructure.

iDRAC (or BMC) network

The iDRAC (or BMC) network is a secured and isolated network for switch and server hardware management, including access to the iDRAC9 module and Serial-over-LAN. This network optionally connects to corporate network management allowing more direct access to the hardware infrastructure. Each node on this network is assigned an individual IP address from the management address space.

Cluster data network

The Cluster data network is the primary network for internode communication between all server and storage nodes. Each server node on this network is assigned a single IP address on this network.

Core data center network

The Core data center network is the existing enterprise network. The Cluster data network is interfaced with this network through switching and routing allowing cluster services to be exposed to system users.

The management network uses a PowerSwitch S3148 1 GbE switch for iDRAC connectivity and chassis management.

The Cluster data network uses a scalable, resilient, nonblocking fabric with a leaf-spine design as shown in Cluster data network connections. Each node on this network is connected to two S5248F-ON leaf switches with 25 GbE network interfaces. The switches run Dell SmartFabric OS10. SmartFabric OS10 enables multilayered disaggregation of network functions that are layered on an open-source Linux-based operating system.

On the server side, the two network connections are bonded and have a single IP address assigned.

On the switch side, the network design employs a Virtual Link Trunking (VLT) connection between the two leaf switches.

VLT technology enables a server to uplink multiple physical trunks into more than one S5248F-ON switch by treating the uplinks as one logical trunk. In a VLT environment, connected pair of switches acts as a single switch to a connecting server while all paths are active. It is possible to achieve high throughput while still providing resiliency against hardware failures. VLT replaces Spanning Tree Protocol (STP)-based networks, providing both redundancy and full bandwidth utilization using multiple active paths.

The VLT configuration in this design uses four 100 GbE ports between each Top of Rack (ToR) switch. The remaining 100 GbE ports can be used for high-speed connectivity to spine switches, or directly to the data center core network infrastructure.

OpenShift Container Platform is an enterprise-grade declarative state machine that is designed to automate application workload operations based on the upstream Kubernetes project. In Kubernetes, declarative means that developers can code a configuration for an application or workload without knowing how that application is going to be deployed. OpenShift Container Platform is based on Kubernetes. It includes native support for cluster snapshots, which enable cluster backup and recovery. OpenShift Container Platform gives administrators and developers the tools that they require to deploy and manage applications and services at scale.

Red Hat OpenShift Container Platform, built on top of Kubernetes, adds useful features such as:

• Platform-level services, including building and packaging applications
• Preconfigured, preinstalled, and self-updating monitoring stack
• Integrated web console

Red Hat OpenShift Container Platform provides the following benefits for production-grade environments:

• It eliminates the complexity of installing Kubernetes and adding authentication, management, logging, security, and networking.
• It provides self-management capabilities that are not found in Kubernetes due to the tightly coupled toolchain:
  • The default containers-first operating system (Red Hat Enterprise Linux CoreOS)
  • A Kubernetes-first container run-time (CRI-O)
  • A rigorous testing and certification process for additional Red Hat and vendor middleware

The platform supports several different types of storage for ephemeral and persistent storage, and for modern data stack storage. The CSI Drivers by Dell implement an interface between CSI  (spec v1.5)-enabled Container Orchestrator (CO) and Dell storage arrays. It is a plug-in that is installed into Kubernetes to provide persistent storage using the Dell storage system. 

See Supported Storage Platforms for more information.

Stateful applications create a demand for persistent storage. OpenShift Container Platform uses the Kubernetes persistent volume (PV) framework to enable cluster administrators to provision persistent storage for a cluster. Developers can use persistent volume claims (PVCs) to request PV resources without having specific knowledge of the underlying storage infrastructure.

The Container Storage Interface (CSI) specification standardizes the exposure of block and file storage systems to workloads running on container orchestration systems such as Kubernetes. Kubernetes can now be readily extended to support any storage solution with CSI drivers that the vendor provides. Vendors can use an operator to manage the life cycle of their drivers directly, without waiting until the next core Kubernetes release.

Dell CSI drivers enable a wide range of storage options to be employed with the OpenShift Container Platform. For information about the available CSI drivers, including their features and capabilities for each of the supported Dell storage options, see Container Storage Interface (CSI) Drivers Series.

Dell CSMs are modules that enable simple and consistent integration and automation experiences, extending enterprise storage capabilities.

The following figure shows the Dell CSMs. The CSMs that are available on request as technical preview features, Encryption and App Mobility, are identified with an asterisk symbol.

CSMs expose enterprise features of storage arrays to Kubernetes so that the developers can use these features in their deployments efficiently and seamlessly. CSMs reduce the complexities of management, making it easier to consume enterprise storage. CSMs can automate daily operation tasks such as provisioning, snapshotting, replication, observability, authorization, application mobility, encryption, and resiliency. For more information and the support matrix, see Dell Technologies Container Storage Modules.

The platform supports multiple options for modern data stack storage using the protocols that are shown in the table below.

These storage systems are managed and scaled independently from the core platform, providing a decoupled storage and compute architecture. It is also possible to use more than one of these choices in any deployment.

Applications can connect to one or more modern data stack storage options directly over the network from the application-level code:

• The Apache Hadoop client libraries (hadoop-hdfs-client) provide the hdfs:// protocol.
• The Apache Hadoop AWS libraries (hadoop-aws) provide the s3a:// protocol.
• Other language libraries which implement the s3a:// or hdfs:// protocols.

Applications can access multiple modern data stack storage backends during:

• Heterogenous queries
• Scenarios like reading source data from one object system while loading it into a Delta Lake table on a different system

Iceberg and Delta Lake are limited to one backend storage system per table.

Depending on the application and its implementation, the images and application bundle may require that the necessary libraries be included. The application must handle authentication to any external storage.

All system administration and management are handled through a single web interface. This web console can be accessed using an optional Cluster System Admin Host (CSAH) deployed during Red Hat OpenShift Container Platform deployment. Operations can also be performed and automated through the command line.

You can view and manage your project data and perform administrative, management, and troubleshooting tasks through the OpenShift Container Platform web console. Within the openshift-console project, the web console is run as pods on the control plane nodes.

The platform also includes highly efficient monitoring and metrics collection of hardware components, pods, and applications. These features provide holistic observability of all platform activity in a single location.

See the Red Hat OpenShift Container Platform 4.12 on Intel-powered Dell Infrastructure Design Guide for more information.

The platform uses software-defined networking (SDN) to handle communications for the cluster. SDN includes container-to-container connections, pod-to-pod connections, ingress to pod connections, and pod to external services like ObjectScale, PowerScale, and ECS. These network services are provided through the Kubernetes Container Network Interface (CNI) and supported by CNI-compatible network plugins.

Each physical node in the cluster is assigned an IP address for its connection to the Cluster data network. Each connection uses a pair of physical network ports that are bonded with IEEE 802.3ad Dynamic Link Aggregation. This configuration provides both load balancing across physical links, and fault tolerance if a link fails. All networking above this layer is software-defined and uses IP addresses that are private to the cluster.

When applications running on the cluster request IP addresses, the addresses are allocated from the tenants IP pool. The IP pool specifies both the range of addresses available and the CNI driver to use. Dell Technologies recommends using the OVN-Kubernetes network plug-in for most use cases, since it provides the best support for inbound access to applications from outside the cluster.

Inbound network access to applications is handled through a Kubernetes NodePort or LoadBalancer service. The NodePort or LoadBalancer configuration is specified in the application configuration and includes the external and internal port number mapping. Outbound network access from applications is handled through IP routing that is based on the destination.

See the Red Hat OpenShift Container Platform 4.12 on Intel-powered Dell Infrastructure Design Guide for more details about OpenShift Container Platform networking.

Using Kubernetes as its backbone, OpenShift Container Platform provides an enterprise-grade declarative state machine for managing application workloads. Developers can specify, in code, a configuration for an application or workload without knowing how it can be deployed in a Kubernetes context. OpenShift Container Platform gives administrators and developers tools to deploy and manage applications and services at scale.

OpenShift Container Platform supports multiple application deployment strategies. However, this document uses only two of them:

• Red Hat Marketplace OperatorHub
• Building your own image or application

Red Hat Marketplace OperatorHub enables administrators to deploy certified Red Hat software from an operator-based catalog onto OpenShift Container Platform. OperatorHub is integrated into the OpenShift Container Platform Web console interface. Cluster administrators who choose to use Red Hat-certified operators are entitled to commercial support.

Catalogs available on Red Hat Marketplace OperatorHub are classified into the following categories:

Red Hat operators

Red Hat products that Red Hat packages and ships

Certified Operators

Products from leading independent software vendors

Red Hat Marketplace

Certified software packages that can be purchased directly from the Marketplace

Community Operators

Relevant representatives in the Red Hat ecosystem maintain products in this category without any support from Red Hat.

Custom Operators

Self-added custom operators

See the Red Hat OpenShift Container Platform document Understanding OperatorHub for more details about OperatorHub.

See the Red Hat OpenShift Container Platform document Configuring the Cluster Samples Operator for more details about building your own operator.

The concepts of containers and images are crucial when creating and managing containerized software. Containers are instances of container images running in real time, while images contain ready-to-run software. Containers in OpenShift Container Platform are based on OCI- or Docker-formatted container images. OpenShift Container Platform provides redundancy and horizontal scaling to a service packaged into an image by deploying the same image into containers across multiple hosts.

The podman or docker CLIs can also be used to build images. OpenShift Container Platform also provides builder images that help you create images by adding your code or configuration to existing images.

Once the image build process completes, you must push these images to the Image registry. A registry contains a collection of one or more image repositories, which contain one or more tagged images. OpenShift Container Platform can also provide its own OpenShift image registry for managing custom container images, similar to the Red Hat registry at registry.redhat.io.

Dell Technologies implemented and deployed workloads as part of the solution verification process. The following sections describe the implementations of three of these workloads.

Spark can run on Kubernetes managed clusters. With this feature, Spark uses the native Kubernetes Scheduler which is more robust than the stand-alone scheduler that is packaged with Spark by default. It can be further enhanced with better resource allocation, monitoring, and logging, making Kubernetes an ideal Spark companion.

The process can be as simple as supplying Spark applications to Kubernetes with the spark-submit command. As a result, resources are created inside a Kubernetes pod and orchestration activities are passed to the Kubernetes API server and Scheduler.

An alternative solution is to use the Operator to manage and monitor your Spark jobs in a declarative manner. You do not have to check job statuses or log files, or track job versions, as it streamlines the process for tracking jobs. ML experts can take advantage of the best of both worlds when Spark and Kubernetes are used together.

See the Red Hat OpenShift Container Platform blog Getting Started running Spark workloads on OpenShift for more information.

The Spark operator available on Red Hat Marketplace OperatorHub helps managing the life cycle of Spark applications on a Kubernetes cluster. However, the operator available on Red Hat OperatorHub at the time of this release provides older versions of Spark and other dependencies that are required to run the Delta Lake stack. Dell Technologies has created its own customized image with the following versions:

• Apache Spark 3.4.1
• Apache Hadoop client libraries 3.3.4 (for HDFS access)
• Apache Hadoop AWS libraries 3.3.4 (for S3 access)
• Delta Lake 2.4.0 and 3.0.0rc1 libraries
• Apache Iceberg 1.3.1

Several executors can be specified, along with their resource requirements, when running the spark-submit command. For example:

spark-submit \
  --packages io.delta:delta-core_2.12:3.0.0,org.apache.hadoop:/
    hadoop-aws:3.3.4,org.apache.hadoop:hadoop-common:3.3.4 \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql./
    delta.catalog.DeltaCatalog \
  --master k8s://https://your-kubernetes-api-server:port \
  --deploy-mode client \
  --conf spark.kubernetes.driver.request.cores=1 \
  --conf spark.kubernetes.executor.request.cores=1 \
  --conf spark.kubernetes.executor.limit.cores=2 \
  --conf spark.kubernetes.executor.request.memory=2g \
  --conf spark.kubernetes.executor.limit.memory=4g \
  --conf spark.executor.instances=4 \
  local:///your-spark-python-script.py 

In the example above Dell Technologies uses four Spark executors, each with one vCPU and 4 GB of memory.

Apache Kafka is a widely used technology for event and data streaming, which is at the core of Confluent Platform. Kafka is designed for building next-generation, event-driven applications. It provides organizations with a secure, enterprise-ready platform for real-time and historical event processing.

The Kafka on Kubernetes platform helps solve data pipeline scaling issues. It reduces the onboarding time of publishers and subscribers from several weeks to a few days. Kafka clusters can be scaled on demand on the OpenShift Container Platform using the HorizontalPodAutoscaler (HPA).

Confluent for Kubernetes is a Red Hat Certified OpenShift Operator available in the Red Hat Marketplace OperatorHub. This operator includes standard Kafka components, including:

• Kafka Broker
• Kafka Schema Registry
• Kafka REST proxy
• Confluent Control Center
• ZooKeeper (for runtime support)
• ksqlDB

The persistent volume storage class that is derived from Dell PowerStore storage is used to fulfill Kafka pods Persistent Volume Claim (PVC) requirements. The number of cores and memory per broker can be configured using:

• The OpenShift Web UI
• The Kafka broker configuration YAML file, provided in the wizard while deploying Kafka brokers

Using Confluent Kafka, Dell Technologies built a sample ETL pipeline that demonstrates how to stream live data into Kafka topics. It also demonstrates how to subscribe to a Spark instance that includes Delta Lake and Iceberg libraries. The modern data stack is configured on top of Dell PowerScale, ECS, or ObjectScale storage to write data into Delta Lake or Iceberg formats.

Starburst is a SQL-based Massively Parallel Processing (MPP) query engine for data lakes, data warehouses, and data meshes. Customers can address data silos and speed of access issues with enterprise and cloud solutions by Starburst. The Trino open-source query engine (formerly Presto) is the foundation of Starburst, which aggregates data from distributed data sources.Starburst provides access to over 50 enterprise data sources ranging from data lakes and warehouses to streaming systems, relational database systems, and more. Starburst Enterprise enables you to use different protocols for querying your different data sources, regardless of whether they are structured, semistructured, or unstructured.

See Starburst Connectors for more information about the list of available connectors.

The Dell Technologies and Starburst Multicloud Data Analytics solution uses the Starburst federated query engine to enable customers to:

• Run their business intelligence, analytics, and data science workloads.
• Access data spread across the enterprise.

This operator includes standard Starburst components:

• Starburst Hive
• Starburst Enterprise
  • Starburst Coordinator
  • Starburst Workers

See Multicloud Data Analytics with Dell Technologies Powered by Starburst for more information about the Dell Technologies and Starburst published referenced architecture.

Delta Sharing enables open, secure data sharing without any vendor dependence. Databricks builds Delta Sharing into its Unity Catalog data governance platform. This platform enables a Databricks user (data provider) to share data with a person or group outside their organization (data recipient). Delta Sharing supports many open data formats besides SQL, and it can scale to support big data. Several features are available to consumers with support for the Delta Lake storage structure. Organizations can collaborate with customers, establish new partnerships, and generate new revenue by sharing and consuming data from external sources.

See the Dell Technologies blog Dell and Databricks Announce Multicloud Analytics and AI Solution for more information about the Dell Technologies and Databricks partnership.

See the Dell Technologies white paper Power Multicloud Data Analytics and AI using Dell Object Storage and Databricks for more information about Dell object storage and Databricks.

Processing large amounts of data creates a significant amount of operational overhead after a while, especially when updates are made to existing data. This scenario is where a modern data stack can help, since it can process and upsert data incrementally. There are many features that Delta Lake and Iceberg provide on top of a data lake. One of the most powerful and useful features is to include relational database management system (RDBMS)-like features, such as ACID transactions, in a data lake. An existing data lake becomes a transactional data lake or modern data stack when any of these frameworks are added to it.

There are two common storage formats for big data workloads:

• Delta Lake storage format
• Iceberg storage format

Delta Lake is an open-source storage format, provides Atomicity, Consistency, Isolation, and Durability (ACID) transactions and scalable metadata management functionality. In addition to Parquet support for columnar storage, Delta Lake also offers Apache Avro support for data serialization. These features make it compatible with popular big data processing engines such as Apache Spark, Apache Hive, and Apache Presto.

Delta Lake offers several optimization features, including:

• DeltaLog
• Time Travel
• Schema Evolution
• Delta Cache

Iceberg is another open-source storage format that provides ACID transactions and schema evolution for big data workloads. Iceberg is designed to work with large datasets, and supports common big data engines such as Apache Spark, Apache Hive, and Apache Presto.

Iceberg offers several optimization features, including:

• Snapshot Isolation
• Time Travel
• Metadata Management
• Columnar Storage

The following sections detail the efforts that Dell Technologies undertook to ensure that the Dell Validated Design for Analytics — Modern Data Stack integrated hardware and software created a valid working design. Exhaustive testing of each hardware and software component was not performed. Dell Technologies verified that all the components work together as designed.

The following sections feature some of the tests that were performed and outline the goals and reasons for testing.

The software components and versions that are validated for the Dell Validated Design for Analytics — Modern Data Stack are listed in the tables below. The validated components may not precisely match the Dell Technologies-recommended configurations. For more information, email Analytics.Assist@dell.com, or contact your Dell Technologies sales representative.

The testing objective was to run sample checks to validate the functionality of different components and services that Dell Technologies installed as parts of the Dell Validated Design for Analytics — Modern Data Stack.

Dell Technologies tested the Delta Lake modern data stack that was deployed on the Red Hat OpenShift Container Platform. Dell Technologies also tested integrating Dell PowerScale, Dell ECS, and Dell ObjectScale software-defined storage with Delta Lake. This platform enables the running of various data analytics applications such as Spark, Kafka, and Starburst on OpenShift Container Platform with different storage systems.

Dell Technologies used Delta Lake 2.4.0 and 3.0.0rc1 with Spark 3.4.1 to validate all modern data stack functionality. All the features of Delta Lake with Spark were validated, including:

• Atomicity, Consistency, Isolation, and Durability (ACID) transactions
• Scalable metadata handling
• Updates, deletes, and merges
• Audit and time travel
• Schema enforcement and evolution
• Unified batch and streaming

These tests validate running a Spark bundle application on the OpenShift Container Platform using Delta Lake as its modern data stack storage. Dell Technologies used the Apache Spark 3.4.1 prebuilt Hadoop version with Delta Lake 2.4.0 and 3.0.0rc1.

Dell PowerScale, ECS, and ObjectScale acted as the modern data stack and Iceberg storage in order to access the data for read and write operations.

Data can be written to or read from Spark using different API protocols, such as:

• Hadoop (hdfs://<IP>)
• S3 object storage protocol (s3a://<bucket Name>/)

Dell Technologies has tested deploying Starburst Enterprise on top of Red Hat OpenShift Container Platform. This test validates the integration of PowerScale, ECS, and ObjectScale as modern data stack storage. The S3 protocol was leveraged across all storage platforms for reading and writing data. The deployed Starburst instance includes a Hive metastore service that is used for Iceberg metadata.

See Starburst Iceberg catalog and Starburst Delta Lake catalog for Starburst connector information.

The following is an example Starburst Iceberg catalog file.

iceberg: |- 

      connector.name=iceberg 

      iceberg.file-format=PARQUET 

      iceberg.unique-table-location=true 

      iceberg.hive-catalog-name=iceberg 

      iceberg.catalog.type=HIVE_METASTORE 

      iceberg.max-partitions-per-writer=6000 

      hive.metastore.uri=thrift://<mysql pod IP>:9083 

      hive.metastore-timeout=2400s 

      hive.metastore.username=hive 

      hive.s3.path-style-access=true 

      hive.s3.ssl.enabled=false 

      hive.s3.endpoint=http://<enpoint IP>:9021 

      hive.s3.aws-access-key=XXXXXXXXX 

      hive.s3.aws-secret-key=XXXXXXXXXXXX 

The figure below shows the Starburst Iceberg catalog in the Starburst Enterprise web UI.

The following is an example Starburst Delta Lake catalog:

delta: |- 

      connector.name=delta-lake 

      delta.hive-catalog-name=delta 

      delta.enable-non-concurrent-writes=true 

      hive.metastore.uri=thrift://<endpoint IP>:9083 

      hive.metastore-timeout=2400s 

      hive.metastore.username=hive 

      hive.s3.path-style-access=true 

      hive.s3.ssl.enabled=false 

      hive.s3.region=us-east-1 

      hive.s3.endpoint=http://<endpoint IP>:9021 

      hive.s3.aws-access-key=XXXXXXXXX 

      hive.s3.aws-secret-key=XXXXXXXXXX 

The figure below shows the Starburst Delta Lake catalog in the Starburst Enterprise web UI.

This test validates deploying Red Hat OpenShift Container Platform in high availability (HA) mode. For a highly available Kubernetes cluster, Dell Technologies recommends configuring a Kubernetes cluster with a minimum of three control plane nodes.

Red Hat OpenShift uses the keepalived and HAProxy services to provide high availability to the Kubernetes API server. The keepalived service is responsible for managing a Virtual IP address (VIP) where all requests to the Kubernetes API server are sent. The HAProxy service is responsible for redirecting API server requests to instances of the API server running on each of the control plane nodes.

Dell Technologies has proved the basic Red Hat OpenShift Container Platform cluster HA functionality by manually simulating failover of a Kubernetes control plane node. When the primary manager node fails or becomes unhealthy, one of the secondary manager nodes takes over as primary.

All the tests ran successfully. They show that it is possible to integrate powerful software solutions with Dell hardware to create value greater than the individual parts.

The Dell Validated Design for Analytics — Modern Data Stack is a valuable solution for any group seeking to analyze large amounts of data. The ease and flexibility of integrating various vendor products into the solution bring great value to the enterprise.

Cluster sizing and scaling are two different but related considerations. Sizing is concerned with ensuring the cluster meets the workload requirements for storage and processing throughput. Scaling is concerned with cluster growth over time as capacity needs increase.

The architecture is a parallel scale-out system with decoupled compute and storage. Some sizing requirements can be addressed through scaling while others must be addressed through node level sizing.

Sizing and scaling of a cluster are complex topics that require knowledge of the workloads. This section accentuates the main considerations that are involved but does not provide detailed recommendations for workload sizing. Design guides for specific workloads running on the platform include workload specific sizing guidance. Your Dell Technologies or authorized partner sales representative can help with detailed sizing calculations.

There are many parameters that are involved in cluster sizing. The primary parameters are:

Storage capacity

Storage capacity is usually the first parameter that is used to size a cluster. Calculating storage capacity is important and straightforward. However, storage capacity should be calculated while taking the other sizing parameters into account to maintain a balance between storage and processing capability. The use of decoupled storage and compute simplifies this consideration since the balance can be adjusted at any time during the life of the cluster.

Data volumes and growth rates

Data volume and growth rates each have multiple impacts on cluster sizing. data stack storage capacity should account for growth due to data ingestion and growth of ingest volumes over time. Data ingestion also impacts network utilization. Since the modern data stack storage is external to the cluster nodes, network bandwidth is required to access it. The processing throughput requirements must be considered as well as the data size.

Memory and processor capacity

Memory and processor requirements for jobs running on the cluster must be considered when sizing. Memory and processor capacity increases as nodes are added to the cluster. You can create heterogenous node configurations for workloads with specific requirements, which may be necessary.

Service-level agreements

Production cluster sizing must meet any performance requirements that SLAs specify. Critical path jobs that must meet a specific execution time or throughput may require adjusting the cluster sizing and balance between compute and storage accordingly. Overall cluster throughput is as important as storage capacity, and often influences the number of nodes independent of the required storage capacity.

Example cluster configurations lists some cluster level starting points for possible deployments.

The proof of concept (POC) configuration is a minimal configuration for basic evaluation. In this scenario, three worker nodes host the control plane, runtime services, and workloads. This configuration provides limited resources for workloads but is adequate for basic functionality evaluation. More worker nodes can be added to this configuration. For anything larger than five workers, Dell Technologies recommends starting with, or upgrading to, the pilot configuration.

You can engage the Dell Technologies Customer Solution Centers to host POC testing. See Dell Technologies Customer Solution Centers for more information.

The pilot configuration is a minimal production grade configuration. In this scenario three dedicated control plane nodes host the control manager roles, etcd, and the API server. Four worker nodes host the compute, along with any workloads. This configuration provides isolation between control plane and runtime functions. Dell Technologies recommends it for preproduction or development and test usage.

The pilot configuration can be scaled up by adding additional worker nodes to the existing cluster.

The small and medium business (SMB) configuration is a small production grade configuration. Three control plane nodes host the control manager roles, etcd, and the API server. Ten worker nodes are available for any workloads. This configuration provides enough resources to support one or two teams running analytics workloads.

The SMB configuration can be scaled up by adding additional worker nodes to the existing cluster.

The enterprise configuration is a large production grade configuration. Three control plane nodes host the control manager roles, etcd, and the API server. Twenty worker nodes are available for any workloads. This configuration provides substantial resources for running analytics workloads supporting multiple teams.

The amount of expected data primarily determines modern data stack storage sizing. This aspect of the sizing is independent of the compute cluster sizing.

The available network bandwidth between the compute and storage clusters must also be considered. Bandwidth on the storage and compute clusters scales in direct proportion to the number of nodes. However, dense storage capacity is possible with ECS, ObjectScale, and PowerScale. Such density can result in a large storage capacity without enough bandwidth to support the data transfer requirements. An analysis of workload data transfer requirements is necessary to correctly size the storage for both capacity and bandwidth.

The architecture is not limited to a single type of modern data stack storage. PowerScale can be used with the HDFS protocol, or ECS and ObjectScale can be used with the S3 protocol. Any workload can reference any or all these storage types. It is also possible to use multiple external PowerScale, ECS, and ObjectScale storage systems.

The network architecture allows both compute and storage clusters to use the same fabric. This configuration enables the network bandwidth to scale as either storage or worker nodes are added. The bandwidth available to the external storage systems should also be considered when referencing external storage that is not connected to the core Cluster data network.

The control plane node sizing that Dell Technologies recommends in Modern data stack control plane node is adequate for all production clusters and should not be changed. The controller manager, API, and etcd services must be deployed on three individual nodes.

The control plane services also consume a small quantity of worker node resources. Sizing in this design guide allocates 49 GB and four cores to these services. Approximately 320 GB of storage for the control plane is also required on worker nodes. This space is allocated from the boot drives and does not impact available server storage for user workloads.

All the available cluster resources across all worker nodes are pooled and allocated on demand. This configuration provides an abstraction where workloads can be mapped to available resources independent of the physical node used. Accelerators are considered a resource, and any workload pod that requires accelerator resources must run on a node with an available accelerator.

The recommended worker node sizes in this design are based on general-purpose usage. These worker nodes can support various analytics workloads without modification. However, there are scenarios where it is appropriate to change the configurations to match the intended workloads.

Heterogenous node configurations are possible. A cluster can include nodes with different memory, compute, and storage sizes. The resources from all these nodes are added to the overall resource pool.

From a resource point of view, there is little difference between many small nodes and a few large nodes. If the nodes have enough resources to handle the largest expected pod resource request, the difference between nodes is transparent. However, three additional considerations are involved in this tradeoff; network bandwidth, fault zones, and operational overhead.

Available network bandwidth is proportional to the number of nodes. A few large nodes have less bandwidth than many small nodes, even if the aggregate memory and compute resources are the same. The bandwidth requirements for workloads should be factored into cluster and node sizing.

Fault zones are important for the overall reliability of the infrastructure. Although the cluster can continue running when a node fails, resources from that node are lost on failure. Large node configurations in a small cluster can have a substantial impact on available resources when the node fails, even if it is a temporary failure. Sizing should ensure the loss of a node only impacts a small proportion of the overall cluster capacity.

Operational overhead is another consideration for sizing. Every node entails some operational overhead in terms of maintenance and monitoring, so larger nodes can be more efficient. One larger node can also be more energy-efficient than several smaller nodes. Operational capacity should be part of the overall sizing effort.

For parallel, scale-out workloads like Apache Spark and Starburst Enterprise, the resources are allocated based on availability at the cluster level and multiple workload pods that are launched. As a result, workload pods can run on any physical node that can meet the resource requirements. Depending on the Spark job workload, many small pods or a few large pods may be appropriate. The container platform runtime is flexible in this aspect. It is possible to deploy Spark clusters dynamically based on the job itself, instead of requiring a fixed Spark cluster optimized for many types of jobs.

Some workloads such as Starburst may have large memory requirements that cannot be achieved by scaling out. You may have to increase the memory size in some or all nodes to account for the largest expected memory allocation for that workload.

The platform can support up to 250 pods and 2000 worker nodes in a single OpenShift cluster. Cluster and node sizing should aim for substantially fewer pods than this limit.

The resources requirements for the intended cluster workloads must be factored into node and cluster sizing. Detailed sizing of workload requirements is complex. However, once workload requirements are known, the mapping into cluster requirements is straightforward. The flexibility of the platform also allows for ongoing adjustment and fine-tuning, so the sizing does not have to be exact.

Apache Spark is used here as an example of how workload sizing should be mapped to cluster requirements. Example Spark instance requirements summarizes the resource requirements for three sample Spark clusters.

In the table above, three clusters with varying resource requirements and scale have been included. The resources for each Spark worker have been specified, and the expected number of worker pods is included. Based on these requirements, the total resource requirements for each cluster are calculated.

The amount of net new modern data stack storage that is required is used in the calculation. If the jobs are expected to process existing data, no additional storage is required. If the jobs generate data, significant storage may be required. In the large instance, only 24 GB of modern data stack storage was estimated, while the medium instance requires 64 GB. The medium instance is expected to generate more data than the large instance even though it uses fewer compute resources.

Based on these calculations, the cluster level resources can be determined. For this example, the pilot cluster configuration can support four medium Spark clusters before it runs out of cores, or it can support two large instances.

During the lifetime of the system, it usually becomes necessary to scale the platform to support larger workloads or increase compute and storage capacity. The architecture is designed to scale at the compute, storage, and workload levels. The design incorporates network scaling as part of the infrastructure scaling. Compute and storage can be scaled independently.

Compute scaling is accomplished by adding additional nodes to the cluster. After physical installation and provisioning of the operating system, the Red Hat OpenShift Container Platform can be added from the OpenShift management interface or the Red Hat Hybrid Cloud Portal. The nodes become parts of the cluster. The control plane and network are automatically scaled as new nodes are added.

For more details of the node provisioning process, see Deployment options from the Dell Technologies document Red Hat OpenShift Container Platform 4.12 on Intel-powered Dell Infrastructure Implementation Guide.

Storage scaling is accomplished by adding or upgrading nodes in the ECS, ObjectScale, or PowerScale storage cluster, using the storage cluster management tools.

Persistent volumes storage can be scaled by adding PowerStore, PowerMax, or Unity storage arrays.

This architecture scales network bandwidth as compute or storage nodes, such as ObjectScale nodes, are added. When scaling either compute or storage, you must consider any balance-related network bandwidth effects since the bandwidth increases as nodes are added.

Substantial changes in expected data transfer volumes should be considered to ensure that the available bandwidth on the compute and storage clusters is aligned.

The Dell Validated Design for Analytics — Modern Data Stack has been developed to address the needs of organizations deploying advanced analytics, and AI and ML workloads. It incorporates the concepts of a modern data stack architecture along with a container platform using decoupled compute and storage.

This document provides design guidance for data analytics infrastructure managers and architects by describing a predesigned, validated, and scalable architecture for advanced analytics and machine learning on Dell hardware infrastructure. Topics that were discussed include:

• The cluster architecture that was designed for this application, including cluster server and storage infrastructure and its role in the system
• The cluster physical and logical network designs
• Details of the PowerEdge server; PowerScale, ECS, ObjectScale, and PowerStore storage; and PowerSwitch networking configurations
• The recommended software infrastructure components that were used in the architecture, including the Red Hat OpenShift Container Platform, Starburst Enterprise, and open table formats - Delta Lake and Apache Iceberg
• Examples of workload packaging, deployment, and validation, including Apache Spark, Apache Kafka, Starburst Enterprise, and Databricks Delta Sharing
• Cluster sizing and scaling guidance

Dell Technologies and the authors of this document welcome your feedback on the solution and the solution documentation. Contact the Dell Technologies Solutions team by email.

Authors: Dell Technologies Integrated Solutions Engineering, Technical Marketing, and Content Engineering and Translation teams

This document may contain language from third-party content that is not under Dell's control and is not consistent with Dell's current guidelines for Dell's own content. When such third-party content is updated by the relevant third parties, this document will be revised accordingly.

The following Dell Technologies documentation provides additional and relevant information. Access to these documents depends on your login credentials. If you do not have access to a document, contact your Dell Technologies sales representative.

Additional information can be obtained at the Dell Technologies Info Hub for Data Analytics. If you need additional services or implementation help, contact your Dell Technologies sales representative.

The following documentation on the Delta Lake documentation website provides additional and relevant information.

The following documentation on the Red Hat OpenShift documentation website provides additional and relevant information.

The following documentation on the Starburst documentation website provides additional and relevant information.

The following documentation on the Apache Iceberg documentation website provides additional and relevant information.

Our global network of dedicated Dell Technologies Customer Solution Centers are trusted environments where world class IT experts collaborate with customers and prospects to share best practices; facilitate in-depth discussions of effective business strategies using briefings, workshops, or Proofs of Concept (PoCs); and help business become more successful and competitive.

Dell Technologies Customer Solution Centers reduce the risks that are associated with new technology investments, and can help improve speed of implementation.

All of the services of the Customer Solution Centers are available to all Dell Technologies customers at no charge. Contact your account team today to submit an engagement request.

The Dell Technologies Info Hub is your one-stop destination for the latest information about Dell Solutions products. New material is frequently added, so browse often to keep up to date on the expanding Dell portfolio of cutting-edge products and solutions.

For more information, including sizing guidance, technical questions, or sales assistance, email Analytics.Assist@dell.com, or contact your Dell Technologies or authorized partner sales representative.