Application Modernization for High-Volume Workloads with DataStax
Executive summary
Summary
Introduction
Intended audience
We value your feedback
The promise of application modernization
Introduction
The challenges of distributed data
While a microservices architecture can help transform legacy applications into modern, scalable applications, another challenge of application modernization is designing a distributed database model that can span data centers, provide high availability, and deliver high-performance data processing capabilities.
Apache Hadoop, a scalable data-storage and analytics system, provides a model for storing and processing large, distributed datasets. The architecture of Apache Hadoop is distributed, meaning that data is distributed across multiple nodes. These clusters of Apache Hadoop nodes can be in a single data center, or they can span multiple data centers.
While Apache Hadoop paved the way for large-scale analytics across large, distributed datasets, it is better suited for batch processing than for real-time analysis of unstructured data. NoSQL databases, such as Apache Cassandra, fill the gap for real-time storage and analysis of high-velocity unstructured, semistructured, and structured data.
Like Apache Hadoop, NoSQL databases can scale horizontally to provide fast read/write access to various datatypes. Apache Hadoop and NoSQL databases differ in that Apache Hadoop is not intended to be a database-management system (DBMS); rather, it is designed to store and analyze massive amounts of unstructured data. NoSQL databases share the same ability as Apache Hadoop to store and analyze large, distributed datasets. Additionally, NoSQL databases are also designed to store and analyze unstructured, semistructured, and structured data.
The flexibility and variety of NoSQL databases and data models provide an answer to the inherent limitations of relational databases. Data models designed for relational databases have historically contained rigid schemas. While these rigid schemas have served enterprises well for decades, new data sources are pushing the limits of relational database designs. Data sources can now generate data with various structures that do not follow any specification, data model, or organization. Examples of unstructured data include raw sensor data from IoT devices and hand-written patient information created by medical professionals. Semistructured data typically contains tags or markers within the data that identify information, but the data does not comply with a specific relational database schema. Examples of semistructured data include Extensible Markup Language (XML) or JavaScript Object Notation (JSON) documents and email messages.
To avoid the limitations of relational databases, structured and unstructured data must be modeled differently when using a NoSQL database. NoSQL data models include:
- Key-value store—This data model is one of the simplest NoSQL data models. A key-value database stores data as a set of key-value pairs, where the key is a unique identifier. Data within a key-value store can be easily partitioned to allow horizontal scaling. For example, web applications can use key-value stores to manage session information. The key in this example identifies the user, while the values can be user information, user preferences, discounts, or promotions.
- Wide-column store—This data model is similar to relational databases in that data is stored in rows, with each row of data consisting of columns. In a relational database, each column is rigidly defined and maintained across each row of data. For example, in an industrial or manufacturing scenario, a row of data from an IoT refrigeration sensor might contain the sensor’s device ID, the time, the date, and the temperature. Wide-column stores differ in that each row of data can contain a different number of columns, different column names, and different column datatypes than other rows in the same table. Using the IoT refrigeration sensor example, a wide-column store might store the device ID, temperature, and humidity data from a warehouse refrigeration sensor on one row. The data from another device that monitors refrigeration on a truck might include device ID, temperature, humidity, latitude, longitude, and speed on the next row. The wide-column store can capture rows of information from both devices in the same table, even though the columns vary.
- Document store—A document store is designed to store documents that are typically formatted as JSON documents. Like other NoSQL data models, a document store does not adhere to a defined schema. Rather, the structure is internal to the document itself. For example, a health sciences research system can use a document store to consolidate patient data and doctor consultation notes from an electronic health records (EHR) system. Data scientist can then query these records to discover health trends within the consultation notes and correlate them with various patient attributes.
- Graph databases—This data model uses graph structures to represent relationships. Graph databases are ideal for capturing relationships between different data artifacts. For example, financial institutions can use graph databases to analyze the relationships between people, bank accounts, and transactions to discover fraudulent behavior.
Figure 1. Four NoSQL database models
Enterprises with unstructured, semistructured, and structured data can now rely on NoSQL databases to enable modern microservices-enabled applications that meet the needs of diverse organizations and customers with many different data models.
DataStax Enterprise overview
Introduction
DataStax Enterprise enables application modernization in finance and banking
Apache Cassandra architecture
DataStax Enterprise deployments rely on the node-based ring architecture of Apache Cassandra, which provides resilience, data integrity, and linear scalability.
In a typical Apache Cassandra deployment, each node consists of the Cassandra software running on a commodity physical server, such as a Dell EMC PowerEdge server with Intel Xeon processors. Apache Cassandra nodes can also run in virtual machines (VMs) or Docker containers based on Linux, either on premises or on cloud platforms. Each node runs the same Java-based Cassandra software and is configured through a single configuration file. Clusters can consist of any combination of on-premises physical servers or VMs and cloud resources.
Apache Cassandra clusters are organized into three logical categories:
- Nodes—A node is a physical server, VM, or container that runs an instance of Apache Cassandra.
- Racks—A rack is a set of Apache Cassandra nodes that are near to one another. For example, an on-premises deployment might contain multiple Apache Cassandra nodes in multiple racks within a single data center. In cloud deployments, a rack can refer to multiple instances of Apache Cassandra running in the same availability zone.
- Data centers—A data center is a collection of Apache Cassandra racks. In an on-premises deployment, these racks reside in the same data center and use the same data center network. In a cloud deployment, a data center typically refers to a cloud region.
Apache Cassandra organizes data using a ring topology. When data is written to a database, Apache Cassandra creates a token that determines in which nodes the data will reside. Apache Cassandra stores data redundantly across racks and data centers to ensure data availability if nodes become unavailable.
Figure 2. Overview of a DataStax Enterprise deployment across multiple data centers; clients can connect to any node within any cluster, while DataStax Enterprise OpsCenter manages all clusters and nodes
When a user or application queries an Apache Cassandra database through client software, the client connects to any node within the cluster. The node to which the client connects then becomes the coordinator node for that query. A coordinator node interacts with other nodes to collect query results, and then it returns the result to the client.
DataStax Enterprise cluster management
DataStax Kubernetes Operator for Apache Cassandra
Solution architecture
Introduction
Server components
The Dell EMC PowerEdge R640 server delivers the ideal balance of density and scalability, providing a scalable computing and storage platform with an ideal mix of performance, cost, and density for most data center and database applications.
The Dell EMC PowerEdge R640 server is a two-socket, 1U server platform designed for dense scale-out data center computing and storage. Users can benefit from the flexibility of 2.5-inch or 3.5-inch drives, the performance of NVMe, and embedded intelligence to ensure optimized application performance in a secure platform. With embedded diagnostics and SupportAssist, the Dell EMC PowerEdge R640 server delivers maximum uptime in a worry-free environment.
Figure 3. Dell EMC PowerEdge R640 server
The Dell EMC PowerEdge R640 server is built on a scalable system architecture and provides the choice and flexibility to easily meet performance demands and support the multiple configurations needed to optimize for the multiple Apache Cassandra database use cases defined in this project. The Dell EMC PowerEdge R640 server allows you to:
- Simplify deployment and easily scale from 3 to 1000+ nodes
- Maximize performance with up to 10 NVMe drives and 24 DIMMs, 12 of which can be DCPMMs or NVDIMMs
- Scale compute resources with 2nd Generation Intel Xeon Scalable processors, and tailor performance based on your unique workload requirements.
While the validation and performance testing for this project was performed on the Dell EMC PowerEdge R640 server, since the time of this study, the Dell EMC PowerEdge 650 server has been released. The Dell EMC PowerEdge R650 server is an enterprise server powered by 3rd Generation Intel Xeon Scalable processors designed to optimize workload performance and data center density. It is fully compatible with this application of DataStax Enterprise and delivers even higher performance than is documented in this white paper.
Infrastructure overview
Testing methodology
Performance results
Introduction
Red Hat OpenShift and VMware Tanzu configuration with SATA SSDs
Figure 3 displays the combined results of the DataStax Enterprise benchmark tests for the read-intensive, write-intensive, and balanced read/write scenarios on the Red Hat OpenShift cluster, while Figure 4 displays the same results for the VMware Tanzu cluster running on SATA SSDs. The x-axis displays the total throughput in operations per second performed during each test’s 10-minute run, while the y-axis displays the mean latency in milliseconds for each operation during the run.
Figure 3. SATA SSD benchmark results for the DataStax Enterprise read-intensive, balanced read/write, and write-intensive scenarios on the Red Hat OpenShift cluster
The Red Hat OpenShift cluster test consisted of five test runs for each read-intensive, write-intensive, and balanced read/write scenario. Each data point on the chart represents a test run with a specific number of client threads that simulate client loads. The number of client threads for each test run were:
- First test run: 40 client threads
- Second test run: 80 client threads
- Third test run: 160 client threads
- Fourth test run: 320 client threads
- Fifth test run: 640 client threads
Figure 3 shows that latency moderately increased for the 90 percent read/10 percent write scenario, while the latency dramatically increased during the mixed 50 percent read/50 percent write (balanced read/write) and 10 percent read/90 percent write scenarios as the number of client threads increased. The maximum operations per second occurred with the 90 percent read/10 percent write workload at 49,433 with 640 client threads.
Figure 4. SATA SSD benchmark results for the DataStax Enterprise read-intensive, balanced read/write, and write-intensive scenarios on the VMware Tanzu cluster
The VMware Tanzu cluster test consisted of five test runs for each read-intensive, write-intensive, and balanced read/write scenarios. As with the Red Hat OpenShift tests, each data point on the chart represents a test run with a specific number of client threads that simulate client loads. The number of client threads for each test run were:
- First test run: 18 client threads
- Second test run: 72 client threads
- Third test run: 144 client threads
- Fourth test run: 288 client threads
- Fifth test run: 576 client threads
Figure 4 shows latency moderately increased during the mixed 50 percent read/50 percent write (balanced read/write) and 10 percent read/90 write scenarios, while the latency increase was slightly higher during the 90 percent read/10 percent write scenario. The maximum operations per second occurred with the 10 percent read/90 percent write workload at 115,817 with 576 client threads.
Red Hat OpenShift and VMware Tanzu configuration with NVMe
Figure 5 displays the combined results of the DataStax Enterprise benchmark tests for the read-intensive, write-intensive, and balanced read/write scenarios on the Red Hat OpenShift cluster, while Figure 6 displays the same results for the VMware Tanzu cluster running on NVMe. The x-axis displays the total throughput in operations per second performed during each test’s 10-minute run, while the y-axis displays the mean latency in milliseconds for each operation during the run.
Figure 5. NVMe benchmark results for the DataStax Enterprise read-intensive, balanced read/write, and write-intensive scenarios on the Red Hat OpenShift cluster
The Red Hat OpenShift cluster test consisted of five test runs for each read-intensive, write-intensive, and balanced read/write scenarios. Each data point on the chart represents a test run with a specific number of client threads that simulate client loads. The number of client threads for each test run were:
- First test run: 20 client threads
- Second test run: 80 client threads
- Third test run: 160 client threads
- Fourth test run: 320 client threads
- Fifth test run: 640 client threads
Figure 5 shows that latency moderately increased for the 90 percent read/10 percent write, mixed 50 percent read/50 percent write (balanced read/write), and 10 percent read/90 percent write scenarios as the number of client threads increased. The maximum operations per second occurred with the 90 percent read/10 percent write workload at 201,797 with 640 client threads.
Figure 6. NVMe benchmark results for the DataStax Enterprise read-intensive, balanced read/write, and write-intensive scenarios on the VMware Tanzu cluster
The VMware Tanzu cluster test consisted of six test runs for each read-intensive, write-intensive, and balanced read/write scenarios. Each data point on the chart represents a test run with a specific number of client threads that simulate client loads. The number of client threads for each test run were:
- First test run: 10 client threads
- Second test run: 40 client threads
- Third test run: 80 client threads
- Fourth test run: 160 client threads
- Fifth test run: 320 client threads
- Sixth test run: 640 client threads
Figure 6 shows that latency significantly increased for the 90 percent read/10 percent write scenario at 320 client threads, while latency moderately increased for the mixed 50 percent read/50 percent write (balanced read/write) and 10 percent read/90 percent write scenarios as the number of client threads increased. The maximum operations per second occurred with the 90 percent read/10 percent write workload at 135,815 with 640 client threads.
Red Hat OpenShift configuration with Intel Optane persistent memory
Figure 7 shows the combined results of the DataStax Enterprise benchmark tests for the read-intensive, write-intensive, and balanced read/write scenarios on the Red Hat OpenShift cluster configured with Intel Optane PMem. The x-axis displays the total throughput in operations per second performed during each test’s 10-minute run, while the y-axis displays the mean latency in milliseconds for each operation during the run.
Figure 7. Benchmark results for the DataStax Enterprise read-intensive, balanced read/write, and write-intensive scenarios on the Red Hat OpenShift cluster configured with Intel Optane persistent memory
The Red Hat OpenShift cluster test consisted of six test runs for each read-intensive, write-intensive, and balanced read/write scenarios. Each data point on the chart represents a test run with a specific number of client threads that simulate client loads. The number of client threads for each test run were:
- First test run: 20 client threads
- Second test run: 80 client threads
- Third test run: 160 client threads
- Fourth test run: 320 client threads
- Fifth test run: 640 client threads
- Sixth test run: 960 client threads
Figure 7 shows that latency moderately increased for both the 90 percent read/10 percent write scenario and the mixed 50 percent read/50 percent write scenario. The latency showed a more dramatic increase in test four in the 10 percent read/90 write scenario. The maximum operations per second occurred with the 10 percent read/90 percent write workload at 203,099 with 960 client threads.