Data has become so important and valuable that it is driving the fourth industrial revolution. Elastic wants organizations to spend less time gathering data and more time focusing on using data. Once they can ingest multiple data streams, including textual, numerical, geospatial, structured, and unstructured data, businesses need to analyze, search, and distribute the data. Dell EMC VxRail hyperconverged infrastructure offers a foundational platform that can scale up or out to support your business requirements to make the best use of your data.

This white paper provides guidelines and best practices for the optimal deployment of the Elastic Stack on Dell EMC VxRail. This paper documents the design that was used to validate the Elastic Stack solution.

The Elastic Stack on Dell EMC VxRail offers a scalable advantage to traditional three-tier architecture. The results from the Elastic Stack performance benchmarking tests, executed using the benchmarking tool Rally, clearly show that VxRail can support the performance requirements of the Elastic Stack. Other key benefits of deploying the Elastic Stack on VxRail include:

• Flexibility of deployment options
• Indexing and searching performance
• Easily scalable solution to multiple Elasticsearch primary and data nodes

This white paper focuses on design considerations for deploying the Elastic Stack on VxRail. While there are other deployment options, this is an Elastic-validated solution design to show the scalability and performance as deployed. This paper also provides more information on best practices for deploying the Elastic Stack on VxRail and some performance benchmarking results from tests conducted using Rally.

This white paper is for VMware, Elastic Stack administrators, system engineers, partners, and Dell EMC or partner professional service community members.

The reader of this document should be familiar with the following technologies:

• Dell EMC VxRail
• Elastic Stack

The following table defines acronyms and terms that are throughout this document:

Table 1.              Terminology

Dell Technologies and the authors of this document welcome your feedback on the solution and the solution documentation. Contact the Dell Technologies Solutions team by email or provide your comments by completing our documentation survey.

Author: Vic Dery

Contributors: Reed Tucker

Note: For links to additional documentation for this solution, see the VxRail Info Hub.

VxRail is a Hyperconverged Infrastructure (HCI) platform jointly developed by Dell EMC and VMware. It is the only fully integrated, pre-configured, and tested HCI system optimized for VMware vSAN software-defined storage and the VMware vSphere ESXi hypervisor. VxRail is managed through the VMware vCenter Server interface, providing the familiar vSphere experience.

VxRail is the only HCI system that fully integrates Dell EMC PowerEdge Servers with VMware vSphere and vSAN. VxRail seamlessly integrates with and can fully leverage the existing VMware ecosystem and cloud management solutions.

For more information about VxRail, or about VMware vSphere and vSAN, refer to the References section.

VxRail HCI System Software is integrated software that delivers a seamless and automated operational experience. It offers 100 percent native integration of VxRail Manager and vCenter so administrators can perform familiar vCenter Server tasks. Intelligent life cycle management automates non-disruptive upgrades, updates, and node addition or retirement while keeping VxRail infrastructure in a continuously validated state to ensure that workloads are continuously up and running. VxRail HCI System Software includes:

• Software-as-a-service (SaaS) multicluster management and orchestration for centralized data collection 
• Analytics that use machine learning and AI to help customers keep their HCI stack operating at peak performance and ready for future workloads 

Figure 1.               HCI System Software

The Elastic Stack includes four open source products — Elasticsearch, Kibana, Beats, and Logstash — that provide data collection, transformation, ingestion, storage, search, analytics, correlation, and visualization capabilities. While the logical architecture described and tested in this white paper did not use Logstash and Beats, they are relevant to explaining how the Elastic Stack works. In addition to acting as a search engine, the Elastic Stack:

• Provides users with a centralized logging and monitoring platform for their IT environment
• Enables organizations to ingest and explore operational logs and metrics from their IT infrastructure 
• Provides security incident and event management (SIEM) 

Image Caption

 

Figure 2.              Elastic Stack components

The Elastic Stack includes numerous capabilities that help users easily build solutions focused in three areas:

• Enterprise Search: application search, site search, and workplace search
• Observability: logging, APM, and metrics
• Security: SIEM, endpoint, and security analytics

Value-added features such as machine learning, alerting, mapping services, Canvas, cross-cluster replication, cross-cluster search, field- and document-level security, and cluster authentication can be leveraged to meet the mission and business requirements of each organization. Many of these capabilities are free, and others can be obtained through an Elastic subscription that includes support. 

The Elastic Stack is designed to help users import data of any format and source and to search, analyze, and visualize imported data in real time.

The following figure shows the components available in the Elastic Stack:

• Beats are agents that ship the data from different systems.
• Logstash collects logs and events data. It parses, filters, and transforms the data.
• Elasticsearch stores, searches, and analyzes the data.
• Kibana is used to visualize the data.

Figure 3.       Elastic Stack architecture

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for fast searches, fine-tuned relevancy, and powerful analytics that scale with ease.

Elasticsearch is an open-source analytical search engine that allows you to store, search, and analyze large volumes of structured, semi structured, and unstructured data in near real time. Elasticsearch can be deployed on premises for specific environments. It is a distributed system, enabling high scalability and fault tolerance. Elasticsearch is a full-featured, full-text search engine with the capability to aggregate textual and numeric data in many ways.

Some common uses cases for Elasticsearch include website search for sites like Wikipedia and Yelp; IT observability such as mining logs and transaction data for trends, stats, or anomalies; and mission analytics to quickly investigate, analyze, visualize, and ask ad-hoc questions on huge datasets.

An Elasticsearch cluster is a collection of one or more nodes (servers) that together holds the entire data collection and provides federated indexing and search capabilities across all nodes. A node is a single server that is part of the cluster, stores the data, and participates in the cluster’s indexing and search capabilities.

An index is a collection of documents that have similar characteristics. A document is a basic unit of information that can be indexed. When an index is created, the number of shards can be defined. Each shard is a fully functional and independent "index" that can be hosted on any node in the cluster. Sharding is important for two primary reasons:

• It enables you to horizontally split or scale the content across a distributed cluster of nodes.
• It allows you to distribute and parallelize operations across shards (potentially on multiple nodes), thus increasing performance/throughput.

For more information about Elasticsearch, see Elastic Stack and Product Documentation.

Logstash is an open source data extract-transform-load (ETL) tool packaged with many plugins. Its role in the Elastic Stack is as a server-side generic data collector and processor. Logstash provides the capability to receive data from multiple inputs, apply a multitude of filters to the inputs, and output them to multiple types of sources simultaneously. Logstash operates in three stages for event aggregation and processing: collection, processing, and dispatching. The collection stage consists of inputs that generate events; processing consists of filters for modifying those events; and dispatching sends the events elsewhere as output.

Figure 4.              Logstash

Kibana is an open-source, browser-based UI that can be used to search, analyze, and visualize data stored in Elasticsearch indices. It contains several different analytic applications, such as Graph, Machine Learning, Metrics, and Logs, as well as configuration management and administration capabilities for the Elastic Stack. Kibana visualizations include histograms, line graphs, pie charts, sunbursts, and more. You can also design your own custom visualizations to suit your needs and all the visualizations leverage the capabilities of Elasticsearch.

A sample Kibana dashboard is shown here:

Figure 5.              Sample Kibana dashboard

Beats are open source, lightweight agents that collect data from system components. Beats can send collected data to different destinations, though Logstash and Elasticsearch are the most common. Along with the many Beats contributed by the community, Elastic provides and supports seven: Filebeat, Metricbeat, Packetbeat, Heartbeat, Auditbeat, Winlogbeat, and Functionbeat. Each Beat collects a specific set of data from the systems on which it is deployed. Beats also contain a “module” capability that allows for the automated collection, ingestion, and visualization of supported data types with a single command.

For example, Metricbeat is used to collect host metrics for memory usage, CPU usage, disk space, and network utilization, whereas Filebeat is used to send file data such as logs.

Figure 6.              Sample Metricbeat dashboard

We deployed this Elastic Stack solution on a five node, E560F, Dell EMC VxRail cluster. These VxRails are configured into a single VxRail vSphere cluster, in which each ESXi host contributes to the overall solution.

The diagram below shows the logical architecture of the Elastic Stack on Dell EMC VxRail. We configured the VxRail hyperconverged cluster using VxRail Manager, which automated the entire deployment process.

Figure 7.              Solution architecture

We created a single VxRail cluster with the associated vSAN datastore for Software-Defined Storage (SDS). We populated each host with two 1.92 TB SSD vSAN capacity drives and two 400 GB SSD vSAN cache drives. We used these disks to create a datastore storage pool to which VM virtual disks (VMDKs) were stored.

We deployed the Elastic cluster VMs onto the vSAN datastore. The Elastic cluster comprised five VMs of Elasticsearch, one on each VxRail.

For details regarding the configuration of the VxRail nodes, see Appendix B: Configuration details. 

The following figure describes the high-level network architecture of the VxRail cluster:

Figure 8.              Network architecture

We built this network using 2 x 10GbE connectivity. At no point during our testing did we encounter network saturation. VxRail can use 25 GbE connectivity when greater networking performance is required. See the VxRail Networking Guide in Dell EMC VxRail Documentation.

Table 2.              VxRail networking details

This section provides a detailed description of the methodology and configuration used to test the Elastic Stack hosted on VxRail with Rally's benchmarking tool. If we think about Rally's use, we would describe it as follows: Rally is the event or load generator; tracks are the races in this case data, and the http_log shows the final score of performance.

Rally

Rally is a benchmarking framework designed to measure the performance of Elasticsearch in very small and specific pieces.   It also can be used to perform the following tasks:

• Set up an Elasticsearch cluster for benchmarking, and tear it down afterward
• Manage benchmark data and specifications across Elasticsearch versions
• Run benchmarks and record results
• Find performance problems by attaching telemetry devices
• Compare performance results

For more information about Rally, see Getting Started with Rally.

Modifications made to the track JSON file:

We modified the Rally configuration as follows, as part of the setup:  

Applied a number to each IP conversion as suggested in the original readme file.

• Removed illegal characters from the "object_mappings.sort"
• Transformed the source data to a bulk-friendly JSON format (ignoring all entries that contained unrecognized/problematic characters and invalid IP addresses like "0".  Approximately 0.001% of the source data was lost due to this modification.)

Rally-Tracks

Tracks are repositories that contain the default track specifications for the Elasticsearch benchmarking for the Rally tool. Tracks are used to describe benchmarks for Rally and to provide a consistent and repeatable test.

Rally has created a tutorial on how to create your tracks.

For this paper the NYC Yellow cab data was used for the track.

This dataset includes trip records from all trips completed in yellow taxis from in NYC from January to June in 2015.  Records include fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts. The data used in the attached datasets were collected and provided to the NYC Taxi and Limousine Commission (TLC) by technology providers authorized under the Taxicab Passenger Enhancement Program (TPEP).  The TLC did not create the trip data, and TLC makes no representations as to the accuracy of these data.

The dataset can be found here:  

http://www.nyc.gov/html/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf[2]

HTTP Logs Track

At the end of each Rally race, Rally will produce metric, task and score values, so that an administrator can tune and maintain consistency. 

The final score result of the NYC yellow taxi cab Rally with the complete values are shown in Appendix A: Detailed results.

Test scenarios

We implemented the following test scenarios during our solution testing:

1. We first tested a single Elastic node with the benchmarking tool to reveal baseline numbers.
2. We then tested a five-node Elastic node cluster with the benchmarking tool to reveal performance results.

Rally runs on a separate host to benchmark the performance of the Elasticsearch cluster that is hosted on a VxRail cluster, as shown in the figure below. 

Figure 9.              Commands used for the  http_logs track

Use Case

Using Rally, we selected and ran the http_logs track. This track contains data gathered about the trips of Yellow company Taxis in New York in 2015. It can be downloaded from  https://data.cityofnewyork.us/Transportation/2015-Yellow-Taxi-Trip-Data/ba8s-jw6u

Some key results for track http_logs are represented in the following table:

For detailed results, see Appendix A: Detailed results

The following sections outline the best practices that Dell EMC engineers implemented in deploying this solution:

• In the tested configuration, we set the RAM amount for a virtual system to either 32 GB or 64 GB.  We recommend configuring up to 64 GB RAM for each virtual machine.
• In the tested configuration, we set the JVM Heap size to 50% of the RAM amount, with a maximum JVM Heap size of 30 GB. Keep at least 50% RAM available for other purposes.
• If Elasticsearch must scale vertically, then add more vCPUs.
• Start with a proof of concept, then test, optimize, iterate, and scale your solution.
• Elasticsearch recommends increasing the limit of File descriptors to 65,536.
• Disable the swap file.
• Depending on your SLAs, you may not need to see data refreshed each second. Increase the refresh interval to larger values depending on your use case and SLA to improve overall performance.
• Benchmarking VM with Rally should run on a different VM that is not part of the Elasticsearch cluster.
• For more information on best practices, see this Blog – Seven tips for better elasticsearch benchmarks
• Additionally, you may find more useful information at Important Elasticsearch configuration

• The recommended number of VxRail nodes should be based on a sizing exercise with Dell Technologies sales team or partner.
• All-Flash nodes are recommended for predictable performance.

• Follow the VxRail Network Planning Guide for best practices related to the VxRail network.
• Example: Separate management and control traffic from production application traffic. You can provide this separation by using VLANs.
• Separate VMware vSAN and vSphere vMotion traffic from production traffic according to the VxRail Network Planning Guide

To deploy Elasticsearch, we followed this link and selected the preferred OS version we wanted to deploy Deployment of the Elastic Stack supported on several different operating systems. For this testing and validation paper, we chose to deploy the Elastic Stack on CentOS 7 VMs on Dell EMC VxRail.

Our tested Elastic cluster deployment consisted of five Elastic Node VMs. We deployed each of the Elasticsearch clusters as a Virtual Machine with CentOS 7, one per VxRail ESXi host.

We installed Elasticsearch 7.9 by following the procedure in the following link https://https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html.

We installed Kibana to access the dashboards using the procedure in the following link https://www.elastic.co/guide/en/kibana/current/rpm.html.

The solution provided in this paper demonstrates how you can deploy the Elastic Stack on VxRail to meet performance, resiliency, and scale. It provides the best practices for deployment of the Elastic Stack on the VxRail system. The solution showcases how VxRail can address some of the key Elasticsearch application challenges.

Elastic customers are looking for performance and scalability, which are key features of the VxRail. VxRail provides the scalable compute capabilities required for the Elastic Stack with the utmost flexibility.

The following Dell Technologies documentation provides additional and relevant information. Access to these documents depends on your login credentials. If you do not have access to these documents, contact your Dell Technologies representative.

• Dell EMC VxRail
• VxRail Network Guide
• VxRail with SmartFabric Services Planning and Preparation Guide
• VxRail Planning Guide for vSAN Stretched Cluster
• VxRail vCenter Server Planning Guide
• VxRail Comprehensive Security by Design

The following VMware documentation provides additional and relevant information:

• VMware vSphere storage resources
• VMware vSAN technical resources
• VMware vSAN Caching Algorithms white paper
• VMware vSAN Space Efficient Technologies White Paper
• vSphere Virtual Machine Administration Guide
• VMware vSAN Licensing Guide

• Elasticsearch
• Rally
• Elasticsearch Tuning
• Deployment of Elastic Stack with CentOS 7:
• Elasticsearch Best Practice

Here are the results of our testing for this validation paper. This includes the Rally command used as well as the final scores via the http_logs track file.

Table 3.              Configuration details