There are many lessons that have been learned in the last 4 to 5 decades of using computers for data processing that have led to our current enthusiasm for the possibilities of advanced analytics.  Machine learning, deep learning, and artificial intelligence are beginning to solve significant business and societal challenges.  How far we can expand those achievements is impossible to predict.  The required interplay between data, computer hardware, software, and human creativity is complex.  Identifying the right combination of equipment, tools and knowledgeable practitioners that can produce good results with reasonable investments is even more challenging today than it was 40 or 50 years ago.  The vast number of technology options available today and the speed of change makes it difficult to construct a plan for a project that can deliver timely results.  It is easy to fall into the trap of waiting to make decisions until there is more research or the next promising technology reaches the next stage of maturity.

This eBook outlines a practical approach to the problem of tool selection and staffing for projects with machine learning and/or deep learning requirements using technology that is available today.  We will describe how to use a combination of open-source projects and commercial software including Spark, Spark SQL, MLlib,  BigDL, and Kubernetes/Docker.   We cover both issues of interest to both data scientists and IT pros.  We will also describe how to use Docker and Kubernetes to manage the infrastructure that can be integrated with software developers working on cloud-native applications.  We use a use case story centered on inventory management for a large fictitious retailer with both online and physical brick and mortar outlets to help weave the pieces of the technology puzzle together focusing on one stage of the analytics pipeline at a time.

Why Spark and Hadoop?

One of the main reasons for our continued interest in the Hadoop ecosystem for advanced analytics is the breadth and flexibility of the platform.  It is possible to combine a relatively small number of projects that handle everything from data ingestion to data cleaning, to model development and model hosting for inference.  An easy to couple and scale platform can become the rallying point for IT, data engineers, business analysts and data scientists to focus investments and share resources.  

The two main challenges when moving from data analytics on a single computer to a scale-out approach is how to distribute the computations along with a subset of the data to a coordinated and interconnected set of compute nodes, and, how to scale-out data storage to keep up with the I/O demands of multiple compute nodes.  The original two projects that made up the first version of Hadoop solved both problems in the mid to late 2000s.   The original distributed compute engine, MapReduce, was not capable of doing advanced analytics such as machine learning or deep learning, but that has been largely replaced and augmented with new distributed computing frameworks with Spark being the most important for data science.  Spark started with a concept familiar to everyone that works in data science, the data frame, and devised an approach to distribute it across many computers taking advantage of the combined memory and computing cores in a way that didn’t force data scientists to change the way they traditionally work with data.  We’ll have a lot more to say about Spark, Spark data frames and distributed computing in later sections of this eBook.

Why Kubernetes

As we noted in the previous section, modern data science had to expand beyond the computing power of a single machine to escape the memory and computing core limits imposed by the physical design of widely available, low cost, x86 computers.  Managing distributed computing systems with dozens, hundreds, or thousands of machines to handle the scaling needs of data scientists working with increasing volumes of data presents new challenges for the architects and IT operations professionals that support these large data science initiatives.  This is where Kubernetes is making a significant contribution.

The relationship between Spark and Kubernetes is conceptually simple.  Data scientists want to run many Spark processes distributed across multiple computers to have access to more memory and computing cores.   If we use container virtualization like Docker to host those Spark processes, Kubernetes will orchestrate the creation, placement, and life cycle management of those multiple processes across a cluster of low-cost x86 servers.  There are many Kubernetes implementations available today and so, we will have a lot more to say in later chapters about how we recommend using Kubernetes with Spark and the trade-offs that should be considered for your needs.

A Use Case Story

We still find the analytics “pipeline” is a useful metaphor for how data engineers and data scientist work through data ingestion, data cleaning and transformation, data merging, model training and testing, and finally model hosting for inference.  For this eBook, we choose to develop a story for explaining both the challenges and most successful approaches for each of these steps in the analytic pipeline.  That forced us to select a data set that had enough volume and complexity to highlight some real-world challenges and solutions without incurring the cost and lead time of solving an actual enterprise-class problem.  We choose to build a story describing a machine learning model-based approach to inventory management for a very large retailer with hundreds of thousands of individual stock units (SKUs).  We will show how Spark together with a set of related big data technologies can provide an end-to-end solution for solving real-world data analytics problems by developing the individual pieces using a simplified data set and requirements.

Overview

The terms data analytics, data analysis, and data analyst have been in use for many decades and therefore have very broad definitions that everyone will not always agree on.   For the purpose of this eBook we use data analytics to mean the process of extracting information from raw data either visually, via summaries or using a data modeling process like machine learning.   This chapter is going to describe some of the challenges and methods used for processing large sources of raw data (big data) through inspection, cleansing, transforming, modeling, and publishing results for external consumption by other processes.   Every one of these components of data analysis has been written about extensively, but we believe that there is an additional value that can be added by discussing methodologies for developing complete pipelines for data processing in the context of interesting use cases.  As the title reveals, we will be using Apache Spark as the primary technology for implementing an end-to-end pipeline for one use case in later chapters, but in this chapter, we will describe some important aspects of data analytics and working with data in general terms.  This background will help us describe what Spark can and cannot do during the use case discussion.   We will not cover any topics in the areas of decision science, information theory, organizational dynamics or business metrics/KPIs related to the processes for incorporating the information derived from data analytics into business decision making.

Ever since humans decided to collect data starting with scratch marks on cave walls, other humans have tried to develop tools to make the task easier.   Our experience has shown that the collection, storing, and analysis data along with the communication of insights to others can all be improved with the right tools.   This work has turned into a huge industry. The field of statistics was born from the need to have better ways to characterize and summarize data than what could be accomplished with other types of applied mathematics.  Later, machine learning and deep learning were developed as the need for tools to analyze larger datasets with more sophisticated relationships emerged.   Innovation in the development of the first analog and now digital computing hardware has also been intimately connected to the drive for more capable tools for doing data analysis.  

The volume of data available in the age of digital transformation has created some of the most challenging problems that IT professionals face today.  At the same time, we have been so successful in the race to produce better and better tools for data analytics capable of working with these large data volumes of data that we have created another problem – an almost overwhelming set of software and hardware choices that is difficult to evaluate and choose from.  How does a data analytics team comprised of IT professionals, data engineers, and data scientists agree on a combination of hardware and software that meets everyone’s needs for ease of use, scalability and future resiliency?  

The needs of data scientists today cover an extremely broad spectrum of techniques and tools that often require specialized skills and may produce many different types of outcomes.   IT needs to be flexible but also cost-conscious and therefore can’t implement everything that might be in demand.   In a field with this much diversity, organizations will want to adopt one or more classification schemes to organize the discussion of use cases and techniques that can aid in decision making.   There are many relevant dimensions and therefore many parallel and overlapping grouping schemes exist.   The table below shows just a few of the more commonly discussed classifications of types of analytics and data.

In the following sections of this chapter, we will discuss some less common and slightly more technical ways to classify data along with the implications for the types of specialized analytics that are required to do advanced analytics with complex data.

Types of Data Analytics

Interactive Analytics

The ability to speak with confidence about the validity of data analytics results begins with building trust in the source data.  Virtually every entry-level course in statistics and data science begins with a section on data visualization focused on how to use interactive visual techniques to learn about the interesting aspects of a data set prior to doing modeling.  You can think about interactive analytics in the same way you might approach preparing a vehicle for a long trip.  A walk around the vehicle might prompt you to look more closely at a small crack in the windshield or the need to check the air pressure in a tire that looks low.  The process follows a trajectory of asking a question, look at the result, evaluate what questions that answer suggests.

The term interactive analytics usually refers to an interaction between a human and a dataset that is too large and/or complex to understand using simple descriptive statistics.   Interactive analytics is also often referred to as interactive visualization reflecting the reality that it almost always involves producing and evaluating graphical summaries of the data.   The goal is to use a combination of “best practices” and ad hoc approaches together in an iterative process of getting to “know” your data where each result typically generates more questions than answers in the beginning.  

A researcher beginning to work with a new dataset that includes physical characteristics of people may want to know the proportion of people with blonde hair and the proportion of those with blue eyes in the sample.   If these results seem reasonable, she may then compute the proportion of people with both blonde hair and blue eyes to uncover more information that will confirm that the sample is representative of the population in general.

The order and types of questions that are explored during an interactive analysis session mainly derive from the results of previous questions.  The data scientist starts by exploring the aspects of data they already have some expectations to see if there are signs of observable relationships that confirm those expectations.  The output from these queries together with domain knowledge prompt the data scientist to look at other summaries or details in the data.  The cycle continues and the data scientist begins to either build trust in the quality of the data or begins to form one more hypothesis about why the data may not be of good quality, e.g., the data are too problematic to be trusted for model building.

The efficiency of interactive analytics depends on many factors including:

1. The experience, skill, intuition and domain knowledge of the researcher
2. The initial quality of the source data
3. The power and elegance of the tools and platforms used by the researcher.

The data analytics industry is very concerned about the amount of time analysts and data scientists spend on preparing/munging data prior to applying algorithms and training models.   No one likes to write code for processing filthy data or document decisions regarding what data was kept, discarded or modified in order to even get a model to train.  However, time spent “getting to know the data” should not also be targeted as an unproductive use of time that should be eliminated.  Especially in the era where data science generalists develop specialties in fields like language processing or computer vision and travel between subject domains, there needs to be time budgeted for exploratory data analysis that relies heavily on tools that support human interaction with the data.

Batch Analytics

Data analytics is about extracting information from data.  Data generated in the last 5 minutes will contain some information but typically data collected over the last day will have richer information.   If you want to study the December buying patterns of retail consumers, you will need several to many seasons of data to assess if any trends are discoverable.  For these types of research questions, you will need to store data and perform batch data analytics and therefore, we do not believe that batch analytics is outdated and soon to be replaced by a sole focus on “real-time” analytics.  We need to continue developing improvements for all types of analytics including batch analytics.

All machine learning and deep learning model training have historically been batch processing oriented.   If you constantly add new data to the training set during the training session it would be impossible to tell what went wrong if the model failed to converge.  Also, there would be no reproducibility.  Once a model has been trained and validated, adding additional near-real-time observations to the training set is feasible for some use cases.

When working with historical data batches one of the most difficult decisions is deciding the best time frame to use for a training set.  Many relationships that we want to study change over time, especially when humans are a factor.  For example, studying the factors that lead consumers to purchase electric vehicles has changed over the last 50 years as more convenient options have been developed and attitudes regarding energy use and the environment have changed.  Training the model on 25 or 30 years of historical purchase and demographic data might be biased by the old relationships in the data.  But should the researcher use just the last 2-3 years of data or perhaps 10.  Exploring options for choosing the best sample of data together with what type of model along with the specific hyperparameters is largely an intuitive and time-consuming process.  More research into improvements in batch analytics is underway but certainly not getting the same level of attention as streaming analytics.

Time Series Analytics

Although most people would have a very difficult time defining what a time series is, they are relatively easy to identify once you see an example.  A simple definition of a time series is:

a collection of data points with a one-to-one association to discrete points in time where the time points are most commonly equal in length.  

While even this simple formal definition of a time series may seem overly complex, see how the examples below are easier to understand and explain than a formal definition.  Here is a series of outdoor air temperature readings in degrees Fahrenheit taken every 5 minutes:

In this example, the time interval between observations is equal in length (5 minutes) and we have one data point (number), a temperature, that corresponds to each time interval.   This is consistent with the definition of a time series from above.  What the definition does not include is how the single measurement was calculated for the 5-minute interval.  It could be the maximum, minimum, average, beginning or ending value.  Details like this should be captured in the metadata (data about data) for the time series and stored with the source data since it may be important to the design of our analysis and/or interpretation of the results.

If you have some familiarity with machine learning and/or data models you probably think of data as either features (inputs or explanatory variables) or outcomes (a number or probability that you want to forecast).  In modeling a time series, we have one feature, time, and the outcome is the data associated with each time interval.  In the example above the outcome is outdoor air temperature and the explanatory feature is time.

Time series analytics research began long before computers were invented.  Most of this research has been conducted by statisticians.   Very recently machine learning tools including regression trees and fully connected shallow neural networks are being used for the analysis of time-series data, especially when there are other available features that can be used to explain some other related influences such as geography or demographics for example.

While most time series involve numeric data types, we can also have a time series of discrete data represented as character strings or a discrete set of integers.  There are entire textbooks (for example, An Introduction to Discrete-Valued Time Series ) devoted just to the analysis of times series of discrete data values.  It pays to keep both types of time series in mind when exploring a new dataset and be aware that the techniques for analysis are vastly different.  Below is an example of data tracking the status of a traffic signal every minute for a period of 10 minutes.

We used a set of unique character strings (RED, YELLOW, GREEN) but we could have also used integers (1,2,3).  The difference between this example and the one above is that temperature can be expressed along with a continuous number range with almost limitless possibilities depending on how precisely we can measure where the time series of discrete values have a fixed number of values.  The traffic light example above can produce exactly 1 of 3 possible values for any single measurement.  

Time series data may be the largest source of raw data generated in the world today.  It is impossible to say how much storage would be required to collect all the time-series data that is available in the world since only a small fraction of it is ever accessed and/or analyzed.  The potential to capture and analyze time-series data for a multitude of uses cases is an area that every data scientist should be considering.  The laptop or another computer that you are using to read this document is generating thousands of data points every second related to performance characteristics, environmental factors, and security.  Every imaginable aspect of a disk drive, network, graphics card, and central processing unit (CPU) metrics are being monitored by the system and made available to system administrators for viewing and analysis if they choose to do so.  Most systems are configured to write a small amount of historical of data to the local drive that constantly gets overwritten by new data, therefore keeping the drive from filling up with diagnostic data that would eventually halt operations.  This is just one example that anyone with a computer can explore for themselves.  There are billions of devices that are both capable of generating operational time series data and have network connections, but the vast majority are either not connected to a network or are only streaming a very small subset of the data series they can produce.

Virtually every mid-sized and larger commercial building in the world has a building energy management system that monitors and stores detailed temperature, humidity, airflow, lighting status, and energy consumption details for all major environmental components at regular time intervals varying from minutes to seconds.   Every modern piece of industrial equipment including pumps, fans, boilers, furnaces, conveyor belts, and robotics to name just a few can produce thousands of data points every second and made accessible by a remote network data connection just like we discussed above for computers.  Imagine the amount of equipment, processes, and data that could be accessed from a hospital network, a transportation system, a smart home or a smart city.  And this list only scratches the surface of the total sources of time series data available in the digital world.  Many of the sources of data listed above are grouped together under the concept of the Internet of Things (IoT) umbrella but there are many other scientific and engineering applications that are either just called operational data or research data.

Event Data Analytics

Everyone can probably relate to the frustration experienced when your computer “crashes”.  In information technology support circles this is referred to as an “event”.  Other examples of events that might be familiar to most people are your airplane is late leaving the gate, you have an automobile accident or perhaps you win the lottery.  If one of the primary characteristics of time series is the regularity of the time between data samples, then as you can see from the examples above, the time interval between events is typically difficult to determine.  In fact, the challenge of predicting when the next event of interest is going to occur is what makes data analytics with event data so challenging and interesting.

Most of the event processing pipelines you will encounter start with descriptive analytics, e.g, counts, sums, and simple statistics like average, minimum and maximum values summarized over regular time intervals and possibly grouped by related features like geography or some population characteristic.  This type of “pre-processing” can turn event data with irregular intervals into time series of numbers with equal intervals that are easier to visualize and see trends.   Predictive analytics for event data involves modeling and therefore is probabilistic.   A common type of research question is “what is the probability that event A will occur in population B during the time interval T.  For example, what is the probability of a computer crash happening for either Windows or Linux users in the next 6 months.  Or, what is the probability of a person aged 40-45 being diagnosed with skin cancer in Florida compared to New York?  The list of possible research questions related to event data modeling is limitless.

You can also frame event data analytics in terms of numerical forecasting or regression.   There is obviously a relationship between event counts, probability of occurrence, population size, and time intervals.   Both machine learning and deep learning can be used for virtually any type of use case and research question formulation, however, special care needs to be exercised with event data when the set of possible outcomes is extremely imbalanced.  Credit card fraud seems pervasive these days but the ratio of non-fraudulent to fraudulent transactions is still very high given how often people use credit and debit cards. Data analytics techniques employing “oversampling” and other techniques can help improve the understanding of the relationships between the input features and “rare” of imbalanced outcomes but require advanced training to select the right techniques and for interpretation of results.

Another interesting modeling opportunity with events in the area of prescriptive analytics to answer questions about what actions an actor takes to avoid or trigger an event occurrence.   What factors would encourage a competitor’s customer to leave their existing mobile phone service contract and switch to another service.  What actions would extend the expected duration before the next failure event for a piece of equipment?  Answers to these questions based on data modeling are also described in terms of uncertainty and probabilities that decision-makers need to be comfortable evaluating.

The data associated with an event may be as simple as a single number (0,1) or string of text (“it happened again”) but could also be a complex multi-part data object.  Computer crashes often generate metadata describing the time, user name, version of various software and drivers as well as files storing part or all the data that was in memory when the event was detected.   Part of the challenge when working with event data is anticipating how much data each event will generate and what is needed for analysis.  A philosophy of “keep everything” provides future flexibility but may limit how much history can be affordably retained.

Streaming Data Analytics

We most often think of streaming data as a near-continuous set of values coming from a single source, however, we can also have “event streams” from many sources each of which has irregular and potentially long intervals between values but appear at the target as a steady high-volume stream. The most salient characteristics of a streaming data analytics application are that data arrives in units with very low latency between items and is processed as soon as possible, often while still in the memory of the receiving system and before hardening to disk. Very high-velocity data may be temporarily stored in a local queue and then sent for processing in “mini-batches” to cut down on communication overhead but would most likely be considered a data streaming analysis so long as the latency between generation and processing was low.

Engineers have been interested in streaming data for centuries.  In 1874, French engineers built a system of weather and snow-depth sensors on Mont Blanc that transmitted near real-time information to Paris and so began the “modern” era of data streaming.  Extreme weather conditions can come and go rapidly on and around Mont Blanc.  Public safety concerns and research interests were important factors in the decision to undertake the expense of establishing the capability to stream data to a remote and safer monitoring site.  More recently, the hype suggesting that real-time analytics is replacing batch analytics is growing all most as fast as the capability of technology for streaming data.  System architects should not give in to over-engineering every application by adding the cost of designing and engineering a streaming data solution without careful consideration of the expected value of a lower data latency on the use case being considered.  

Data streaming technology and streaming data analytics technology can either be packaged together in a single platform or chosen separately and integrated by a developer.   The last 10 years of technology development in the “big data” and Hadoop ecosystems have produced a number of advances for designing streaming data applications.  The open-source projects for Flink, Spark Streaming, and Kafka give system architects many options to choose from but increase the risk associated with developing on a platform that may not have enough momentum in such a crowded field.  

Kafka began primarily to address message queuing technology that later evolved into a platform with both data streaming and analytics capability.  Even the primary message queuing technology in Kafka can be configured with different size message payloads and the processes that read new messages from a topic can be scheduled to meet a wide range of processing latency that can still meet the requirements of many streaming applications.  In some uses cases, we see Kafka satisfying both the data management and analytics requirements of a streaming analytics application wherein other use cases specialized analytics such as Druid, Hive or even custom software are used with Kafka.

The variety and complexity of uses cases that can benefit from streaming data analytics are staggering.   At the simplest end of the continuum would be a filter for a single numeric or string value.  For example, if the temperature of this pump motor is more than 150 degrees Fahrenheit – sound an alarm.  This type of low complexity application can typically be programmed on or close to the piece of equipment and require very little computing power or memory.   Moving toward the complex end of the spectrum are uses cases that require multiple adjacent data points from the stream, often referred to as windowing the data.  Going back to our pump example, if the temperature rises in 3 consecutive readings from a normal of 100 degrees to 150 degrees – shut the pump off.  However, if the motor temperature is oscillating up and down between 100 to 150 degrees over 10 or more values – log an incident report.  Moving further up the complexity continuum would be an application that needs data from multiple streams available in a common window.  The streams may come from sources far away physically or on disconnected networks and so the data needs to be routed to a commonly accessible device and would require much more complex analytics programming.  

These are only a few of the many opportunities and issues that organizations encounter when working with streaming data and stream analytics.  In general, the adage “push analytics to the edge” should be followed when possible.  Any efficiency that can be realized by data filtering, aggregation, or conversion of continuous streams to a set of less frequent events is worth exploring.  However, statements such as “all analytics are moving to the edge” are hyperbole and only add to the existing confusion caused by the rapid expansion of technology options for data analytics and data streaming.

Working with Multiple Large Data Sets

Much has been written about the four (4) Vs of big data – Volume, Velocity, Variety, and Veracity, however,  you could have a data pipeline that processes high volumes of data generated frequently consisting of numbers, text, audio, and images and not have to deal with much complexity if all that data is from a single source with a consistent format.  While the four Vs are all important characteristics of data there is a complexity factor introduced when handling multiple sources of data that will affect the choice of tools and platforms for data analytics.  

When information about a single object like a physical facility is assembled from multiple sources there must be a key that uniquely matches a single entity in each of the sources.   Managing keys, joining data based on keys, and handling key errors are just a few of the many significant challenges encountered with doing multi-source data analytics.  We have been working on this problem for decades in the context of enterprise data warehousing and still the number of difficult to join data sets may be getting worse.  Application modernization and the transformation to agile methods for cloud-native applications cannot address the challenges of data integration unless the requirements are documented and implemented early in the design.  

The resource impacts of joining large datasets from multiple sources are also both well documented and challenging.  How the data are keyed, sorted, and coded never seems to be optimal in the source systems.   The overall impact is multiplied by the many steps in an analytics pipeline that are frequently devoted to data lookups, sorting and writing out of intermediate results.  Platforms capable of performing these large-scale data integration jobs typically require significant manual tuning that relies on trial and error to find a good set of hyperparameters for any given job.

The realization that timelines for gaining business benefits from data analytics are being heavily impacted by the effort that needs to be expended on the acquisition, transformation, persistence, exploration and cleaning of data is moving the focus from tools for model development to tools for data management.  The front end of the analytics pipeline is time-consuming and expensive.

The most frequently cited estimate is that data analytics professionals spend on average 80% of their time/resources on these non-model development activities.  When you have requirements that include the need to integrate large data sets from multiple sources you will almost certainly encounter significant complexity and be at the top end of the percentage range spent prior to model development.

These effort estimates are fueling a growing misconception among non-practitioners that skills specialization and division of labor can lead to lower project costs and shorter time to value.  The argument is that data engineers should manage the pre-processing of data and data scientists should just build models with the cleanly structured and sanitized data. Our experience shows that data modelers/data scientists need to be intimately connected to all the stages of data ingestion and transformation process.   We believe that data science is a team sport and there are many opportunities to involve people with different skills and levels of experience in the most labor-intensive aspects of a project, however, managers should not try to separate the people building models from the data pipeline that is feeding the models in an effort to save money.   Data science teams should be composed of data engineers, data analysts, data scientists, subject matter experts and project management specialist working in lockstep at each stage of the analytics pipeline for every non-trivial project.

There is also some interesting development of tools that automate some aspects of data pre-processing pipeline.   It is still very early days for seeing significant benefits from intelligent data joining, exception handling and coding of input data but the rate of investment in this category of tools is accelerating.  While there are clear patterns of data processing that can benefit from automation, it is our opinion that when dealing with multiple large datasets the benefits of generic automation are going to be marginal for some time to come.

Summary

When approaching the design of a solution for a new analytics use case, start with an understanding of what information could improve the business process or inform a decision-maker.   Then realize that there is not now, nor ever likely to be a single tool or technique that will do everything to meet those requirements.  The other extreme of piecing together a broad collection of specialized open source and commercial software will most likely lead to worse results than trying to force all problems into one tool.  Most organizations have long fought against both tool and data platform silo proliferation vs the anticipated benefits of introducing “best of breed” data analytics technologies.  Investments in tools that require proprietary data formats and hardware are especially limiting and difficult to break free from once implemented.  Perhaps the best way to characterize the challenge is to ask - what is the minimum number of tools and platforms that will meet all the requirements of the organization and create the fewest islands and silos.

The chapter covered some types of data analytics and data you may encounter in analyzing use cases.   There are of course specialized systems for doing all the above types of analytics.  In the remaining chapters, we will show how the Spark platform and related tools can be used to implement a complete analytics pipeline that covers many of the topics introduced above based on the requirements of a specific use case.   Our solutions engineering team built a lab environment to test interesting features of the Spark platform and toolset.   We wrap-up with some conclusions, lessons learned and resources for continuing to learn about and use Spark.

What is Spark?

Spark is commonly referred to as an “open-source, general-purpose, distributed computing platform”.   We will break that sentence down in the rest of this chapter starting with a discussion of why this class of platforms exists.  There are a handful of application development sub-systems that are so broadly applicable that virtually every software developer will use at least one of them during their career.  However, building platforms that can take advantage of multi-threaded processors, memory caching and/or distributed processing using a cluster of computers is beyond the skill set of many professional software developers let alone most data scientists or data engineers.  This need for saleable computing has fortunately led to the development of many open-source and commercial offerings.  Spark is one example of an open-source scalable distributed computing platform.

The most successful of the scalable platforms are both robust and flexible enough to be used by millions of application developers that otherwise would not be capable of building scalable foundations for critical supporting services on their own.   Another characteristic that you will see in all successful platforms is that application developers will use them in creative ways not anticipated by the core developers. This can be either a benefit or a liability to the organizations that adopt them.   

Some examples of platforms that have become a core part of so many applications are worth mentioning before we talk about Spark.   Relational database management systems and distributed message systems meet all the criteria mentioned above of sub-systems used by millions of application developers that would be difficult or impossible for most software professionals to recreate for their own dedicated use.  Developing multi-user, ACID compliant, transaction processing systems with online backup capabilities and reliable crash recovery is only achievable by the best-trained computer scientists with years of experience.  Of the major offerings available only Oracle has developed a truly distributed offering for transaction processing (Real Application Clusters) however the successful development of scale-up systems using advances in multi-core CPUs and memory technology have has made these platforms an essential component of the success of nearly every organization we see.  If every one of these organizations needed to find and hire resources with the necessary skills to build an RDBMS, we would not have the widespread success associated with the last 40+ years of digital transformation. (Oracle introduced in 1979, IBM DB2 in 1983, and Microsoft SQL Server in 1989 to name a few).  Millions of application developers have built scalable and resilient applications using these RDBMS platforms that they otherwise would not have been able to do alone.

There are also many software applications that require the advantages that come from using scale-out message processing.  Although the applicability is perhaps not as general as ACID-compliant transaction processing, the software industry still has produced several successful message processing systems including IBM MQ, Microsoft Message Queue (MSMQ) and the widely used Apache open-source Kafka project.  While these messaging systems have different approaches and applicable use cases, the important point is that it was more efficient for the software industry to have these platforms developed by a small number of exceptionally talented platform developers rather than having hundreds of thousands of application developers each trying to write their own foundational platform for messaging.

Spark is a relative newcomer to the list of highly scalable distributed computing platforms that were first available from the Apache Software Foundation around 2010.  Spark is robust and has the flexibility to be called a “general purpose” distributed computing platform. The breadth of Spark use cases can be seen from the large number of organizations that publish papers and speak at conferences including making movie recommendations at Netflix, performing DNA sequencing, analyzing particle physics at CERN, and protecting financial payments at MasterCard and PayPal.  The remainder of this chapter is focused on providing enough details of the architecture, advantages, and limitations of Spark at the beginning or intermediate level.   Our intent is to provide background so that data professionals can follow our use of Spark for data transformation and model training as described later in this eBook.  You will see how our engineers used Spark to efficiently execute those tasks on a cluster of machines without having to code the distributed computing details directly.  Spark allows data professionals to achieve a faster time to solution for many enterprise-class problems including data processing and data science.

History

During the incubation of Google, the founders realized that in order to revolutionize the efficiency and relevance of web search, they were going to need to develop new computing tools to do it.  The number of URLs that existed on the Internet in the early 2000s plus the complexity of analyzing the relationships of inter-page linking meant that Google needed both a new scale-out file system and a new scale-out computing platform.   The first publicly available descriptions of how those two challenges might be solved were published as white papers in 2003-2004.  Researchers at Yahoo that developed the first versions of the Hadoop Distributed File System (HDFS) and the MapReduce computing platform credit those early Google papers for the architecture foundations that started the Hadoop open-source initiative.

The rapid accumulation of data that came with digital transformation created an ever-growing need for distributed computing and storage platforms.  At the same time, interest and acceptance of open-source software were on the rise.  The rapid expansion in the development and use of Apache Hadoop ecosystem projects including HDFS and MapReduce demonstrated the applicability of distributed compute and storage for many areas beyond the web search interests of Google and Yahoo.   It is a testament to the original architects that well-designed generalized tools can produce many benefits beyond the narrow application requirements that initially prompted the research.

As the volume and veracity of the data that organizations wanted to explore grew, MapReduce began to show signs of severe limitations.  The name reveals the first limitation, the architecture is limited to just two operation types: mapping and reducing.  Mapping was defined as the process of transforming a file with one format into a new file with a different format.  The most common example used when teaching developers about MapReduce is a word count map.  Given a file of text: transform that unstructured format into a two-column list of all the words in the original document in the first column and a count of the number of times they appear in the second column.  

If there are multiple files in the source or if you wanted to split several large files into smaller chunks to have more parallelism in processing, then you might also add a reduce step to the map task described above.  For example, a simple reduce task would be to combine the union of all the results from the map into a single “master list” of word counts, i.e., input a lot of files and reduce those to a single output file with the summary results.

The MapReduce design limited the scope of a job to, at most, one map task and one reduce task.  The analyst could split the tasks and have one job perform the just the map before writing those results to disk.  Then another job could read the output of the map task from disk, perform the reduce step, and then end with writing the single summary output file out to disk.  Breaking the work into two jobs with all the extra reads and writes might seem inefficient but there may be an operational justification for doing it.  

However, combing jobs to cut down on reads and writes was not possible with MapReduce.  If you wanted to do another step to filter the summary to eliminate any common words plus any in all capital letters you can’t string that together with the first map and reduce.  There can only be one of each at most that come with the overhead of the initial reads from disk and writes to disk.

These limitations led researchers at the UC Berkeley APMLab to review the internals of MapReduce to see if they could improve the efficiency of distrusted computing compatible with the Hadoop ecosystem.  The result of that work was the design for a new parallel processing framework – Spark.

Architecture

Entire books are available on the Spark architecture and internals.  Our goal is to provide enough summary material to understand the remainder of this eBook for those without much background.  For those more familiar with Spark, we hope this treatment gives you some new insights and possibly adds some new approaches to how you explain Spark to others.  The generally accepted way to describe a generic computer object that is common to many data science disciplines is with the two word form - data frame.  Spark dataframes are always written as a single word in the documentation.  We try to consistently follow this convention for this paper. 

Spark Core

Many problem domains that need an approach for efficiently describing and processing a sequence of events have used a tool from mathematical graph theory called a directed acyclic graph (DAG).  The MapReduce architecture for defining jobs can be described with a very simple (almost trivial) DAG.  

The designers of Spark wanted to expand support for more complex DAGs while simultaneously isolating application developers from having to define the DAG before submitting a job to Spark.  The new approach would define a larger set of common tasks like union(), randomSample(), sortBy(), filter(), etc. that could be strung together to perform a wide range of useful jobs or programs.  Application developers could describe what they wanted Spark to do with the data and the Spark core runtime would construct an efficient DAG that it could execute to produce the desired result.  Developers working with MapReduce were responsible for building and sharing reuseable building blocks of code to execute in either a map or reduce task.   Spark provides a rich set of prebuilt functions grouped into transformations and actions available through an API.  This built-in shared library makes Spark development available to a wider range of developers and speeds delivery of results for experienced developers.

The other limitation of MapReduce that was addressed in the Spark core design was removing the requirement to read from and write to disk after each 1 or 2 step job (map, reduce, or map and reduce).  Spark DAGs could execute an arbitrary number of transformations or actions in one job that was determined the requirements of the application developer.  The preferred data location during multi-stage job processing is memory.  Spark will manage moving data between memory and disk when there is a need, however, some subsets of data that Spark cannot split between memory and disk may still cause out-of-memory errors.  

Spark core defines an abstraction for managing and working with distributed data called a resilient distributed dataset (RDD).  RDDs can be pinned in memory for efficient execution of multistep transformation pipelines, can recover from many types of node failures when distributed across multiple nodes in the cluster without requiring a complete restart for a job, and can manage spooling or spilling data to disk when necessary under some conditions. Together the more flexible job composition capabilities, DAG processing, and RDD functionality addressed many of the limitations of MapReduce discussed above.

Data Structures

You may encounter the term “rectangular data” when reading about data engineering and data science.   Understanding the appeal of Spark starts with understanding the near universal adoption of a common type of rectangular data structure in the data science community – data frames.   A data frame is a 2-dimensional data structure with labeled columns.  Each column must be formatted for a single data type that might include real numbers, character strings, integers, Boolean or logical values, etc.  Every cell value in the column must match the data type for that column.  These requirements place constraints on how a data scientist constructs a data frame, but it also ensures that they are compatible with many types of statistical and data manipulation libraries.

For Python developers, the data frame object and functions implemented in the Pandas library are used in virtually every data science project.  Data scientists that work with R have data frames implemented as a data type in the base libraries.  The Java open source community has developed several data frame libraries that have features applicable to data science including data importing, filtering, aggregation and columns creation but vary considerably in ease of programming.

Data frames are by far the most important type of rectangular data for data science work but there are other common types of rectangular data in the world including relational database tables, comma separated variable (csv) files, and spreadsheets to name a few.  Comparing data frames to spreadsheets can help conceptually, but in practice, data scientists generally avoid spreadsheets for data work in favor of a data frame library supported for their development environment.   Most machine learning and some deep learning algorithms used for model training are designed to accept data frames as inputs.  Even in those instances where the final step of a data pipeline does not accept a data frame, data scientists will often work with a data frame to import, sort, filter and augment the data using a data frame object and then convert to another data structure as late as possible in the pipeline.

The main limitation of data frames is in most cases memory.  Many implementations of data frames that we will discuss shortly are limited to the availability of memory on a single computer.  Many implementations are also either restricted to a single processor core on a single computer.  While many laptops today can have 16GB and more of memory and can, therefore, hold large data frames in memory, sorting, filtering and selecting operations can become painfully slow with data frames over 1-2GB.  Spark let’s data scientists create distributed data frames that can spread across the memory of many computers and use multiple cores of each machine in the Spark cluster using the same types of operations that are used with Python, R and Java data frame libraries and data types.

Resilient Distributed Dataset RDD

Traditional databases that support asynchronous, fine-grained updates to shared state typically use centralized update logging and data checkpointing to provide fault tolerance.  That approach has not scaled well for large data transformations.  RDDs, on the other hand, are particularly well suited to batch processing applications that apply a set of sequential transformations to large datasets and still provide efficient fault tolerance.

Programmers can specify that the rows of an RDD be partitioned across multiple machines in a cluster based on a key in each record.  Partitioning aids both performance as well as fault tolerance. One the design goals for RRDs was that only the RDD partitions on a failed node would need to be recomputed without having to roll back the whole program. Another goal was that the failed partitions could be recomputed in parallel on different nodes.  

In order to achieve these goals, Spark needs to track only the definitions for the transformations (DAG) that would need to be replayed to change an RDD partition from an initial state to the final state.  This frees Spark from having to log the large amounts of data need for fault tolerance to support fine-grained updates.  There are some properties of RDDs that are necessary to support job fault tolerance:

1. RDDs can only be created by loading data from stable storage (reading from disk) or through the transformation of other RDDs.  
2. Once instantiated, RDDs are immutable.  
3. These properties are necessary to achieve the scale and fault tolerance benefits of Spark.

RDDs are one of the key Spark developments that isolate application programmers from the details of distributed computing.  Programmers see RDDs as two-dimensional data structures with typed columns that can scale larger than the memory of any typical workstation or single server. From the Spark internal perspective, an RDD is a distributed memory abstraction with fault tolerance that allows programmers to perform in-memory computations using one or more nodes of a computer cluster.  The in-memory nature of RDDs has significant performance benefits over MapReduce for iterative algorithms and interactive data science.

Dataframes

Spark dataframes can seem indistinguishable from Python and R data frames however there are very important differences.  Spark dataframes are a higher-level abstraction of a two-dimensional row and column data structure that is implemented on top of Spark Core RDDs.  The primary motivation for this additional lever of abstraction was to provide a more familiar experience to developers that use Structured Query Language (SQL) for data transformation and analysis.   Providing a SQL interface to Spark data also allowed the Spark developers to implement an optimizer that would interpret SQL code and emit an efficient physical plan for performing work on the cluster.

Spark dataframes are accessible from Scala, Java, Python, and R.  This broad support for languages is possible even though Python and R  lack of compile-time type-safety because dataframes do not impose type safety.  This design choice for Spark dataframes is considered a significant downside, especially in large environments with lots of users.  A considerable amount of cluster resources can be wasted when jobs fail due to analysis errors that are difficult to avoid for large datasets with lots of columns.

Datasets

Spark datasets are a newer option that is available only to Scala and Java developers since the API is typed.  Datasets share many of the benefits of RDDs including typing and use of looping with inline code expressions (lambda functions) plus the performance gains from the Spark optimizer for SQL.

Many developers feel that Spark datasets will be the preferred path forward although they are still relatively new. Python and R developers do not have an option for using datasets today and so will continue to generate demand for new features in Spark dataframes.  Conversion between RDDs, dataframes, and datasets is possible if your programming language supports the implementation but can be costly for large amounts of data.

Libraries

Spark SQL

The introduction of the Spark SQL library made Spark distributed computing accessible to the very large population of developers that work with Structured Query Language (SQL).   Spark SQL allows developers to write common SQL expressions that get passed through an optimizer that produces an “execution plan” to perform the work on the cluster and return the results as either a Spark dataframe or dataset.

Since Spark SQL generates the low-level instructions to access the underlying core RDDs, all Spark SQL jobs have the advantages of fault tolerance and scale-out partitioned performance of RDDs.   Spark SQL can also be used to directly query other data sources including Hive, Avro, Parquet, JSON, and JDBC including joins across multiple sources.

Our use case described later in this eBook uses Spark SQL for all data access and transformations.

MLlib

Spark was designed to improve the performance of tasks that require or benefit from iterative computation and parallelism.  Many machine learning algorithms use iteration with an optimization technique like stochastic gradient descent.   MLlib contains many algorithms for regression, classification, clustering, topic modeling, and frequent itemset analysis that leverage iteration and can yield better results than the single-pass approximations.  There is also a wide range of decision tree models including an extreme boosted gradient version XGBoost that we use later in our lab work.

Our use case described later in this eBook uses MLlib for machine learning.

Spark Streaming

Developers working with Spark Streaming access a high-level abstraction called discretized streams (Dstream) to represent a continuous stream of data.  The physical implementation of a Dstream in Spark is based on a series of RDDs that hold a “window” of data.  Therefore, operations on Dstreams can be distributed across partitions of the RDDs leveraging the parallel execution engine of Spark core to achieve both high performance and fault tolerance.

Spark Streaming has native support for file systems and socket connections built-in.  Advanced sources like Kafka and Flume require interfacing with external non-Spark libraries with complex dependencies.

There are Scala, Java and Python APIs for Spark Streaming.  The R community has developed several packages for working with Spark Streaming available on CRAN.

We do not use Spark Streaming in the use case described later in this eBook

GraphX

We have shown how all the Spark libraries discussed above leverage the Spark Core distributed memory abstraction referred to as an RDD and the Spark parallel execution engine.   Since graphs in computer science are an abstract data structure, it presents an opportunity to leverage the Spark Core engine for another form of specialized data analysis.

Graphs are defined by a set of vertices (nodes) and the corresponding edges (arcs).  GraphX uses two RDDs, one for the definition of the vertices and one for the definition of the edges for the physical data structures.   The abstract graph object along with any defined properties of the vertices and edges can be read and modified using the underlying RDDs.

GraphX includes a set of fundamental graph operators (subgraph, joinVertices, numVertices, …) as well as a variant of the Pregel API for implementing graph-parallel iterative algorithms such as Page Ranking.  GraphX comes with a variety of graph algorithms including PageRank, Connected components, Strongly connected components, and Triangle count are some of the many which were contributed by the Spark community.

We do not use GraphX in the use case described later in this eBook

Spark Programming

The availability of so many language choices available to Spark developers has “sparked” much debate over which is the best language.  The answer to that question is nuanced.  It is safe to say larger organizations with substantial investments in Spark are more likely to have developers using multiple languages.   Larger organizations also are more likely to have standards and a published usage policy especially for production clusters or shared development environments hosting time-critical projects.   In this section, we try to highlight a few of the differences that can be important and let our readers decide on the best policy for their organization and environment type.

There are two fundamental ways that developers can interact with Spark.  The safest way is writing code to automate Spark operations through a controller application.  An example would be reading data from HDFS, using Spark SQL to reshape the data and then calling a model training function on the transformed dataframe using MLlib. This example is part of the workflow that we implement in our test use case using Python.  

Even with the restrictions above there is still a reason to be concerned about the impact of choosing Python or R for these scenarios.  Since Python and R cannot enforce data type consistency between code and data structures in a compile phase, the corresponding Spark language APIs only support untyped dataframes.   Scala and Java developers can choose between Spark untyped dataframes and typed datasets.  The advantage of using a compiled language and a typed dataset means that developers can be notified of errors such as referencing a non-existent column of a dataset before the code is submitted to the cluster.   Jobs that are submitted to the cluster with errors may consume a significant amount of resources before encountering a bug and halting. Experienced developers will write and test code in non-production environments to avoid wasting precious resources, but things still can and do go wrong sometimes.

The other way of writing code for Spark is to have the Spark driver execute a user-supplied code block that is run as part of a job.  An example is a User Defined Function (UDF).  Experienced Spark programmers try to limit their use of UDFs whenever possible, but many forms of input data transformation and business logic need the flexibility of procedural code applied to an entire data set to accomplish a critical task.  This code will be executed by each worker for a partition started by the driver.  UDFs written in Scala and Java are typically a full order of magnitude (10x) faster than standard Python UDFs.  There is very little documentation available for UDFs written in R so proceed with caution if you are exploring that approach.

Understanding this difference in performance for UDFs written in different languages begins with the initial design choice to develop Spark with Scala compiled to run with the Java Virtual Machine.  UDFs written in Scala or Java and complied for the JVM can communicate with the workers with very little overhead.  Python and R code must run outside the JVM in another process since they cannot be compiled into Java bytecode.   Passing data and code between the worker running on the JVM and the outside process must be converted to a common protocol that requires serialization on one end and deserialization on the other.   This additional processing overhead is costly and gets more significant as the size of the data being process increases.

For both the reasons discussed above, Scala and Java both have advantages over Python and R.  So how do developers rate Scala vs Java?   The overwhelming consensus is that Scala code for Spark is more compact and easier to both write and maintain compared to Java.  Some of the criticisms of Java code complexity for Spark development were addressed with the release of Java 8 but even the latest articles on Spark development strongly favor Scala over Java.

Despite these facts, the use of R and especially Python for Spark seems to be growing rapidly.  Some of the very large organizations that often make presentations at conferences and publish technical papers on Spark are multi-language environments.  For some organizations, any loss of efficiency on a Spark cluster resulting from the use of Python or R might be acceptable in exchange for productivity gains of using those tools.  There are also new libraries and packages being developed by the Python and R communities to smooth the rough interfaces between these interpreted languages and Spark.

Also new developments from the Spark community are reducing or eliminating the need to use Sparks low-level data structures, transformations and actions.  Many analytics professionals may be able to achieve near performance parity with Scala with applications written completely in Python or R.   The future capabilities of the higher-level Spark libraries like Spark SQL, Spark Streaming, MLlib, and or GraphX combined with more sophisticated optimizers and perhaps code generators may someday eliminate most of the need for programming in Scala or Java directly with RDDs and low-level operators.

When Not To Use Spark

Spark is not a transactional processing system

The foundational philosophy of the Hadoop project was to develop data processing tools that leveraged the write-once-read-many pattern.  The Spark project including Core, SQL, GraphX and even Streaming all rely on this pattern for their speed and scale.  If data in the source system(s) change, it is better to bulk delete and reload portions of the data that Spark draws from rather than attempting to use RDDs or higher-level abstractions to control fine-grained updates.

 We discussed above some of the changes that were made to traditional RDBMSs to improve their ability to meet the challenges of analytics on the rapidly growing data volumes that came to be referred to as “big data”.  The scalable multi-user transactional semantics that those systems excel at just as limiting to big data analytics as the write-once-read-many optimizations in Spark are to transaction processing (TP).   If your application needs strong transactional consistency support, use a tool that is designed for that.  If the volume of historical data that the application maintains can be efficiently analyzed in the same TP system, then you may not need big data tools at all.  If your application accumulates more data than can be efficiently analyzed on the TP system or you need to consolidate data from multiple sources for analysis, then you most likely need big data tools like Spark.

Spark Streaming is streaming not messaging

The terminology used for data streaming and messaging applications is not well defined, and therefore, not used consistently.  In this discussion, we will both define our use of the terms and explain why we think treating these technologies separately is important.

When the stream is made up of relatively small units of information packaged as a complex data type it is likely that the application receiving this data will want to inspect each package as it arrives. The “listener” application will probably evaluate some logic using data in the package and then make one of more decisions regarding next steps.  The application could transform or extract a subset of the original package.  All the results of the inspection and logic could get routed to another microservice or the stream may be split into two or more new streams based on the application logic.  These are not good use cases for Spark Streaming.

The best use cases for Spark Streaming is to buffer incoming streams and fill the partitions of an RDD.   Programmers can have a single interface to all the data in the RDD for doing any length of time window analysis, however, Spark RDDs are especially useful when operating on multiple partitions in parallel.