Computer vision and machine learning (ML) solutions are integrated into automobiles to improve safety, convenience, and the driver experience. Sensor data coming from multiple sources such as camera, lidar, radar and ultrasonic sensors are processed and used to extract information and characteristics. Modern vehicles have adopted active safety systems – such as anti-lock braking and autonomous emergency braking (AEB), as they move from passive safety systems such as seat-belt pre-tensioning and airbags (designed to minimize injury during an accident).

With the evolution of artificial intelligence (AI) and deep learning (DL), it is possible to develop embedded control units (ECUs) into vehicles with computer vision algorithm that can interpret content from images and cloud point data and make corresponding predictions or actions.

The significant automotive safety improvement in the past was passive safety, designed to minimize damage during an accident. Advanced driver assistance systems (ADAS) can proactively help the driver to avoid accidents by using innovative deep learning (DL) technologies. For example, blind spot detection can alert a driver as they try to move into an occupied lane. Pedestrian detection notifies the driver that pedestrians are in front of or behind the car. AEB activates the brakes to avoid an accident or pedestrian injury. More ADAS features combine with sensor fusion. These bring us closers to the goal of an autonomous vehicle. As the level of accuracy and sophistication increases, autonomous driving (AD) can realize full capabilities. The critical success is the combination of improved safety algorithms, increased computational power, and access to large, comprehensive verification datasets.

This paper is exploring the infrastructure challenges faced by automotive Original Equipment Manufacturers (OEMs), Tier-1s, and Tier-2s in developing DL algorithms for ADAS / AD. Meanwhile, this paper also proposes a scale-out compute and storage solution. Optimized for ADAS / AD workloads - delivering high performance, high concurrency, massive scalability, and flexibility. It is optimized for ADAS / AD workloads - delivering high performance, high concurrency, massive scalability, and flexibility

Dell Technologies and the authors of this document welcome your feedback on this document. Contact the Dell Technologies team by email.

Author: Frances Hu

Contributors: Florian Baumann and Frank Han (Frank_Han1@Dell.com)

Other: Darren Miller (darren_miller1@dell.com)

Note: For links to other documentation for this topic, see the Powerscale (Isilon) Info Hub.

This chapter reviews the ADAS development cycle and the challenges associated with the DL development phases.

Figure 1 illustrates the typical ADAS development life cycle for automotive OEMs and suppliers leveraging the Dell   PowerScale scale-out NAS as the central data lake:

Figure 1.   ADAS Development Lifecycle

1. Data Acquisition: Huge volumes of sensor data are captured by a fleet of test vehicles, including camera video sequences, ultrasonic proximity, radar, Light Detection and Ranging (LiDAR), Global Positioning System (GPS), and other sensor data. Sensors with very high-resolution such as 4K (or greater) cameras are being adopted. Some of these sensors will be beta samples of the actual sensors planned for the production vehicle, while other sensors will be capturing high-resolution reference data (“ground truth”) around the test vehicle. Another important reference is the actions of the test driver – such as accelerator/brake/steering functions. Typically, we see ADAS developers generating real-world test data on the order of 2 TB per hour, or around 30 – 80 TB per car per day with some developers running test fleets with 50 or more vehicles. The data is stored with each vehicle in real-time using dedicated industrial data-logging hardware with removable solid-state storage disks. These drives are swapped out either daily or at the end of each shift – depending on the amount of data captured per shift. The drives are then either shipped directly to a centralized ingest server, transferred virtually through WAN lines, or transferred locally to tape, with the tapes then being shipped to a centralized ingest server for upload to the data lake (centralized storage).
2. Data Ingestion: During the data ingest process, which includes moving data from the vehicle to the data lake, custom copy stations apply data cleaning and lossless data compression algorithms to reduce the final amount of required storage and, therefore, costs. Data cleansing can occur in-line to avoid multiple copies or move operations. Typically, 10% - 15% of the recorded data is used to train the ML/DL algorithms and over 50% of the recorded data is used to run re-simulations as well.
3. Data Preparation: Once the data has been ingested, engineering teams will prepare the data, which may include trimming, decoding, data enrichment (labeling or ground truth generation), processing and adding metadata such as weather and traffic conditions. This requires a vast amount of compute and GPU resources in the high-performance computing (HPC) cluster to process the data, as well as fast storage - such as Dell   PowerScale storage, which meets both high capacity and high sequential read and write performance needs.
4. Test Preparation: Engineers can build test suites with various test use cases, as well as required closed-loop simulation and open-loop re-simulation (replay) validation jobs to verify ADAS models. With massive raw datasets, it is vital to be able to search through metadata quickly to find the corresponding sensor data for specific scenarios from within the vast data lake. Tests developed using captured sensor data tests against possible corner cases, with discrepancies between the ECU validation and test driver actions identified as potential bugs.
5. Design and Development Phase: When the data is ready, the ADAS engineering teams can develop and build algorithms for smart cameras or ECU models through DL. ML models are created by training them on a huge amount of data (traffic signs, lanes, vehicles, and people). With huge amounts of data, it is strongly recommended to run multi-GPU training with data parallelism. Data parallelism allows a higher throughput of images during training, and therefore reduced training time.
6. Re-simulations: After algorithms are developed, it is crucial to run iterative tests using data fusion of all the sensors, GPS, weather, and road/environment data. On small projects, individual sensors and ECUs may be tested independently. Then all subsystems are tested together at the system level (sensor fusion).It is important to test corner-cases and complex interaction of various ADAS functions, sophisticated scenarios involving various road environments, pedestrians, other vehicles, and driver behavior to ensure the safety of the system. With various test cases, the engineering teams can schedule re-simulation jobs on the hardware-in-the-loop (HiL) and software-in-the-loop (SiL) computer clusters. Re-sim involves “replaying” the captured raw sensor data back through the test farm – usually with hundreds or even thousands of iterations running in parallel. For HiL this will be replayed in real-time, and for SiL it is often replayed faster than real-time. This workload requires the inherent high-concurrency benefit of the Dell   PowerScale scale-out NAS architecture.
7. Analysis: Once testing is complete, engineers need to analyze the test results and determine whether additional validation is required. In-place analytics compare ECU operation to original test driver actions to quickly identify potential bugs. Algorithms are refined to achieve the expected output results, and the revised ECU version can be uploaded to the test vehicles adopting a continuous improvement process. All the results are sent to data center storage to provide the engineering teams with on-demand access.
8. Archive: Following final validation, data used to develop the ECU algorithms can move to lower-cost archive storage. Archiving must meet regulatory and contractual commitments, which typically span multiple decades – the agreed “life of the vehicle.” Many OEMs stipulate service-level agreements (SLAs) of 1 to 30 days for simulation data restoration time – for example, in the event of a safety recall – to allow quick turn-around of updates. Dell   strongly recommends an active archive solution such as ECS cloud neutral object storage, implementing the S3 API.

Note: The above steps are not time sequential. Steps are concurrent and continuous to ensure high-quality solution outcomes and efficiency.

Training and validating new deep neural networks, such as those used in ADAS / AD development, require large datasets along with significant IT infrastructure include compute power, networking, and storage. The right infrastructure is crucial with safety-critical system development where accuracy requirements standards are much higher than in other industries. These advanced algorithms must operate even within complex circumstances like varying weather conditions, visibility, and road surface quality.

Key challenges of the DL training workload for ADAS are:

• Explosive Data Growth – A typical vehicle used for sensor data collection in the ADAS system test use case includes multiple sensors such as LiDAR, RADAR, ultrasonic, GPS, and cameras – all of which continuously generate data. Also, the vehicle controller area network (CAN) bus data and test driver captures the control information. While the reality is impossible to predict with 100% accuracy, this high level of visibility and redundancy builds a detailed picture to enable the vehicle to make reliable decisions in adverse weather conditions or during an event of an individual component failure. Due to the safety requirements for driving, we need to ensure that the system used can detect objects sufficiently far away to operate safely at high speeds; this combination demands higher image resolutions than other industries use, which in turn generates more data. Massive challenges occur in terms of the scale of the unstructured sensor data (videos, cloud point, images, text) that must be captured and replayed to test ADAS subsystems.

To illustrate, a typical Society of Automotive Engineering (SAE) level 2 ADAS project, capturing 200,000 km (typical for SAE level 2) of driving at an average speed of 65 km/h, would generate over 3,076 hours of data, requiring approximately 3.8 petabytes (PB) of storage for one single sensor.

Note: that even within SAE level 2 solutions, the total number of ADAS sensors required varies with functionality (lane departure warning, self-parking, and more). Multiple sensors are typically required. For example, an SAE level 3 ADAS project, which typically requires 1,000,000 km of driving, could generate 19.3 PB of raw sensor data per car. As most ADAS developers have multiple cars, typical total storage requires averages between 50 – 100 PBs of data per vehicle model.

• Fast training cycle – To assure safety and reliability, the neural networks designed must utilize millions of parameters which generate more compute-intensive requirements for the underlying systems and hardware architecture. To accelerate time-to-market, neural network training must be as fast as possible. First, the deeper the network, the higher the number of parameters and operations need to store many intermediate results in GPU memory. Second, training usually proceeds in the method of mini-batches, I/O throughput is thus the primary performance metrics of concern in DL training.

To illustrate, running AlexNet – one of the least computationally heavy image classification models – together with a substantially smaller ImageNet dataset achieves a throughput of ~200MB/s on a single V100 GPU. For 1 PB of data, if we use one single V100 GPU with 50 epochs of training in average on ImageNet dataset, takes:

        50 epochs * 1PB of data / 200 MB/s = 7.9 years to train an AlexNet network

A single GPU server cannot address the large amount of data training and computational capability required for ADAS / AD. The ability to scale neural networks and train large datasets across GPU servers and scale-out storage is critical for ADAS / AD and required to support the distributed DL training.

• Test and validation – Validation is a key stage of the ADAS development cycle. Since most ADAS systems directly affect safety, the robustness and reliability of the trained model is paramount. This demands exhaustive testing and verification on the trained algorithm to represent diverse traffic scenarios and dimensions, which might include road geometry, driver and pedestrian behaviors, traffic conditions, weather conditions, vehicle characteristics and variants, spontaneous component faults, security, and more.
• High quality labeled data – The availability of labeled data is critical for ADAS DL training. High quality labeled data yields better model performance. Labels are added either manually (often via crowd sourcing) or automatically by image analysis, depending on the complexity of the problem. Labeling massive collections of training data with high quality is a tedious task and requires significant effort.

This chapter lists the key components recommended for distributed DL targeted at ADAS /AD development.

Dell   PowerScale all-flash storage platforms, powered by the PowerScale OneFS operating system, provide a powerful yet simple scale-out storage architecture to speed access to massive amounts of unstructured data, while dramatically reducing cost and complexity. With a highly dense design that contains four nodes within a single 4U chassis, PowerScale all-flash delivers extreme performance and efficiency for your most demanding unstructured data applications and workloads – including ADAS / AD. The Dell   PowerScale family includes four all-flash nodes recommended for DL workloads:

• Dell   PowerScale F200: Provides the performance of flash storage in a cost-effective form factor to address the needs of a wide variety of workloads. Each node allows you to scale raw storage capacity from 3.84 TB to 15.36 TB per node and up to 3.8 PB of raw capacity per cluster. The F200 includes in-line compression and deduplication. The minimum number of PowerScale nodes per cluster is three while the maximum cluster size is 252 nodes.
• Dell   PowerScale F600: With new NVMe drives, the F600 provides larger capacity with massive performance in a cost-effective compact form factor to power the most demanding workloads. Each node allows you to scale raw storage capacity from 15.36 TB to 61.4 TB per node and up to 15.48 PB of raw storage per cluster. The F600 includes inline software data compression and deduplication. The minimum number of nodes per cluster is three while the maximum cluster size is 252 nodes.
• Dell   PowerScale F800: Provides massive performance and capacity and delivers up to 250,000 IOPS and 15 GB/s aggregate throughput in a single chassis configuration and up to 15.75M IOPS and 945 GB/s of aggregate throughput in configurations of up to a 252-nodes cluster. Each chassis houses 60 SSDs with a capacity choice of 1.6 TB, 3.2 TB, 3.84 TB, 7.68 TB, or 15.36 TB per drive. This allows you to scale raw storage capacity from 96 TB to 924 TB in a single 4U chassis and up to 58 PB in a single cluster.
• Dell   PowerScale F810: Provides massive performance and capacity along with inline data compression and deduplication capabilities to deliver extreme efficiency. The F810 delivers up to 250,000 IOPS and 15 GB/s aggregate throughput in a single chassis configuration and up to 15.75M IOPS and 945 GB/s of aggregate throughput in a 252-node cluster. Each F810 chassis houses 60 SSDs with a capacity choice of 3.84 TB, 7.68 TB or 15.36 TB per drive. This allows you to scale raw storage capacity from 230 TB to 924 TB in a 4U chassis and up to 58 PB of raw storage in a single cluster. Depending on your specific dataset and workload, F810 inline data compression and deduplication delivers up to a 3:1 reduction in storage requirements, this increasing the effective capacity up to 138 PB per cluster. For more information, see the document PowerScale All-Flash Scale-Out NAS Specification Sheet.

Figure 2.   Dell   Isilon F800/F810

Dell   PowerScale families have the following features to benefit DL:

• Low latency, high throughput and massively parallel I/O for AI. This shortens time for training and testing analytical models on data sets from tens of TB to hundreds of PB on AI platforms such as TensorFlow, SparkML, Caffe, or proprietary AI platforms. Isilon F810/F800 performance characters are listed as below:

        Up to 250,000 file IOPS per chassis, up to 15.75M IOPS per cluster

        Up to 15 GB/s throughput per chassis, up to 945 GB/s per cluster

        230 TB to 924 TB raw flash capacity per chassis; up to 58 PB per cluster (All-Flash)

• The ability to run AI in-place on data using multi-protocol access. Most data used for DL training is also used for other workloads, like HiL and SiL validation. These workloads, which use captured sensor data, typically require lower cost hybrid storage, such as Isilon H5600 but in one PowerScale cluster. This eliminates the need to migrate/copy data and results over to a separate AI stack. Organizations can perform DL and run other IT apps on same data already on PowerScale by adding PowerScale all-flash nodes to the existing cluster.

        Multi-protocol support such as SMB, NFS, HTTP, and native HDFS to maximize operational flexibility

• Enterprise grade features out-of-box. This enables organizations to manage AI data throughout the lifecycle of the ADAS project with minimal cost and risk, while protecting data and meeting regulatory requirements.

        Enterprise data protection and resiliency

        Robust security options

        Economical, long-term archival with fast recovery

        Data Management System using a single pane of glass

        Container Storage Interface (CSI) driver for provisioning of persistent storage

• Extreme scale support. Organizations can achieve AI at scale in a cost-effective manner by leveraging PowerScale for DL as well as other ADAS workflows. Enabling them to handle multi- petabyte data sets with high resolution content and high performance without the need to re-architect their data center, and/or performance degradation.

        Seamlessly tier between all flash, hybrid, and archive nodes via SmartPools.

        Grow-as-you-go scalability with up to 58 PB capacity per cluster

Up to 64 chassis (252 nodes) may be connected to form a single cluster with a single namespace and a single coherent cache

        Depending on your specific dataset and workload, F810 inline data compression and deduplication delivers up to a 3:1 reduction in storage requirements, this increasing the effective capacity up to 138 PB per cluster

DL methods would not have gained success without the computational power to drive the iterative training process. Therefore, a key component of DL solutions is highly capable nodes that can support compute intensive workloads. State-of-the-art neural network models in DL have more than 100 layers which require computations to be able to scale across many compute nodes. The Dell   PowerEdge C4140, an accelerator-optimized, high density 1U rack server, is used as the compute node unit in this solution. The PowerEdge C4140 server used is Configuration M, which supports four V100 SMX2 GPUs.

The PowerEdge C4140 is an incredibly dense rack server designed to handle the most demanding technical computing workloads in DL, ADAS development and HPC. The PowerEdge C4140 provides maximum density and outstanding thermal efficiency for scale-out performance. Speed up your DL development workloads with four GPUs/FPGAs in a 1U, 2-socket package.

The PowerEdge C4140 ML reference designs include:

• NVIDIA V100 SXM2 modules, each with 16 GB of memory
• Dual Intel Xeon Gold Scalable 6148 processors (Skylake/Purley generation)
• 384 GB of DDR4 memory at 2667 MHz
• Two 120GB M.2 SSDs
• Bright Computing’s Bright Cluster Manager

Figure 3.   Dell   PowerEdge C4140 Server

The NVIDIA V100 GPU is the most advanced data center GPU ever built to accelerate AI, high performance computing (HPC), data science and graphics. Powered by NVIDIA Volta™ GPU architecture, V100 GPUs enable data scientists, researchers, and engineers to tackle challenges that were once difficult. With 640 Tensor Cores, the V100 GPU is the first GPU to break the 100 teraflops (TFLOPS) barrier of DL performance.

In the PCIe version of the V100 GPU, all GPUs communicate with each other over PCIe buses. With the V100-SXM2 model, all GPUs are connected by NVIDIA NVLink™ technology. In use-cases where multiple GPUs are required, the NVLink interconnect used by V100 cards provide the advantage of faster GPU-to- GPU communication when compared to PCIe. V100 GPUs provide six NVLink lanes per GPU for bi- directional communication. The bandwidth of each NVLink lane is 25 GB/s in uni-direction and all four GPUs within a node can communicate simultaneously, therefore the theoretical peak bandwidth is 6*25*4=600 GB/s in bi-direction. However, the theoretical peak bandwidth using PCIe is only 16*2=32 GB/s as the GPUs can only communicate serially, which means the communication cannot be done in parallel. In theory the data communication with NVLink lane could be up to 600/32=18x faster than PCIe. Because of this advantage, the PowerEdge C4140 compute node in the DL solution uses V100 GPUs instead of PCIe-based V100 GPUs. In this solution, both 16 GB and 32 GB V100 GPUs are supported.

Figure 4.   NVIDIA V100 GPU

• The first piece is Bright Cluster Manager which is used to easily deploy and manage the clustered infrastructure and provides all cluster software including the operating system, GPU drivers and libraries, InfiniBand drivers and libraries, MPI middleware, the Slurm schedule, so on
• The second piece is the Bright ML which includes any DL library dependencies to the base operating system, DL frameworks including PyTorch, Theano, TensorFlow, Horovod, Keras, DIGITS, CNTK and MXNet, and DL libraries including cuDNN, NCCL, and the CUDA toolkit.
• The third piece is the Data Science Provisioning Portal which was developed by Dell  . The portal was created to abstract the complexity of the DL ecosystems by providing a single pane of glass which provides users with an interface to get started with their models. The portal includes spawner for Jupyterhub and integrates with

        Resource managers and schedulers (Slurm)

        LDAP for user management

        DL framework environments (Tensor Flow, Keras, MXNet, PyTorch etc.) from Bright’s module environment, Python2, Python3, and R kernel support

        TensorBoard

        Terminal CLI environments.

DL models are very complex and large, and the DL framework is an interface, library, or a tool which allow developers to building DL easily and quickly, without requiring in-depth understanding of all the details of the underlying algorithms. These frameworks provide a clear and concise way for defining models using a collection of pre-built and pre-optimized components. The popular DL framework includes TensorFlow, Keras, PyTorch, Caffe so on

Below are some key characteristics of a well-designed DL framework:

• • Optimized for GPU performance
• • Easy to understand and code
• • Extensive community support
• • Process parallelization to reduce computation cycles
• • Automatically computed gradients
• • Cloud-native app development capability (Docker, Kubernetes so on)

Training with large datasets and deep learn networks can be accelerated by using multiple GPUs and/or more servers, but only if the underlying infrastructure is architecturally correctly.

In the market, there are some popular platforms and toolkits to allow developers to test distributed execution of different DL platforms on GPU clusters including MPI based Uber Horovod and Microsoft website Microsoft Distributed Machine Learning Toolkit (DMTK). Horovod is a distributed training framework for TensorFlow, Keras, PyTorch, and MXNet. The goal of Horovod is to make distributed DL fast and easy to use. These platforms are designed to make large-scale parallel distributed DL jobs easy and better.

Figure 5 illustrates the reference architecture showing the key components of the solution as it was tested and benchmarked.

Note: In a customer deployment, the number of Dell   PowerEdge compute servers and PowerScale storage nodes will vary and can be scaled independently to meet the requirements of the specific DL workloads.

Figure 5.   Dell   ADAS / AD DL Reference Architecture

The hardware architectures include these key components:

• Compute: High performance GPU server nodes that can support compute intensive workloads are central for any DL solution. We used 16 PowerEdge C4140 servers with four V100 GPUs each (64 GPUs total) in this solution for the distributed DL training workload.

The Dell   PowerEdge R740xd is used for the role of the head node. This is Dell  ’s latest two socket, 2U rack server that can support the memory capacities, I/O needs and network options required of the head node. The head node will perform the cluster administration, cluster management, NFS server, user login node and compilation node roles.

• Storage: A critical component of DL solutions is high performance storage. It is uniquely suited for modern DL applications delivering the flexibility to deal with any data type, scalability for data sets ranging in the PBs, and concurrency to support the massive concurrent I/O request from the GPUs. This solution also mounted NFS share on either the head node or each compute node.
• Networking: The solution consists of three network fabrics. The head node and all compute nodes are connected with a 1 Gigabit Ethernet fabric. The Ethernet switch recommended is the Dell Networking S3048-ON which has 48 ports. This connection is primarily used by Bright Cluster Manager for deployment, maintenance and monitoring of the solution. The second fabric connects the head node and all compute nodes through 100 Gb/s EDR InfiniBand. The third switch in the solution is called a gateway switch and connects the Isilon F800 to EDR switch with Isilon’s front end 40 Gigabit Ethernet connection.

The software architectures include these key components

• Resource scheduler and management: Bright Cluster Manager is used in this solution to easily deploy and manage the clustered infrastructure and provides all cluster software including the operating system, GPU drivers and libraries, InfiniBand drivers and libraries, MPI middleware, the Slurm scheduler, and more.
• Docker containers or virtual machines: There has been a dramatic rise in the use of software containers for simplifying deployment of applications at scale. You can use either virtual machines or containers encapsulated by all the application’s dependencies to provide reliable execution of DL training jobs. A docker container is more widely used now for bunding an application with all its libraries, configuration files, and environment variables so that the execution environment is always the same. To enable of portability in Docker images that leverages GPUs, NVIDIA developed the Docker Engine Utility for NVIDIA GPUs which is also known as the NVIDIA Container Toolkit, an open-source project that provides a command-line tool. The publicly available CSI driver for the scale-out NAS PowerScale provides support for provisioning of persistent storage.

Apart from the DL architecture, the development environment needs a rich ecosystem of open-source tools and partner-led solutions specific to the unique requirements of each customer, as shown in Figure 6. Dell Technologies has put together the following map of workflow and possible ecosystem solutions. More information can be collected from a blog “What DevOps for AD/ADAS looks like” here: Link.

Figure 6.   Dell Autonomous Drive Ecosystem

This chapter describes the key considerations and best practices of designing distributed ADAS DL solution.

Here are some key design considerations for preparing ADAS DL datasets:

• Collect real-world raw dataset: An end-to-end DL system requires large amounts of training data for the development of Deep Neural Networks (DNN) that can successfully interpret virtually all driving scenarios that are likely to be encountered in the real world. OEMs and Tier-1 suppliers typically deploy fleets of test cars outfitted with sensors to capture this data. Sensor data rates of around 2TB/hour per car is common. The typical OEM or Tier-1 will have over 50 cars capturing data around the world, generating a total raw dataset on the order of 20 petabytes per year.
• Data and scenarios diversity: The recorded sensor data is used to train Deep Neural Networks in the data center, with over 1 million images per DNN viewed as a best practice for relatively simple DNNs (complex DNNs typically will require up to 3 million images). A RAND report highlights that 11 billion miles from realistic road scenarios are needed to assure the safety of ADAS/AD DNN models. The data also must be diverse enough to cover different type of road scenarios to verify the model accuracy.
• Public ADAS / AD dataset utilization: Existing public AD datasets can be heavily leveraged to verify the algorithm accuracy as well. As shown in Table 1, many multimodal datasets from different countries are now available, where each are typically comprised of images, range sensor (lidar, radar) data and GPS/IMU data. These are available to researcher and developers to accelerate the development and validation of ADAS system (subject to individual license restrictions).

Table 1.      Major Public ADAS / AD Dataset

Here are some key design considerations for data ingestion and data management for ADAS DL workflow:

• Large scale sensor data and metadata management: Dealing with data access, data ingestion, and metadata management with HPC jobs can become a significant waste of time and effort for the data scientist. To alleviate this, Dell Technologies has created a custom-tailored Data Management System (DMS) that is available as part of our PowerScale-based ADAS solution offering, as shown in following diagram. DMS is integrated into the managed service offering for ingesting data from test vehicles in an automated fashion. Specific metadata types and tags are configurable and can be customized for each customer. The service provider can manage logistics to receive the storage media from the vehicles and insert them into ingestion stations. This will automatically initiate processes that dynamically decide which cluster/location to upload the raw data to – in such a fashion that all ingestion processes are well balanced across the storage environment. DMS then initiates the data transfer automatically while enriching the data with appropriate metadata tagging, resulting in all data and tags being logged in a global index that tracks the data sets across the namespace and clusters. At the completion of the data transfer and indexing, DMS can launch post-processing jobs within the DL and HPC server grid as preconfigured through the scheduling platform. Functions available can include accurately splitting and merging data files as required, decoding the data into

human readable form, automatically updating the index upon completion, and other functions as defined by the customer. These processes are wrapped within a series of real-time monitoring and status dashboards and logs to alert of any errors or anomalies.

Dell   Data Management Solution Dashboard:

• Hybrid Cloud Solution for ADAS Development: Auto organizations are investing heavily in new technologies to develop advanced driver-assistance systems (ADAS). One of the key hurdles these companies face in this pursuit, is the fact that sensor data is often spread throughout the organization’s environment—across platforms and geographic locations. This results in large, disconnected data silos that many find it difficult to manage as these sensor files must be made available to a range of test environments and AI tools. In addition, sensor files are typically kept for multiple years or even decades after testing – the “life of the car” – and must brought online quickly (often within weeks) . With the new release of Dell   DataIQ products as shown in Figure 7, it solves for these challenges with its ability to break down storage silos and unify the unstructured data environment. It provides a single UI to view sensor data across heterogenous storage environments including the public cloud.

DataIQ also equips organizations with the ability to rapidly search across billions of files and locate data, which can then be precisely moved or copied to on-prem or public cloud storage as needed, ensuring the right projects have access to pertinent data. This can help free up space on higher performance tiers and enables organizations to archive data as long as is required. The cost savings can be immense, and data can be retrieved easily from a well-curated, searchable archive for re- simulation and DL development in the future.

Figure 7.   New Hybrid Cloud Solution for ADAS Development with Dell   DataIQ

Training neural networks with images requires developers to first normalize those images. Moreover, images are often compressed to save storage. Developers have therefore built multi-stage data processing pipelines that include loading, decoding, cropping, resizing, and many other augmentation operators. Here are some key design considerations for data ingestion and data management for an ADAS DL workflow:

• Data augmentation strategy: Dataset augmentation applies transformations to training data. Transformations can be as simple as flipping an image, or as complicated as applying neural style transfer. By performing data augmentation, it is possible to increase training dataset size and diversity. This can help to prevent a neural network from learning irrelevant patterns (common with small datasets), essentially boosting overall performance quality. Here are two common methods for data augmentation:

        Offline augmentation is to create a new augmented data which stored on storage. This can help effectively increasing the training sample size many times over with variety of different augmentation technologies.

        Online augmentation is to apply augmentation to data in real time before the images are fed to the neural network. CPUs are used heavily in background / parallel for online augmentations during the training.

• Use NVIDIA Data Loading Library (DALI): The NVIDIA Data Loading Library (DALI) is a portable, open source library for decoding and augmenting images, videos, and speech to accelerate DL applications. As shown in Figure 8, DALI is a set of highly optimized building blocks plus an execution engine to accelerate input data pre-processing for DL applications. DALI provides performance and flexibility for accelerating different data pipelines. DALI reduces latency and training time, mitigating bottlenecks, by overlapping training and pre-processing. It provides a drop-in replacement for built in data loaders and data iterators in popular DL frameworks for easy integration or retargeting to different frameworks. Here are some key features:

        Easy-to-use Python API

        Transparent scaling across multiple GPUs

        Accelerated image classification (ResNet-50), object detection (SSD) workloads and speech recognition models such as Jasper and RNN-T.

        Flexible graphs let developers create custom pipelines

        Supports multiple data formats - LMDB, RecordIO, TFRecord, COCO, JPEG, wav, flac, ogg,

        H.264 and HEVC

        Developers can add custom audio image and video processing operators For more information, refer to NVIDIA blog.

Figure 8.   DALI inside architecture

The ability to train neural network data models with many hidden layers, as well as the ability to train them with large datasets in a short amount of time at scale is critical to ADAS/AD development. To assure safety and reliability, the neural networks designed for driving operations will utilize many permutations of parameters which will generate more compute-intensive requirements for the underlying systems and hardware architecture. In distributed DL platforms, the model needs to be synchronized across all nodes. It also requires the careful management and distributed coordination of computation and communication across all nodes.

Here are some key considerations for designing scalable neural networks:

• Data parallelism vs model parallelism: Data parallelism, as shown below, is generally easier to implement. Each device works with a different part of the overall dataset and the devices collectively update a shared model. These devices can be located on a single machine or across multiple machines.

Data Parallelism:

Most DL frameworks use data parallelism to partition the workload over multiple devices. Following picture shows the detail process of data parallelism approach to distribute training processes across the multiple GPU servers and devices. Data parallelism also requires less communication between nodes as it benefit from high amount of computations per weight. Assume for example, there are n devices, where each device receives a copy of the complete model and train it with 1/nth of the data. The results such as gradients and the updated model itself are communicated across these devices.

To ensure the efficient training, the network bandwidth across the nodes cannot become a bottleneck. Also note, it is inefficient – and bad practice - to store training data on the local disks of every worker node – this forces the copying of terabytes of data to each worker node before the actual training can be started.

parallelism approach to distribute training processes:

When models are so large that they don’t fit into device memory, then an alternative, called model parallelism shown as below. Different devices are assigned the task of learning different parts of the model. Model parallelism requires more careful consideration of dependences between the model parameters. Model parallelism may work well for GPUs in a single server that share a high-speed bus. It can be used with larger models as hardware constraints per node are no more a limitation.

Model Parallelism:

• Leverage open source DL toolkit like Horovod: Horovod is a distributed training framework for TensorFlow, Keras, PyTorch, and MXNet. The goal of Horovod is to make distributed DL fast and easy to use. Horovod use Message Passing Interface (MPI) model to be much more straightforward and require far less code changes than the Distributed TensorFlow with parameter servers. Horovod currently supports models that fit into one server but may span multiple GPUs.
• Use the latest DL framework: Leverage the latest DL framework and toolkit to avoid potential performance issue or bugs, and compatible cuDNN and NVIDIA Collective Communications Library (NCCL). The newer version always delivers great performance improvements and bug fixes.

Here are some key design considerations on designing infrastructure for distributed DL:

• Build high network bandwidth between nodes: Distributed DL is a very compute-intensive task. To accelerate computation, training can be distributed across multiple machines connected by a network. During the training, constant synchronization between GPUs within and across the servers is required. Limited network bandwidth is one of the key bottlenecks towards the scalability of distributed DNN training. Most important metrics for the interconnection network are low latency and high bandwidth. It is recommended to use InfiniBand or 100Gbps Ethernet Network to maximum the network bandwidth between nodes. With enough network bandwidth, GPU utilization across nodes performs similar of single GPU configurations.
• Choose high throughput, scale-out centralized storage: With scalable distributed DL, the distribution of the training data (batches) to the worker nodes is crucial. It is inefficient and expansive to store training data in the local disks or Non-volatile random-access memory (NVRAM) on every worker node and copy terabytes of data across each worker node before the actual training can be started. Using a centralized storage for the training data, like a high throughput, scale-out storage offers a more convenient and effective solution for ADAS / AD DL training.

GPU servers can have 20-30k cores and each one request its own read thread. Most file systems max out at between 2,000 – 6,0000 open read threads, but PowerScale OneFS operating system doesn’t have performance limit on open threads. Meanwhile, DL workloads require high throughput random read during the training and data processing, and sequential write during data ingest. With distributed DL on multiple GPUs, to full utilize GPUs a steady flow of data from centralized storage into multiple GPUs jobs is critical for obtaining optimal and timely results. Data preprocessing (CPU) and model execution of a training step (GPU) run in parallel during the training require high throughput data read from storage.

An ADAS high performance DL system requires equally high-performance storage with scalability to multiple petabyte within a single file system. For a typical SAE level 3 project, this requirement ranges between 50 to 100 PB of data. The storage scalability is crucial for ADAS DL project with high performance to meet different business requirement.

Dell   PowerScale all-flash storage platforms, powered by the PowerScale OneFS operating system, provide a powerful yet simple scale-out storage architecture up to 33 petabytes per cluster. It allows DL workload to access massive amounts of unstructured data with high performance, while dramatically reducing cost and complexity.

• Use Bright Cluster Manager (BCM) to provide cluster management and monitoring: Dell and Bright Computing have had a long track record of working together with both companies’ missions aligning to speed customer time-to-value, discovery, and generating data-based insights. BCM will enable the deployment of complete clusters over bare metal and manage them effectively. It will provide the management for the entire compute cluster, including the hardware, operating system, and users. It also has the capability to manage select applications within the HPC and AI environments such as Hadoop, Spark, and other big data software elements. BCM is making DL more accessible by providing software that makes it easy to deploy and manage the tools and supporting infrastructure required to get the job done. Bright for DL provides a complete platform to scale DL across an AD/ADAS environment with confidence. For more information, refer to Bright Computing DL website.
• Use Slurm Scheduler for DL training jobs across HPC nodes: To use resources on multiple nodes, the user can use Slurm job scheduler. The Slurm scheduler is used to manage the resource allocation and job submissions for all users in a cluster. To use Slurm, the user needs to submit a script on the cluster head node that specifies what resources are required and what job should be executed on those resources once allocated.
• Leverage NVIDIA Container Toolkits on NVIDIA GPUs: NVIDIA offers ready-to-run GPU- accelerated containers with the top DL frameworks. NVIDIA also offers a variety of pre-built containers which allow users to build and run GPU accelerated Docker containers quickly. To deploy DL applications at scale, it is very easy to leverage containers encapsulating an application’s dependencies to provide reliable execution of application and services even without the overhead of a full virtual machine. For more detail information, refer to the NVIDIA Container Toolkit page.
• Use Kubernetes to manage containers for distributed DL: Kubernetes is an increasingly popular option for training deep neural networks at scale, as it offers flexibility to use different ML frameworks via containers as well as the agility to scale on demand. It allows researchers to automate and accelerate DL training with their own Kubernetes GPU cluster. For more information, refer to article How to automate DL training with Kubernetes GPU-cluster.
• Use NVIDIA DL GPU Training System™ (DIGITS) for user interface: DIGITS is a wrapper for NVCaffe™ and TensorFlow™ which provides a graphical web interface to those frameworks in Figure 9 rather than dealing with them directly on the command-line. DIGITS simplify common DL tasks such as managing data, designing and training neural networks on multi-GPU systems, monitoring performance in real time with advanced visualizations, and selecting the best performing model from the results browser for deployment. DIGITS is completely interactive so that data scientists can focus on designing and training networks rather than programming and debugging.

Figure 9.   NVIDIA DIGITS for DL user interface platform

In this section, the performance of DL training is tested for Cityscapes Object Detection and Semantic Image Segmentation, as shown in Table 1. The well-known dataset used was the Cityscapes Benchmark which consist of 2,975 training images with high quality dense pixel annotation, 500 validation images and 1,525 test images. This dataset is commonly used by ADAS DL researchers for benchmarking and comparison studies. Cityscapes has labeled 30 different classes.

Since the entire Cityscapes raw data including label data is only 11 GB and can easily fit into system memory, the size of the dataset was increased 557 times to exercise the storage system (Isilon F800) properly We did this by applying 557 random data augmentation techniques to each JPEG image in the dataset. This is standard practice in data analytics and DL to increase size of data sets. In total, this “557x” dataset contained 1,657,075 images and annotations with 5TB in total. The average PNG image pixel is 1024 x 2048 and average size is 2,193KB. Figure 10 shows DataIQ scan results of the training dataset stored on the Isilon F800.

Figure 10.    DataIQ scan results of the training datasets on Isilon F800

The performance test utilized in this document uses this data to train two different convolutional neural network (CNN) models, shown in Table 2 that are used for semantic image segmentation and object detection.

Table 2.      Convolutional Neural Network models used in the tests

In this test, the hardware comprises sixteen compute nodes, one head node, four Isilon F800 nodes, and network switches. The details are showed in Table 3.

Table 3.      Hardware Configuration

In order to measure the training time and bandwidth requirement for distributed DL platform, different training procedures were carefully executed by using different configuration setups.

Real-time object detection model training test methodology

Here are some key test methodologies that we used to during the training benchmark

• In order to measure the distributed training performance of the solution, we used SSD (Single Shot MultiBox Detector) model from the NVIDIA DL Examples GitHub repository, refer to the link. This suite of benchmarks performs training of augmented Cityscapes labeled images. This dataset contains 972,825 training images in 3TB. This dataset is commonly used by automotive DL researchers for benchmarking and comparison studies. The solution used CityscapesScripts to convert annotations in standard PNG format to COCO format for SSD training.
• Training of SSD requires computational costly augmentations. To fully utilize GPUs during the training, we are using the NVIDIA DALI library to accelerate data preparation pipelines.
• Scale-out trainings were performed from one Dell   C4140 compute node with four V100 GPUs to 16 Dell   C4140 compute nodes with 64 V100 GPUs to measure the training time and throughput performance in order to provide a basic understanding of the training performance.
• In order to measure the training time across multi-GPUs and bandwidth requirement, we trained SSD model for 1 epoch and 2 epochs with the following setup:

        SGD with momentum: 0.9

        Learning rate: 2.6e-3 * number of GPUs * (batch_size / 32)

        batch size:16 per GPU

        number of worker threads: 20

        no warmup

        ResNet-50 is used as backbone

• Prior to each execution of the benchmark, the L1 and L2 caches on Isilon F800 were flushed with the command isi_for_array isi_flush. In addition, the Linux buffer cache was flushed on all compute nodes by running sync; echo 3 > /proc/sys/vm/drop_caches. However, note that the training process will read the same files repeatedly and after just several minutes, much of the data will be served from one of these caches.
• Multi-GPU training with Distributed Data Parallel – the NVIDIA model uses Apex's DDP to implement efficient multi-GPU training with NCCL.
• Excepting the training model, we also used the following command to evaluate the training benchmark:

python -m torch.distributed.launch --nproc_per_node={Number of GPUs} -- nnodes={number of compute node} --node_rank={node_rank}\

main.py --batch-size 16 \

        mode benchmark-training \

        benchmark-warmup 100 \

        benchmark-iterations 200 \

        data {data}

Semantic image segmentation model training test methodology

Here are some key test methodologies that we used to during the training benchmark:

• In order to measure the distributed training performance of the solution, we used deeplab-v3-plus model from the GitHub repository. This training performed on augmented cityscapes labeled images. This dataset contains 1,657,075 training images in 5TB
• Scale-out trainings were performed to measure the training time and throughput performance in order to provide a basic understanding of the training performance. Hardware configurations used for this testing ranged from one Dell   C4140 compute node with four V100 GPUs to 16 Dell   C4140 compute nodes with 64 V100 GPUs.
• In order to measure the training time across multi-GPUs and evaluate bandwidth requirement, we trained deeplab-v3-plus model for 1 epoch and 2 epochs with the following setup:

        batch size:16 per GPU

        number of worker threads: 20

        no warmup

        ResNet-101 is used as backbone

• Prior to each execution of the benchmark, the L1 and L2 caches on Isilon F800 were flushed with the command isi_for_array isi_flush. In addition, the Linux buffer cache was flushed on all compute nodes by running sync; echo 3 > /proc/sys/vm/drop_caches. However, note that the training process will read the same files repeatedly and after just several minutes, much of the data will be served from one of these caches.
• Uber Horovod was used with distributed data parallel. Horovod is a distributed DL training framework for PyTorch and other DL frameworks. Horovod uses Message Passing Interface (MPI) for high- speed and low-latency communication between processes. For example, we used the following command to evaluate the training benchmark:

mpirun -np {number of process}\

--mca oob_tcp_if_include {NIC}\

--mca btl_tcp_if_include {NIC}\

 -H nodexx:4,nodexx:4,nodexx:4,nodexx:4,nodexx:4,nodexx:4,nodexx:4,nodexx:4,no dexx:4,nodexx:4,nodexx:4,nodexx:4,nodexx:4,nodexx:4,nodexx:4,nodexx:4 \

 -bind-to none -map-by slot \

 -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \

 -mca pml ob1 -mca btl ^openib \

 python trainhorovod.py --dataset cityscapes --workers 20 --batch-size 16 --epochs 2

Training image throughput

The Single Shot MultiBox Detector model training benchmark performance results are shown in Figure 11. It is obvious from the results that image throughput scale linearly from four to 64 GPUs.

Figure 11.    SSD Model Training Benchmark – Images/sec from 4 GPUs to 64GPUs

The DeepLabv3+ model training performance results are shown in Figure 12.The test results also show the training throughput grow linearly from 4 GPUs to 64 GPUs.

Figure 12.    DeepLabv3+ Model Training Benchmark – Images/sec from 4GPUs to 64GPUs

Training time and system metrics

SSD model

Table 4 lists the test results about the SSD model training time and bandwidth for 1st epoch. As shown in Figure 13, the average throughput scale linearly from 4 to 64 GPUs and reduced training time as well.

Table 4.      SSD Training Time and Training Bandwidth for each epoch with 3TB dataset

Table 5 lists the system metrics during the training. This indicates that the GPUs were fully utilized by leveraging the NVIDIA DALI library.

Table 5.      SSD Model Training System Metrics

Figure 13.    SSD Model Training Average Bandwidth (MB/s) from 4GPUs to 64GPUs

As shown in the Figure 14, during the training time, the peak storage throughput reached over 10GB/s with 64 GPUs. It was observed that the training is a heavily read intensive workload as it need to read entire dataset from storage for training. For storage configuration considerations, it is crucial to plan sizing for peak throughput on Isilon F800 storage during the training in order to minimize training times as required for the development business needs.

Figure 14.    SSD Model Training with 64GPU storage throughput

As shown in the Figure 15, during the training time, GPUs are fully utilized.

Figure 15.    SSD Model Training with average 64GPU utilization

DeepLabv3+ model

Table 6 lists the test results about the DeepLabv3+ model training time and bandwidth for 1st epoch. As shown in Figure 16, the average throughput scaled linearly from 4 to 64 GPUs and reduced training time as well.

Table 6.      DeepLabv3+ Training Time and Training Bandwidth for each epoch with 5TB dataset

Table 7 lists the system metrics during the training. This indicates that the GPUs were fully utilized by leverage NVIDIA DALI library.

Table 7.      DeepLabv3+ Model Training System Metrics

Figure 16.    DeepLabv3+ Model Training Bandwidth (MB/s) from 4GPUs to 64GPUs

As shown in the Figure 17, during the training time, the peak storage throughput reached to over 2.2GB/s with read intensive.

Figure 17.    DeepLabv3+ Model Training with 64GPU storage throughput

From Figure 18 test results, during the training time, GPUs are fully utilized.

Figure 18.    DeepLabv3+ Model Training with average 64GPU utilization

Image throughput with multiple epochs

We also ran SSD model training with multiple epochs to compare and contrast. Figure 19 compares test results by running 1 epoch versus multiple epochs. It is obvious that running 1 epoch or multiple epochs had no difference on the image/sec performance results.

Figure 19.    SSD Model training Images throughput comparison

Meanwhile, we observed the storage throughput dropped while running more than 32GPUs, as shown in Figure 20. This was because the dataset is only 3TB, while running more than 32GPUs with 8 compute nodes, the server cache exceeded the total dataset, enabling some of the data to cached on the server side - which reduced the storage throughput required during the training.

For training, the system memory is consumed by the following critical variables: parameters - including weights and bias, gradients, input data and intermediate data. Among them, the intermediate data is the most memory demanding. The intermediate data includes the workspace and feature map. The workspace is the memory consumed by the DL framework and varies with each framework developers choose. The feature map, sitting in the middle of two neighbor layers of a CNN model, is generated by one layer, and used as input to the next layer. With more system memory, it gives more cache for intermediate data which will reduce the storage throughput requirement during the training.

Figure 20.    Model training storage throughput comparison

Here are some key considerations for DL training that we observed during the tests:

• Different CNN models generate varying throughput requirement: Comparing the test results, different CNN models can greatly vary the performance requirements for storage bandwidth. Complex CNN models (like DeepLabv3+) may generate less storage throughput than simple CNN models (like SSD). A high-resolution dataset can generate higher storage bandwidth needs even for the same CNN models. Dell Technologies published a whitepaper Deep Learning with Dell   Isilon which provides a sizing guideline for different CNN models.
• Multiple GPUs and servers are particularly beneficial to large dataset training: From our tests, we were be able to observe the training time for 1 epoch reduced from 42 minutes to 4 minutes by using 64 GPUs with 3 TB datasets. It reduced 10x training time for large dataset DL model development.
• Storage throughput requirement grows linearly with the increase of total GPUs during training: From our test results, storage throughput grows linearly with the increase of GPU numbers during the model training. During the sizing for DL infrastructure, it is important to plan for the storage for future GPU growth as well.
• Larger dataset can generate higher bandwidth requirements during training: It is observed that large dataset which exceeded the compute server cache will consistently generate higher bandwidth requirement. During the sizing, it is crucial to size your infrastructure and consider about the future data growth especial for ADAS development.
• NVIDIA DALI allows the training to run at full speed: GPU utilization is generally much higher than CPU utilization. In most of the training cases, the average GPU utilization can reach up to 97%. Low

GPU usage could be caused by CPU bottlenecks. When the GPU utilization is low, the CPU is busy with data fetching from storage to memory or from working on small operations. DALI reduces latency and training time, mitigating bottlenecks, by overlapping training and pre-processing -- which maximizes the training speed. DALI is primarily designed to do preprocessing on a GPU, but most operations also have a fast CPU implementation.

• Batch size should be large enough to reduce training time: It is observed that larger batch size will decrease the training time with higher GPU utilization. The larger batch size will also increase the throughput, so eliminating storage bottlenecks is also required to accelerate the training time. Also, some research by Facebook has shown that in distributed learning, the learning rate scaling rule is surprisingly effective for a broad range of batch sizes. It is recommended to increase learning rate following batch size. For more information, refer to Facebook article Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.

This document discussed key features of the scale-out NAS Dell   PowerScale as a powerful persistent storage solution for ADAS DL solutions. We presented a typical hardware architecture for DL by combining Dell   PowerEdge servers with embedded NVIDIA Volta GPUs and all-flash PowerScale storage. We ran several object detections and reported system performance based on the rate of images processed and throughput profile of I/O to disk. We also monitored and reported the CPU, GPU utilization and memory statistics that demonstrated that the server, GPU and memory resources were fully utilized while the Dell   PowerScale was still capable of providing more IOs.

DL algorithms have a diverse set of requirements with various compute, memory, I/O and disk capacity profiles. That said, the architecture and the performance data points presented in this whitepaper can be utilized as the starting point for building DL solutions tailored to varied set of resource requirements. More importantly, all the components of this architecture are linearly scalable and can be expanded to provide DL solutions that can manage 10s to 1000s petabytes of data.

While the solution presented here provides several performance data points and speaks to the effectiveness of PowerScale in handling large scale DL workloads, there are several other operational benefits of persistent data for DL on PowerScale:

• The ability to run AI in-place on data using multi-protocol access.
• Enterprise grade features out-of-box.
• The ability to share data via tiering with other ADAS workloads (ex: HiL/SiL) without copying data to lower-performance, hybrid storage tiers (typical for HiL/SiL)
• Seamlessly tier to higher density nodes for cost-effective archiving while maintaining SLAs

In summary, PowerScale based DL solutions deliver the capacity, flexibility performance, and high concurrency to eliminate the I/O storage bottlenecks for AI. This provides a solid foundation for large scale, enterprise-grade DL solutions with a future proof scale-out architecture that meets your AI needs of today and scales for the future.

Dell.com/support is focused on meeting customer needs with proven services and support.

Storage technical documents and videos provide expertise that helps to ensure customer success on Dell   storage platforms.

Deep Learning with Dell   Isilon

Dell   Isilon and NVIDIA DGX-1 servers for deep learning

Isilon H5600 Storage Tech Specs

Dell   PowerScale: Storage Solution for Autonomous Driving

Dell   PowerScale for ADAS and Autonomous Driving

Top 5 reasons to choose Dell   PowerScale for AD/ADAS

Solving the storage conundrum in ADAS development and validation

Cityscapes Public Dataset

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

NVIDIA SSD300 v1.1 For PyTorch

Uber Horovod

pytorch-deeplab-xception