Computer Vision is a field of Artificial Intelligence (AI) that provides computers and systems with the ability to extract relevant information from digital images, videos, and other digital sensor inputs. It uses algorithms to process data streams from digital devices and extract the information that helps people understand and manage the environments they rely on. Although humans excel at visual tasks, with approximately 60% of the human brain involved in various aspects of visual perception, current AI vision technology can compete with and even surpass human capabilities for many tasks.

Around the world, organizations are using AI enabled computer vision solutions from Dell Technologies to create safer and more efficient public and private spaces. Data scientists have developed various machine learning and deep neural network tools that can identify and label objects using data from different camera types such as optical, thermal, and infrared. Dell Technologies has combined these hardware and software advancements to create a wide range of AI solutions that are currently being used in industrial, commercial, residential, and public sector applications, providing significant value.

We have recently been working with a new and emerging area of AI known as LiDAR-based 3D perception. LiDAR is an acronym of Light Detection and Ranging, a method for determining ranges by targeting an object or a surface with a laser and measuring the time for the reflected light to return to the receiver. Our testing was designed to understand how LiDAR 3D perception can help create a more complete understanding of how people and objects flow and function within public and private spaces. Our key partner in this research is Seoul Robotics, a leader in LiDAR 3D perception and analytics tools.

For this project we choose to use our Dell Validated Design for Computer Vision platform an enterprise-grade engineered platform that has been leveraged for multiple validated designs and has had many successful deployments in the field. The design supports GPU virtualization and runs on Dell VxRail HCI (hyperconverged infrastructure) with Dell PowerEdge servers.

We chose this platform for LiDAR testing to ensure that LiDAR workloads can run parallel with traditional camera stream analytics processing on a flexible engineered platform and validated the computing, networking, and storage requirements for typical use cases in the transportation industry.

The first three chapters of our white paper are intended for a broad audience, including those with both business and technical interests. The entire paper is relevant to technical staff and managers who are currently working with computer vision technologies and those who are researching how this technology can improve situational awareness.

The movement of people and objects within and around the spaces they occupy, as well as the functions they perform while there, create a complex set of interrelated actions and reactions. Traditional computer vision techniques that use standard optical, thermal, infrared, and other types of cameras are not well suited to capture this complex 3D flow and function information for analysis and decision-making. Cameras capture two-dimensional representations of our three-dimensional environments, which can be easily viewed on flat displays. To analyze the flow and function of a space, extracting depth information from a 2D image can be attempted, but the computing cost is high, and the final representation is a relatively poor approximation.

LiDAR technology is capable of collecting depth information from an environment with precise object locations in a full 3D grid map. This technology overcomes a number of limitations of 2D cameras. LiDAR systems can improve the quality of analysis for data collected during dynamic weather conditions like rain, snow, and fog using AI for improved accuracy. Furthermore, they eliminate the need for ambient lighting, critical for optical cameras to function effectively, and are more robust for lighting environments where glare, reflections, or other harsh conditions are present.

The widespread deployment of high-resolution optical cameras has raised concerns regarding the violation of individual privacy and the potential misuse of the data. Therefore, it is essential to consider a replacement or augmentation of such cameras. LiDAR 3D perception is a promising alternative since it does not contain any biometric data that could be cross-referenced with other sources to identify individuals. This allows operators to do anonymous object tracking while maintaining individuals' privacy.

We are currently evaluating the benefits of integrating LiDAR and other rapidly evolving AI technologies into our Dell Technologies AI solutions for computer vision. We recognize the necessity of adapting swiftly to the ever-changing AI technology landscape in order to aid organizations in monitoring and controlling public and private spaces, enabling them to better comprehend and influence the activities taking place within their managed areas.

There are many technology options available for analyzing historical and real-time data that can be gathered from computer vision solutions. When deciding on the technology and data collection requirements, the focus should be on achieving better organizational outcomes for all stakeholders involved in the use and management of physical environments, such as facilities managers, customers, employees, and constituents.

We have worked with numerous organizations that desire to achieve one or more of the following outcomes while deploying a computer vision solution: enhancing operational efficiency, strengthening safety and security, improving sustainability, enhancing the overall people experience in a physical environment, and generating new revenue opportunities.

In the following sections, we will focus on a few high-value sample use cases that are related to transportation of vehicles and pedestrians. By leveraging LiDAR 3D perception, we will illustrate how a better comprehension of aspects such as vehicle and pedestrian detection, traffic flow optimization, incident detection and response, and data analytics for urban planning can help organizations achieve their desired business outcomes in the context of Intelligent Transportation Systems.

LiDAR is a remote sensing technology that measures distances by illuminating a target with laser light and then analyzing the reflected light. Originally developed in the 1960s, LiDAR has matured significantly and now has widespread applications, ranging from geospatial mapping and environmental monitoring to autonomous vehicles and robotics.

A LiDAR sensor device uses a laser to emit a pulsed or continuous beam of light towards an area that is being monitored. The light strikes objects in the environment, such as a surface, particle, or molecule, and is then reflected back to the LiDAR system. By precisely measuring the time that it takes for the light to travel to the target and back (time-of-flight), the system can calculate the distance between the LiDAR sensor and the target.

A typical stationary terrestrial scanning LiDAR system consists of a laser, a scanner, and a sensor where reflected energy is detected and measured . The laser is responsible for emitting the light beams, while the scanner directs the beams at various angles. Advanced systems can emit hundreds of thousands of pulses per second, capturing a dense 'point cloud' that can be used to generate a highly detailed three-dimensional representation of the environment.

LiDAR systems can be classified based on several criteria, including their position relative to the surface of the Earth, the nature of the targets they are designed to measure, and their operational wavelengths.

• Airborne LiDAR is used for applications such as topographic mapping and forest canopy analysis.
• Terrestrial LiDAR is used in industrial automation, robotics, and various forms of spatial analysis including creating detailed city models and simulations in urban planning.
• Mobile LiDAR systems are mounted on moving vehicles and are commonly used for mapping, road surveys, and obstacle detection and navigation in autonomous driving.

Through a better understanding of how existing systems operate, organizations can improve their insights on how to make cities more efficient, reduce emissions, and improve air quality. This document describes how LiDAR can be used to provide high-resolution, three-dimensional information from the context of a highway environment. The real-time, high-resolution data provided by LiDAR systems can be leveraged for multiple outcomes including optimizing traffic flow, enhancing pedestrian safety, and aiding urban planning.

Seoul Robotics is a South Korean company that specializes in providing 3D computer vision software solutions. Established in 2017, the company aims to create software capable of analyzing and interpreting 3D data in real-time, primarily from LiDAR sensors. Their cutting-edge technology has a wide range of applications, including but not limited to autonomous vehicles, smart cities, and industrial automation.

The company offers various products and solutions that can help businesses and organizations analyze 3D sensor data. SENSR is one of their flagship products that is designed to interpret and analyze 3D data from multiple sensor inputs. The software can detect, track, and classify objects in a given environment, making it highly useful in scenarios like traffic management, warehouse automation, and security.

Seoul Robotics specializes in software solutions that use machine learning algorithms for object recognition and tracking. Their software also employs advanced data analytics techniques to interpret 3D data, making it highly adaptable and reliable for various use-cases.

With the growing field of 3D computer vision, Seoul Robotics has positioned itself as a major player, particularly in industries where accurate interpretation of 3D data is crucial. Their focus on creating efficient and scalable software sets them apart and aligns well with current technological trends in automation and data analysis.

SENSR is an advanced deep learning AI technology designed for LiDAR-based 3D perception. It fuses data from multiple LiDARs to create highly accurate 3D models of environments in real time allowing users to gain valuable insights to help improve the conditions of the monitored area.

Using advanced algorithms and deep learning AI, SENSR detects various objects such as vehicles, pedestrians, and bicycles with high precision and reliability. It also provides object trajectory and movement prediction, which is useful for near real-time applications.

One of the unique features of SENSR is its dynamic weather-filtering AI, which ensures unparalleled accuracy in all weather conditions, including rain, snow, and fog.

QuickConnect is an analytics tool that unlocks the information contained within 3D data. It converts SENSR's output into actionable events and generates statistics for intelligent transportation system applications.

QuickConnect analyzes events including:

• Accidents
• Accident Near Misses
• Speeding
• Illegal Stops
• Illegal U-Turns
• Jaywalking
• Pedestrian Crossing
• Traffic Congestion
• Wrong-Way Driving

This event information enables faster and easier integration of real-time and statistical information to make informed decisions and improve safety and mobility.

This solution assessment used a VxRail V670F 5-node cluster to examine the performance, maintainability, and scalability of processing LiDAR data with software tools supplied by Seoul Robotics. VxRail is an HCI appliance jointly engineered by Dell Technologies and VMware. The system deployed in our lab is a V Series VxRail platform using 2U server nodes with NVIDIA GPU support. It is also configured with all flash drives for optimal performance. A full set of VxRail configuration options are available here. The VxRail platform combines compute, storage, and networking components in a single, compact unit to simplify the deployment and management of virtualized workloads.

Assembling and testing these components as an integrated reference platform forms the basis of our Dell Technologies Validated Designs for Computer Vision. The full details are available here.

The functionality provided by the Seoul Robotics SENSR application was hosted and tested on a nonproduction CV platform system deployed in a Dell engineering lab. The SENSR application was configured using a set of virtual machines with GPU support. Consult with all software suppliers and system integrators before implementing a production-grade safety and security system.

SENSR is built with a modular architecture enabling large scale installations with hundreds of LiDARs along with simple setups limited to a single LiDAR. There are two types of SENSR application nodes required for a complete system:

Main node

Oversees the overall processing.

• Merges the perception result for one or more algo nodes,
• Manages user interfaces and data output,

Algo node

Processes LiDAR data for a related set of LiDAR sensors (typically a geographic location).

• The Algo node is the minimal unit configurable with unique parameters.
• Algo nodes are typically allocated according to the characteristics of the environment that you want to observe (areas with many vehicles, areas with only pedestrians, areas with low sensor installation density, and so forth). Overseparation of Algo nodes may, however, cause excessive load on the main node.
• Multiple Algo nodes can be created using a single virtual machine or server.

SENSR operates within a Kubernetes environment to ensure system availability and to facilitate communication between Main and Algo nodes in distributed systems.

The high-level logical architecture tested for this paper using Seoul Robotics SENSR on our CV platform is as follows:

In the above diagram, we deployed a 1 x Main Node that manages the cluster. We also deployed 3 x Algo nodes that process the LiDAR data. Each Algo Node has a GPU assigned. All nodes were virtualized on the CV VxRail platform.

Note that both a SENSR application configured for distributed processing and the underlying VxRail platform both use the terms cluster and node. In the case of the VxRail, we have a VMware cluster that consists of Dell PowerEdge server nodes. For Seoul Robotics SENSR, one Main node together with one or more Algo nodes are deployed withing Kubernetes cluster environment to ensure system availability and to facilitate communication between the Main and Algo nodes in distributed systems.

The solution was validated on a VxRail V670F 5-node ESXi cluster to assess performance, maintainability, and scalability. The team deployed a distributed configuration of the latest SENSR release (v3.1.1).

The hardware specifications per hardware node in the ESXi cluster are as follows:

CPU

2 x Intel(R) Xeon(R) Gold 6354 CPU @ 3.00 GHz (72 vCPU)

Memory

512 GB

Storage (vSAN)

27 TB

GPU

2 x A16 64GB NVIDIA GPU

The total ESXi cluster capacity is therefore as follows:

CPU

724 GHz (360 vCPU)

Memory

2.5 TB

Storage (vSAN)

135 TB

GPU

10 x A16 64GB NVIDIA GPU

The solution leveraged VMware vSAN storage for all the VMs that ran the SENSR processing services. VMware vSAN storage was provisioned with a RAID 1 storage policy to provide the best fault tolerance with the least write latency penalty. RAID 1 protection consumes more disk space than other available RAID options but provides better file access performance. The vSAN datastores configured on the VxRail cluster provided local disk mapping for all the storage needs of cluster-hosted VMs, including the operating system and all other local files.

The Dell Validated Design for Computer Vision is an enterprise-grade engineered platform that has been leveraged for multiple validated designs and has had many successful deployments in the field. The design supports GPU virtualization and runs on Dell VxRail HCI with Dell PowerEdge servers. We chose this platform for LiDAR testing to ensure that LiDAR workloads can run parallel with traditional camera stream analytics processing on a flexible engineered platform.

The main components from the Dell Validated Design for CV that are leveraged are:

VMs

All VMS and CV systems were fully virtualized.

vGPU

Virtualized GPU capability that is required to process analytic camera streams.

vSAN

All VMs are protected with vSAN storage.

vCenter

All system management and operations are performed through VMware vCenter.

This solution leverages Seoul Robotics technology to process LiDAR systems data. Computing and storage resources are supplied by the Dell Technologies CV platform with VMware virtualization to demonstrate how analytics applications can run in parallel with other workloads.

The SENSR system software and versions tested were:

• Ubuntu 20.04 LTS
• SENSR v3.1.1 cluster
  • 1 - Main node
  • 3 - Algo Nodes

It is important to note that LiDAR systems produce much more data than camera systems, which will impact sizing decisions. Our testing investigated processing LiDAR workloads with a CV Platform where the sensor would be mounted in a remote location and communicating over a network to interpret the data in a configuration typical of camera-based solutions. This option fully supports hybrid scenarios where LiDAR and camera systems could be processed in combination to meet a customer outcome.

We recommended assigning a GPU to each Algo node processing sensor data. Processing data on the Algo nodes without a GPU is possible, but we observed poor-quality classification of everyday objects during our testing. For example, cars in our intersection scene were missed and instead classified as unknown when the GPU was disabled.

The Seoul Robotics guide provides the following guidance regarding the use of the new GPU module:

"SENSR allows running Deep Learning algorithms that improve the tracking and classification performance significantly. Running those algorithms is computationally expensive and therefore it is recommended using a GPU module for optimal performance. "

The GPU module was enabled for all tests evaluated during the testing supporting the findings in this paper.

It is very challenging to size universally for LiDAR streams. This is because the following items impact the system resources required:

• The density of points produced by the LiDAR sensor
• The size of the coverage area that requires object detection analytics
• The number of LiDAR sensors positioned to collect data that will cover the scene

We followed the common practice of testing sizing based on how many LiDAR points per second can be processed while monitoring the latency in the system.

For this sizing exercise, the simulated LiDAR sensors were each publishing at a rate of 1.2 million points a second. Up to four sensors were running on an Algo node at any given time to give a maximum of 4.8 million points being processed per second.

After testing different virtual machine configurations in the lab, the final SENSR server roles were deployed with the following specification. This configuration ensured that each system resource utilization was <75% at any given time.

With this configuration, the maximum number of Algo/Main VMs per VxRail node is 8 assuming no other workloads are competing for CPU or GPU resources on that node.

For these types of larger deployments, consult with your Dell representative who will discuss your scenario in more detail before finalizing a system design.

Raw point data captured from a LiDAR sensor is not typically persisted to a storage device. In a typical application environment, raw data is processed directly from the sensor to generate events and then discarded. These events can then be transmitted to a third-party application for management as needed. We followed this pattern during testing so no special consideration was made for storage of point cloud data during this testing.

If raw data must be retained, then reach out to your Dell representative who can recommend the optimal storage approach for your use case.

Network sizing is extremely important for LiDAR data processing and analytics due to the extremely dense point cloud data. For example, a typical high-resolution camera needs about 4 Mb/s bandwidth per stream forwarded to the analytics processing server whereas current LiDAR sensors consume between 25 Mb/s and 50 Mb/s. This challenge may increase in the future as newer LiDAR sensors come to market with higher point cloud density (up to 256 Mb/s) and higher refresh rates driving even higher bandwidth requirements.

Our current CV Platform offering comes configured with 25 Gb/s network cards in each of the nodes. A sample bandwidth sizing estimation calculation is as follows:

Total network capacity = 25Gb/s  
Assume 80% target utilization = 20Gb/s  
Single LiDAR point cloud stream = 50Mb/s  
Total supportable sensors = 400 units/node

Example theoretical limit of the network bandwidth required to run LiDAR workload.

The bandwidth requirement for specific analytics use cases should also consider the network needs of other applications and systems management. This analysis also assumes that a user can deliver 25 Gb/s to each node in the CV platform over their data center hosting network. When planning an implementation, be aware that data processing computing resources will likely be exhausted before the network theoretical limit is reached.

The Dell Technologies Validated Design for Computer Vision platform is engineered to scale horizontally using the VxRail virtualization architecture. As new processing workloads are needed, additional nodes can be added to the system.

During testing we processed 4 LiDAR streams with a combined ~4.8 million points per second of point cloud data using a single Algo processing node VM configured on the VxRail. The VM was given access to one quarter of the resources from an A16 GPU to provide enhanced object detection and higher performance.

Each additional VxRail node added to the cluster can support up to 8 additional Algo/Main VMs. There are many ways to configure the additional capacity to support combinations of Main and Algo nodes. We found that 1 Main node could manage the traffic for 3 Algo nodes processing 12 data streams reliably. This configuration ratio would result in 2 new SENSR clusters (1 x Main, 3 x Algo) that support up to 24 additional LiDAR streams of 1.2 million points per second.

Some questions arose during our LiDAR testing that we are still investigating.

What is the best way to handle future LiDAR developments with increased resolution, such as points per second? This requires a move towards a more federated solution where LiDAR data is processed at the edge nearer to the sensor. Dell ruggedized servers deployed closer to the LiDAR sensor placements are an interesting candidate architecture to handle this type workload.

What is the actual impact of LiDAR traffic on a production network? We used simulated data streamed from streams captured in files in this testing. Testing with live LiDAR sensors in a closer to real world configuration needs investigation in the future efforts.

The use cases validated during this testing focused on three main areas:

• Smart Places use cases
• Security use cases
• Third-party Integration

This test investigated how LiDAR processing performs for an outdoor scenario like a city intersection where weather might be an issue and the scene is extremely dynamic. This is also and a common CV use case where a camera system might perform poorly due to these complications.

We broke the test down into the following sub-scenarios:

• Multiple LiDAR sensors at an intersection
• Objects detection
• LiDAR images or map integration

This test examined scenarios where more than one LiDAR sensor is used to map a scene. In particular, our scenario used data from four sensors that cover a view of each road at an intersection forming a 360-degree view of the scene. These four sensors are fused together to form a highly accurate 3D view of the intersection.

Results

We found it was possible to overlay four different simulated streams using the SENSR application to present a full view of the entire intersection:

The SENSR application also performs object detection on the incoming LiDAR streams during processing. The goal of this test was to assess what objects could be detected along with what level of object detail can be extracted.

Results

We found it useful to sometimes simplify the viewing of objects by hiding the raw LiDAR point cloud and show the objects that were detected in the scene. An example of this is below:

When viewing the raw LiDAR data, it is often difficult to identify shapes in the point cloud data with just the human eye. To improve this, it is possible to upload an image or map corresponding to the monitored scene where the LiDAR data was captured. This approach is typically easier for indoor scenes such as a room or hallway but is also possible for outdoor scenes.

Results

Inserting the map data was not difficult and did provide a much better experience for users that are not familiar with raw LiDAR point data:

This test focuses on using LiDAR data to identify and track an intruder (person) for a security notification event.

The following scenarios were investigated:

• Create events for intrusion detection
• Capture person height and location
• Showcase Tag and Follow capabilities

Intrusion detection can be a particularly important use case. To simulate this scenario, a zone was configured within the scene that people were not allowed enter.

Results

As pedestrians entered the restricted zone events were triggered that a person with a specific nonprivate ID entered the zone.

LiDAR can capture important metadata about an intruder that is typically not available from camera data. The goal of this test was to see the LiDAR specific data that is returned when an intrusion event is detected.

Results

Data is captured for each object in a scene regardless of if the objects enter a restricted zone or not. A sample of the metadata captured is below:

This test shows how an object can be tracked across multiple LiDAR sensors.

Results

The setup involved 2 LiDAR sensors that partially covered an indoor area.

In this scene LiDAR 01 is covering the north portion of the room and LiDAR 02 is covering the south portion of the room.

We then set the project to runtime mode where people tracking is enabled for the scene.

The diagram above shows that a person was identified on the left side by Lidar 01 and assigned an ID of 22. That person traveled to the south side of the room and was detected by Lidar 02, as shown on the right side, with the same ID since it is the same person.

This section investigates testing the options to integrate SENSR with third-party systems such as a Video Management System (VMS). It can often be very useful for users to aggregate all the alerts for a monitored space in a single portal regardless of whether they originate from a camera or LiDAR-based system.

Typically, VMS solutions (such as Genetec or Milestone) provide APIs to capture events from outside systems to their event log so that security agents can monitor events across thousands of cameras/sensors. To enable integration, SENSR provides an SDK that can be used to connect with any system of choice.

Results

We tested the Python SDK from Seoul Robotics during this validation. A simple example of the syntax required to output events to the console looks like:

python3 console_output.py -- address ip_address -- example_type data

This exports event data as objects trigger alerts in the LiDAR scene. A sample view of this is below:

We have conducted extensive tests with the SENSR product deployed on the Dell Technologies Validated Design for Computer Vision platform. This paper describes our investigation of how LiDAR can provide a more complete understanding of how people and objects flow and function within public and private spaces. Our results show that LiDAR has some unique capabilities that are not available with traditional camera systems. We also documented our findings of the resource requirements when running LiDAR workloads on our CV platform. The outcomes are:

• We showed "Tag and Follow" capabilities that track an object (such as a person or car) across multiple different LiDAR sensors.
• The SENSR software suite runs seamlessly in our virtualized environment and processes LiDAR data reliably.
• The solution is scalable and can be expanded as new LiDAR sensors are added to the system.
• SENSR provides an SDK that can be used to integrate event data with external tools (such as a VMS).