ADAS Elevated: Harnessing Early Fusion Techniques for Superior Nighttime Detection

Fusion Techniques

By fusing the sensor data streams from multiple sensors and different sensing modalities, the goal is to achieve better system performance than when using sensors separately. This applies not only to detection accuracy and reliability but also to computational throughput. Fusion techniques have evolved from the classical approach with the introduction of Machine Learning and AI tools. The following techniques discussed in this text are of particular interest for ADAS:

• Late Fusion
  This is the classical approach in which fusion takes place later in the data processing chain of a sensor system. This is often referred to as high-level fusion, which essentially means the same thing.
  
  The individual sensors process data to a higher degree before the fusion takes place, typically at the point where the perception algorithms have classified something as an object. These objects are classified into different semantic classes (e.g., car, pedestrian, cyclist) that are relevant for a traffic environment. Then the objects, with their derived attributes (e.g., position, orientation, velocity), are passed on to the fusion algorithm.
  
  The information can then be used in a tracking algorithm, in which the system computes a path prediction that is used, for example, in collision avoidance features like Autonomous Emergency Braking (AEB).
  
  The benefit of a system using late fusion is that when both sources give information about an object, you can trust the output, which results in high reliability for the system. The drawback is that each sensor modality needs to provide data, and in challenging conditions—for example, if a camera is exposed to sun glare—sometimes there is not enough signal to perform detection.
  
  This technique fits well with an ADAS architecture consisting of smart sensors in which decision-making is decentralized.

• Early Fusion
  For early fusion, there are two variants we consider. One is mid-level, where one of the sensors provides object data and the other provides data from earlier steps in the processing chain (typically pixels from an imager or detections from a radar).
  
  The other is low-level fusion, where all sensors provide data from the early steps, and the fusion process also performs the object classification. The benefit of early fusion is that each sensor does not have to classify the object by itself. In practice, this means that in some cases where there are challenging conditions for the sensor system, it can combine less certain information from each sensing modality into a reliable detection.
  
  This approach has become more interesting now with centralized architecture, where sensors can pass high-density data over a high-bandwidth connection to a central compute unit (without processing data into objects). Overall, this can result in a more efficient perception system, and the development has been enabled by improvements in SOCs and AI tools.

Safeguarding Pedestrians Through Thermal and Imaging Radar Early Fusion

To understand early fusion's potential linked to ADAS, we can look at the need to address pedestrian accidents in darkness. In the U.S., nighttime pedestrian fatalities have risen over the past 15 years, with nearly all of the increase has been derived to urban arterial roads — where 76% of the fatal accidents occur after dark, according to a AAA report. Current camera technology that operates in the visible light spectrum struggles in low-light conditions and the risk for false-positives increases when increasing the sensitivity.

Magna explores early fusion of thermal and imaging radar sensors as a robust, cost-effective alternative with high availability. Thermal sensors detect pedestrians through heat signatures, while imaging radar provides detailed spatial estimates. Combining raw data from both sensors allows real-time object assessment in the dark.

Thermal and imaging radar early fusion can improve pedestrian safety and offer cost and scalability benefits. However, challenges remain for market acceptance. Industry needs to value this technology when it comes to availability and reliability. System solutions with thermal and imaging radar requires high production volumes and market adoption to support cost-effectiveness.

Given the evolution of ADAS technologies and upcoming safety regulations, the industry is prepared to overcome these barriers. ADAS fusion will likely evolve rapidly, enhancing safety and efficiency in mobility solutions.