If autonomous vehicles are to make the correct decision in every conceivable situation, they must be able not only to locate their own position in traffic, but also to reliably capture their environment with precision. To do this, vehicles must have the ability to collect and fuse data from sources such as laser, camera, and radar sensors. For the algorithms that process such sensor data, artificial neural networks have become indispensable tools. However, these networks need fast, efficient, flexible hardware, which is precisely what the “KI-FLEX” project has been successfully researching over the past four years. “This is an important step toward the safe mobility of the future,” says Michael Rothe, who heads the Embedded AI group at Fraunhofer IIS.

“Usually, it takes a trunk full of storage and computers for the autonomous vehicle to reliably recognize its environment. The energy-efficient AI-FLEX hardware is only about DIN A5 in size. This miniaturization of the hardware was accompanied by extensive adjustments to the software: We had to adapt the input data and the structure of the neural networks and reconnect the sensors. The KI-FLEX hardware can now be installed in autonomous vehicles and used for further research activities," summarizes Bernd Schäufele, research associate at Fraunhofer FOKUS.

On top of every traffic situation

Systems must be able to unambiguously detect and identify objects and road users in traffic situations. The importance and utility of sensors differ according to the type of object and environmental conditions. Both the traffic situation and the weather and light conditions have to be taken into account in order to ensure safe autonomous driving. Moreover, the systems must be able to respond flexibly to potential sensor failures or adversarial attacks in their data. To address this requirement, the project partners developed resource-optimized approaches for the early and late fusion of camera data, lidar data, and detected objects along with an AI-based monitoring system. These components allow vehicles to reliably respond to changed situations by adjusting the algorithms used.

Reconfigurable AI system

Artificial neural networks are currently developing at a rapid rate. The growing number of architectures is making increasing demands on the hardware and software. For this reason, “KI-FLEX” employs a heterogeneous hardware architecture made up of FPGA and ASIC AI accelerators in order to implement the neural networks for object detection in camera and lidar data. This flexibly reconfigurable and programmable AI accelerator system anticipates the future to some extent, as the hardware will be able to support emerging neural network designs. Furthermore, the hardware platform’s computing resources can be allocated dynamically according to load.

The AI chip developed in the project also offers considerable advantages with regard to power consumption, processing speed, and cost savings compared to conventional multi-purpose processors (CPUs) or graphics processing units (GPUs).

Germany-wide research initiative

The project “KI-FLEX – Reconfigurable hardware platform for AI-based sensor data processing for autonomous driving,” which launched in September 2019, was funded by the German Federal Ministry of Education and Research (BMBF) within the guidelines on promoting research initiatives in the field of “AI-based electronic solutions for safe autonomous driving (AI element: autonomous driving).”

Led by Fraunhofer IIS, the project consortium comprises several German research and industry partners: Infineon Technologies AG, videantis GmbH, the Technical University of Munich (Chair of Robotics, Artificial Intelligence and Real-Time Systems), the Fraunhofer Institute for Open Communication Systems FOKUS, the Daimler Center for Automotive IT Innovations (DCAITI, Technical University of Berlin) and the Friedrich-Alexander-Universität Erlangen-Nürnberg (Chair of Computer Science 3: Computer Architecture).