Beyond Horizons - The Rise of the Edge AI Processing Paradigm

Beyond Horizons – The Rise of the Edge AI Processing Paradigm RIVER PUBLISHERS SERIES IN COMMUNICATIONS AND NETWORKING Series Editors: ABBAS JAMALIPOUR MARINA RUGGIERI The University of Sydney University of Rome Tor Vergata Australia Italy MARKO JURCEVIC University of Zagreb Croatia The “River Publishers Series in Communications and Networking” is a series of comprehensive academic and professional books which focus on communication and network systems. Topics range from the theory and use of systems involving all terminals, computers, and information processors to wired and wireless networks and network layouts, protocols, architectures, and implementations. Also covered are developments stemming from new market demands in systems, products, and technologies such as personal communications services, multimedia systems, enterprise networks, and optical communications. The series includes research monographs, edited volumes, handbooks and textbooks, providing professionals, researchers, educators, and advanced students in the field with an invaluable insight into the latest research and developments. Topics included in this series include: • Communication theory • Multimedia systems • Network architecture • Optical communications • Personal communication services • Telecoms networks • Wifi network protocols For a list of other books in this series, visit www.riverpublishers.com Beyond Horizons – The Rise of the Edge AI Processing Paradigm Editors Ovidiu Vermesan SINTEF, Norway Marcello Coppola STMicroelectronics, France Fabian Chersi CEA, France River Publishers Published, sold and distributed by: River Publishers Broagervej 10 9260 Gistrup Denmark www.riverpublishers.com ISBN: 978-87-4380-863-3 (Hardback) 978-87-4380-862-6 (Ebook) ©The Editor(s) and The Author(s) 2025. This book is published open access. Open Access This book is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License, CC-BY-NC 4.0 (http://creativecommons.org/ licenses/by/4.0/), which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, a link is provided to the Creative Commons license and any changes made are indicated. The images or other third party material in this book are included in the work’s Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work’s Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt, or reproduce the material. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. Printed on acid-free paper. Dedication “If a machine is expected to be infallible, it cannot also be intelligent.” – Alan Turing “Wisdom is the daughter of experience.” – Leonardo da Vinci “Wonder is the beginning of wisdom.” – Socrates “Education is not the learning of the facts, but the training of the mind to think.” – Albert Einstein Acknowledgement The editors would like to thank all the contributors for their support in the planning and preparation of this book. The recommendations and opinions expressed in the book are those of the editors, authors, and contributors and do not necessarily represent those of any organizations, employers, or companies. Ovidiu Vermesan Marcello Coppola Fabian Chersi Contents Preface xi List of Figures xiii List of Tables xvii List of Contributors xix 1 Advancing Edge AI Perception Platforms and Sensor Fusion for Last-Mile Delivery Autonomous Vehicles 1 Ovidiu Vermesan, Roy Bahr, Hans-Erik Sand, Simen Marentius Saxegaard, Helge Brudeli, Petter Emanuelsson, and Martin Førrisdahl 1.1 Introduction and Background . . . . . . . . . . . . . . . . . 2 1.2 Sensor Fusion in Last-Mile Context . . . . . . . . . . . . . 7 1.3 Autonomous Vehicle Architecture for Last-Mile Delivery . . 15 1.3.1 Localisation and High-Definition Map . . . . . . . . 15 1.3.2 Perception Implementation . . . . . . . . . . . . . . 15 1.3.3 Prediction, Decision-Making, Planning and Route Optimisation ..................... 23 1.3.3.1 Odometry and path planning . . . . . . . 23 1.4 Edge AI Platforms . . . . . . . . . . . . . . . . . . . . . . 26 1.4.1 Robot Operating System . . . . . . . . . . . . . . . 27 1.5 Future Considerations and Research . . . . . . . . . . . . . 31 1.5.1 Deployment Considerations . . . . . . . . . . . . . 31 1.5.2 Future research . . . . . . . . . . . . . . . . . . . . 32 1.6 Conclusion .......................... 34 vii viii Contents 2 AIDGE: A Framework for Deep Neural Network Development, Training and Deployment on the Edge 41 Fabian Chersi, Olivier Bichler, Cyril Moineau, Maxence Naud, Laurent Soutier, Vincent Templier, Thibault Allenet, Inna Kucher, and Vincent Lorrain 2.1 Introduction and Background . . . . . . . . . . . . . . . . . 42 2.1.1 RelatedWork..................... 43 2.2 Our Framework Overview . . . . . . . . . . . . . . . . . . 44 2.2.1 Internal Graph Representation . . . . . . . . . . . . 47 2.2.2 Platform interoperability . . . . . . . . . . . . . . . 48 2.2.3 Graph Regular Expression (GraphRegex) . . . . . . 48 2.2.4 Network optimization . . . . . . . . . . . . . . . . 49 2.2.5 Exportphase ..................... 51 2.3 Conclusion and future work . . . . . . . . . . . . . . . . . . 52 3 A scalable and flexible interconnect-based dataflow architecture for Edge AI Inference 55 Rohit Prasad and Hana Krichene 3.1 Introduction.......................... 56 3.2 RelatedWork ......................... 57 3.3 Background: dataflow execution models . . . . . . . . . . . 57 3.4 Interconnect-based dataflow architecture . . . . . . . . . . . 58 3.4.1 NGC: Neural Global Controller . . . . . . . . . . . 59 3.4.2 NPE: Neural Processing Element . . . . . . . . . . 59 3.4.3 AINoC: Artificial Intelligence Network-on-Chip . . 61 3.4.4 Global Buffers . . . . . . . . . . . . . . . . . . . . 63 3.5 ExecutionModel ....................... 63 3.6 Experiments and Results . . . . . . . . . . . . . . . . . . . 64 3.6.1 Evaluation Methodology . . . . . . . . . . . . . . . 64 3.6.2 FPGA Implementation Results . . . . . . . . . . . . 65 3.6.2.1 Area .................... 65 3.6.2.2 Latency . . . . . . . . . . . . . . . . . . 67 3.6.2.3 Energy consumption . . . . . . . . . . . . 67 3.6.2.4 Energy efficiency . . . . . . . . . . . . . 69 3.7 Conclusion .......................... 70 List of Figures xv Figure 6.2 Left: BIPBIP weeding system behind a robotized tractor. Right: Inside BIPBIP, the camera and the lighting system [2]. . . . . . . . . . . . . . . . . . 102 Figure 6.3 BIPBIP weeding module. The mechanical intra-row hoeing tool is represented by the rod on the left, the computing system in yellow, the two LED panels and the camera in black inside the vision chamber (in gray) which allows to isolate the vision system from changing light conditions [2]. . . . . . . . . . 103 Figure 6.4 Example of annotations on the image database. Maize crops are annotated in blue and the stems in cyan, bean crops in red and the stems in orange[2]....................... 104 Figure 6.5 Schematic representation of the BIPBIP vision system with both hardware accelerator possible: a GPU for the NVIDIA Jetson case or an ASIC for the platform4.1a..................... 105 Figure 6.6 The adapted network architecture used for this application. The figure presents how the duplicated Mobilenet layers and the SSD head are connected to the NeuroCorgi backbone. . . . . . . . . . . . . . 107 Figure 6.7 Results from the Yolo V4 network (left) and the proposed SSD network (right) on maize. Blue rectangles show the plants. Green rectangles show the stem locations. . . . . . . . . . . . . . . . . . . . 108 List of Tables Table 1.1 Sensor Modality Comparison for Last-Mile Delivery AVs........................... 9 Table 1.2 Summary of ROS 2 Features Compared toROS1[11] ..................... 27 Table 3.1 CNNLayerstype ................... 64 Table 3.2 Breakdown of Versal ACAP VCK190 FPGA resources used by the modules of the proposed architecture after synthesis........................ 66 Table 3.3 Different execution phases in the proposed architecture....................... 68 Table 4.1 Accuracy on test set of FL model with 2 clients for multiple learning steps. . . . . . . . . . . . . . . . . 80 Table 4.2 Accuracy on test set of FL model for different number of clients for 2 learning steps. . . . . . . . . . . . . . 81 Table 4.3 Accuracy on test set of FL models for different dataset overlaps for 2 clients. . . . . . . . . . . . . . . . . . 82 Table 4.4 Accuracy on test set of FL models for different dataset overlaps for 10 clients. . . . . . . . . . . . . . . . . . 82 Table 4.5 Accuracy on test set of centralized models for different datasetsizes....................... 84 Table 5.1 KPIs for Measuring Image Artifacts. . . . . . . . . . 92 Table 6.1 Number of images and annotations for each crop. . . 104 Table 6.2 Detection performance (%) and inference speed (fps) for Yolo v4 on the NVIDIA Jetson Xavier including video acquisition and post-processing for each crop. . 106 Table 6.3 Detection performance (loss function) using the new architecture. ...................... 108 xvii List of Contributors Allenet, Thibault, CEA, France Bahr, Roy, SINTEF AS, Norway Bichler, Olivier, CEA, France Bijani, Sepehr, NXP Semiconductors, Germany Brudeli, Helge, Paxster AS, Norway Chersi, Fabian, CEA, France Da Costa, Jean-Pierre, University of Bordeaux, CNRS, Bordeaux Sciences Agro, France Deshayes, Aymeric, University of Bordeaux, CNRS, France Emanuelsson, Petter, Paxster AS, Norway Førrisdahl, Martin, Paxster AS, Norway Germain, Christian, University of Bordeaux, CNRS, Bordeaux Sciences Agro, France Keresztes, Barna, University of Bordeaux, CNRS, Bordeaux Sciences Agro, France Krichene, Hana, Université Paris-Saclay, CEA-List, France Kucher, Inna, CEA, France Lorrain, Vincent, CEA, France Moineau, Cyril, CEA, France Naud, Maxence, CEA, France Prasad, Rohit Université Paris-Saclay, CEA-List, France Sand, Hans-Erik, NxTECH AS, Norway Saxegaard, Simen Marentius, NxTECH AS, Norway xix xx List of Contributors Serpanos, Dimitrios, University of Patras, Greece Soutier, Laurent, CEA, France Templier, Vincent, CEA, France Vermesan, Ovidiu, SINTEF AS, Norway Xenos, Georgios, University of Patras, Greece 1 Advancing Edge AI Perception Platforms and Sensor Fusion for Last-Mile Delivery Autonomous Vehicles Ovidiu Vermesan1, Roy Bahr1, Hans-Erik Sand2, Simen Marentius Saxegaard2, Helge Brudeli3, Petter Emanuelsson3, and Martin Førrisdahl3 1SINTEF AS, Norway 2NxTECH AS, Norway 3Paxster AS, Norway Abstract In the rapidly evolving landscape of transportation, mobility, and logistics, the last-mile represents the final and crucial leg of the delivery journey. It involves goods travelling from a transportation hub to the ultimate destination. This is the most expensive and time-sensitive part of the supply chain business model. Challenges include navigating dense urban environments with various types of traffic participants (e.g., pedestrians, bicycles, animals, electric scooters, motorcycles, etc.), dealing with traffic congestion, locating specific delivery points, managing a high density of stops, and handling failed delivery attempts. In this context, the intersection of edge artificial intelligence (AI), autonomous systems, robotics, and sensor fusion in perception and navigation advances the development of last-mile delivery autonomous vehicle (AV) platforms that evolve towards software-defined and AI-defined vehicles (SDVs and ADVs). The advancements include multiple sensor systems for perception and communication (e.g., ultrasound, inertial, LiDAR, radar, camera, V2X, etc.), real-time data processing for localisation, and robust algorithms for navigation and interaction with diverse traffic environments. This chapter presents the concept and the implementation of an AI-based 1 2Advancing Edge AI Perception Platforms and Sensor Fusion perception and sensor fusion platform technical solution for autonomous lastmile delivery in controlled traffic environments. Keywords: edge AI, perception, autonomous vehicle, sensor fusion, object recognition, last-mile delivery. 1.1 Introduction and Background The future of mobility is intelligent, electrical, autonomous, connected, and shared, affecting all three broad types of mobility: personal mobility (moving individuals or small groups of people), mass transit (moving large numbers of people), and the movement of goods. The last-mile of logistics refers to the final leg of the delivery journey, typically from a local distribution hub or retail centre to the end recipient’s location, such as a home or business [1]. This segment is notoriously the most complex, inefficient, and expensive part of the entire supply chain, often accounting for over 50% of total delivery costs [3]. Last-mile logistics are increasingly automated, and companies that are prepared for this shift are in a stronger position to compete and take the lead. Autonomous last-mile delivery, utilising vehicles ranging from small sidewalk robots to automated vans, promises significant efficiency gains and cost reductions in logistics. As a result, various types of autonomous vehicles for last-mile delivery have emerged as follows [10] and illustrated in Figure 1.1: • Pedestrian sidewalk vehicles. These are slow vehicles designed to travel at a pedestrian speed of 4-6 km per hour. This low speed offers improved safety and allows the operators to control the vehicle in an emergency. • Bicycle sidewalk vehicles. These are vehicles designed to travel up to a bicycle speed of 12-15 km per hour. Figure 1.1 Types of vehicles for last-mile delivery. 1.1 Introduction and Background 3 • On-road delivery vehicles. These vehicles are built for on-road delivery at up to 45-50 km per hour. Their software algorithms and sensor systems resemble those of autonomous vehicles. Driverless technology users utilise autonomous deliveries for several purposes: • Delivery of goods from warehouses to stores and outlets for restocking inventory and shelves. • Delivery of goods from stores to end consumers. • Delivery of goods and parts between the warehouses and production facilities. Autonomous vehicles (AVs), encompassing road-going vans and smaller sidewalk autonomous delivery vehicles, are emerging as a potentially transformative solution. By eliminating the need for a human driver, AVs offer the potential for 24/7 operation, reduced labour costs (a significant component of last-mile expense), optimised routing, and potentially lower emissions, primarily if electric. They can navigate narrow streets or pedestrian zones inaccessible to larger vehicles and improve delivery times by avoiding human-related delays. The development of last-mile delivery autonomous vehicle fleets is linked to the evolution of Internet of Robotic Things (IoRT) platforms. IoRT serves as the technological backbone, integrating individual autonomous robotic vehicles into an interconnected system of systems. IoRT combines IoT technologies with robotics, edge computing and AI, allowing for the coordination of large-scale fleets of delivery robots that might otherwise operate alone. Connecting and integrating the last-mile delivery autonomous vehicle into fleets that are coordinated using distributed networks or IoRT platforms enables functions such as remote monitoring, intelligent communication, and the management of the entire delivery process, which can be managed from the distribution centre to the customer’s doorstep. The intelligence and real-time responsiveness of the autonomous delivery fleets and the IoRT platforms are significantly enhanced by edge AI. Edge AI embeds data processing and decision-making capabilities directly onto the vehicles themselves, strengthening the processing capabilities and the analytics of each vehicle in the fleet or the IoRT platform. The use of edge AI enables continuous route optimisation, obstacles and traffic participants avoidance, and real-time adaptation to changing environmental conditions, which are critical for navigating pedestrian spaces and complex urban environments safely and efficiently. Edge AI-powered robotics within the IoRT framework 4Advancing Edge AI Perception Platforms and Sensor Fusion calculate the most efficient paths in real-time, ensuring that last-mile delivery is not only automated but also intelligent, fast, and reliable [5, 6]. For autonomous last-mile delivery to become a reality, the core enabling technology is robust perception – the AV’s ability to sense, interpret, and understand its complex and dynamic surroundings. Last-mile environments, whether sidewalks or urban streets, present unique perception challenges: close-quarters manoeuvring around pedestrians, cyclists, pets, parked cars, street furniture, and unpredictable obstacles; navigating varied terrain including curbs and uneven surfaces; interpreting complex traffic signals and signs at intersections; precisely identifying the final delivery location (e.g., a specific doorway or porch); managing a high density of stops, and handling failed delivery attempts [2]. Failures in perception can lead directly to collisions, incorrect deliveries, or mission failure. No single sensor can reliably capture all necessary environmental information under all conditions. Cameras struggle in poor lighting or weather, LiDAR can be expensive and has limitations in adverse weather, radar has lower resolution, and ultrasound has a very short range. Therefore, sensor fusion – the intelligent combination of data from multiple, diverse sensors is key [14]. By integrating complementary data streams, sensor fusion aims to create a unified, comprehensive, and reliable environmental model that is more accurate and robust than what could be achieved with individual sensors alone. Last-mile delivery autonomous vehicles can operate in fleets with individual vehicles acting as cognitive agents using perception modules to process images, GNSS positions or LiDAR scans for autonomous system decisionmaking, resulting in actions, such as actuator commands or V2X messages. The high degree of interdependencies between many functional components of autonomous vehicles requires the implementation of new system architectures and new underlying software frameworks. The concepts of developing AI-based last-mile autonomous delivery vehicles are embedding Robot Operating System (ROS) into compact, scalable, AI-based perception, localisation and sensor fusion platforms advancing the solutions and applications for autonomous transport of goods. An essential aspect of the safe use of last-mile delivery autonomous vehicle technology is determining its capabilities and limitations and communicating these to end users, leading to a state of “informed safety”. The first stage in establishing the capability of an autonomous vehicle is defining its Operational Design Domain (ODD). The ODD is defined in [17] as the 1.2 Sensor Fusion in Last-Mile Context 11 particularly on vehicles operating over bumpy terrain, such as sidewalks. LiDAR is crucial for precise localisation and mapping within complex urban canyons or sidewalk environments where GPS may be unreliable [21]. It excels at detecting low-lying obstacles, curbs, potholes, or changes in terrain that might be missed by cameras alone. The high cost remains a significant challenge for the last-mile business case [4]. The typically lower speeds and shorter operational ranges in last-mile delivery may allow for the use of lower-cost, shorter-range LiDAR sensors compared to those needed for high-speed highway autonomy [14]. Fusion with cameras is essential to add semantic understanding to LiDAR’s geometric data. Radar sensors [19] are one of the key elements for the autonomous vehicle’s perception system due to their resilience to adverse environmental conditions. Radar can see through darkness and fog, and to a certain extent through rain, and snow, conditions that severely challenge or blind other sensors, such as cameras. This capability ensures a baseline of operational safety and functionality, regardless of the time of day or weather conditions, providing data on the range, velocity, and angle of other objects with a high degree of accuracy. The synergistic integration of high-frequency radar, 5G communication, and multimodal radar technologies greatly enhances the sensing capability and environmental adaptability of autonomous vehicle perception systems [20]. However, radars present a few challenges. In heavy rainfalls, the radio signals can suffer from attenuation, slightly reducing their effective range. Additionally, in dense urban environments, the radio waves can bounce off multiple surfaces before returning to the sensor. This multi-path reflection, or clutter, can create "ghost" targets, misleading the vehicle’s perception system into “thinking” an object is present where there is none. The resolution of radar makes it challenging to classify objects with certainty, as it for examples struggles to distinguish between a pedestrian, a cyclist, or a stationary object, such as a signpost, based on its signature alone. These challenges are particularly amplified in the context of last-mile delivery for autonomous vehicles. The ODD for these vehicles involves navigating complex and cluttered environments such as residential streets, sidewalks, and loading zones. Standard automotive radars are optimised for detecting large metallic objects, such as other vehicles, and may fail to reliably detect smaller, low-profile, or non-metallic items in these areas, including delivery packages, curbs, children’s toys, or pets. The proximity to buildings, parked vehicles, and other street furniture exacerbates the multipath reflection problem, making it more challenging to maintain a clear and accurate perception of the immediate surroundings. 12 Advancing Edge AI Perception Platforms and Sensor Fusion IMUs measure the vehicle’s linear acceleration and angular velocity using accelerometers and gyroscopes [16]. This data is integrated over time to estimate changes in velocity, position, and orientation (roll, pitch, yaw). They are fundamental for state estimation and enable dead reckoning navigation during periods when external positioning signals, such as GPS, are unavailable [16]. IMUs provide high-frequency motion data (often 100 Hz or higher), completely independent of external signals or environmental conditions, allowing continuous operation in tunnels, urban canyons, dense foliage, or indoors [16]. They are relatively low-cost, especially Micro-Electro-Mechanical Systems (MEMS) based units, compact, and consume little power [16]. IMU data is critical for stabilising perception data from other sensors (compensating for vehicle motion) and for providing the motion inputs needed for sensor fusion algorithms, such as Kalman filters [24]. The primary limitation of IMUs is drift, minor errors in acceleration and angular velocity measurements accumulate over time, leading to rapidly increasing errors in the estimated position and orientation [16]. This necessitates frequent corrections using absolute positioning sensors (such as GPS) or relative positioning derived from other sensors (e.g., LiDAR/camera-based SLAM). IMUs are sensitive to temperature changes and vibrations, which can affect their accuracy [16]. Accurate calibration is crucial, but it can be complex [16]. Magnetometers, sometimes included for heading reference, are unreliable in urban environments due to magnetic interference from buildings, vehicles, and infrastructure [25]. IMUs are indispensable for last-mile navigation due to frequent GPS signal degradation or loss in urban canyons, underpasses, or near tall buildings [16]. The high-frequency data helps maintain a smooth estimate of the vehicle’s state, which is crucial for controlling robots navigating potentially uneven sidewalks or making frequent stops and starts. Cost-effective MEMS IMUs are generally sufficient, but robust fusion with GPS, LiDAR-SLAM, or visual odometry is essential to bind the inherent drift [16]. Ultrasonic sensors use high-frequency sound waves to detect the presence and distance of objects at very short ranges [26]. They operate on the principle of measuring the time-of-flight of emitted sound pulses reflecting off nearby objects. They are relatively inexpensive and easy to integrate. They can detect objects close to the vehicle (within a few meters), effectively covering blind spots often missed by cameras or LiDAR [23]. Their performance is largely unaffected by lighting conditions (work in darkness) or the colour/transparency of the object [27]. They are relatively robust in some adverse weather conditions [27]. Ultrasound sensors have a minimal detection range, of up to 4-5 meters depending on the sensor and conditions [26]. Their 1.2 Sensor Fusion in Last-Mile Context 13 angular resolution is poor due to broad beam patterns, making it difficult to distinguish between closely spaced objects, determine object shape, or precisely locate small objects [27]. Performance degrades significantly at higher vehicle speeds [23]. They can be susceptible to interference from external ultrasonic noise sources [28]. They may struggle to detect soft, sound-absorbing materials [27]. Their primary utility in last-mile delivery is for low-speed, close-quarters manoeuvring, such as parking assistance for vans, docking at a specific delivery point, navigating very narrow passages, or detecting immediate low-lying obstacles like curbs right next to the vehicle or robot [26]. They can serve as a safety sensor for detecting the presence of people near loading doors [29]. Due to their limited range and resolution, they are unsuitable for primary navigation or obstacle avoidance at typical operational speeds but serve as a valuable, cost-effective complementary sensor for near-field safety and precision manoeuvring. V2X encompasses technologies (primarily DSRC/IEEE 802.11p and CV2X/cellular) that enable vehicles to communicate wirelessly with other vehicles (V2V), roadside infrastructure (V2I), pedestrians (V2P, often via smartphones), and the network or cloud (V2N) [30]. Its key role in perception is enabling cooperative perception, where sensor data and derived information are shared among connected entities [30]. V2X can dramatically extend a vehicle’s perception range and awareness beyond the line-of-sight limitations of its onboard sensors [30]. By sharing data (raw sensor data, processed object lists, or intent information), vehicles can “see” around corners or through obstructions via the sensors of other connected agents. V2I communication can provide critical information, such as traffic signal phase and timing (SPaT), road hazard warnings, and work zone alerts [30]. This enhanced situational awareness can significantly improve safety and traffic efficiency, enabling coordinated manoeuvres such as platooning [30]. C-V2X offers the potential advantage of leveraging existing cellular infrastructure for V2N, potentially providing more exhaustive coverage compared to DSRC’s reliance on dedicated Roadside Units (RSUs) [31]. The effectiveness of V2X, particularly V2V and V2I for cooperative perception, heavily depends on widespread adoption and deployment – a significant network effect challenge. Communication channels have limitations in terms of latency, reliability, bandwidth, and range, which can affect the timeliness and quality of shared perception data. Ensuring the security and authenticity of V2X messages is paramount to prevent malicious attacks (e.g., false hazard warnings, Sybil attacks) [30]. Privacy concerns exist regarding the sharing of vehicle data. Standardisation is still evolving, with ongoing debate and regional 14 Advancing Edge AI Perception Platforms and Sensor Fusion differences between DSRC (IEEE 802.11p/ITS-G5) and C-V2X (LTE-V2X, 5G-V2X) hindering global interoperability [31]. Deploying the necessary infrastructure (RSUs for DSRC, potentially upgraded cellular networks for C-V2X) involves significant cost and effort. Cooperative perception via V2X is potentially very valuable in dense, occluded urban environments typical of last-mile routes, allowing a delivery robot or van to perceive pedestrians or vehicles hidden from its direct view [30]. V2I communication providing traffic light status is crucial for safe intersection negotiation. V2N connectivity can be used for real-time updates to delivery routes, receiving customer instructions, remote monitoring, or potentially teleoperation under challenging situations. However, reliable connectivity (cellular or RSU coverage) might be inconsistent across all delivery zones, including dense urban areas or more remote suburban neighbourhoods. Security is especially critical for autonomous delivery vehicles, as it prevents theft, hijacking, or disruption of service. GNSS plays a key role in autonomous last-mile delivery vehicles by providing essential positioning, navigation, and timing information. It enables these vehicles to accurately determine their location and navigate to delivery destinations, facilitating efficient route optimisation and real-time tracking. One of the primary strengths of GNSS is its global coverage, enabling positioning data to be available virtually anywhere. Its high accuracy, especially when complemented by augmentation systems like Real-Time Kinematic (RTK), can achieve centimetre-level precision, which is essential for operating in complex urban environments. Additionally, GNSS provides real-time data updates that support continuous adjustments during deliveries, making it a cost-effective solution widely available for implementation. GNSS has limitations as signal interference can occur in urban environments, where buildings or tunnels obstruct satellite signals, leading to degraded performance. Multipath effects, where signals bounce off surfaces, can further compromise accuracy. Latency issues may arise, affecting the system’s responsiveness in dynamic traffic situations, and extreme weather conditions or satellite outages may challenge the reliability of GNSS. To effectively utilise GNSS in last-mile delivery vehicles, specific requirements must be met. Integrating GNSS with other technologies, such as inertial navigation systems (INS), LiDAR, and cameras, is vital for enhancing accuracy and reliability. Robust software algorithms are needed to process GNSS data and compensate for environmental errors. Energy-efficient solutions are essential to ensure continuous operation without overburdening the vehicle’s power resources. 1.3 Autonomous Vehicle Architecture for Last-Mile Delivery 15 Real-time data exchange must be established to optimise routes and ensure safety while also adhering to safety standards compliance. 1.3 Autonomous Vehicle Architecture for Last-Mile Delivery Autonomous vehicle for transport of goods considers the use of scalable processing capabilities at the edge with AI-based functions implemented into the perception domain and covering the edge computing capabilities implemented into vehicles of different sizes using the same generic architecture as illustrated in Figure 1.4 [7]. 1.3.1 Localisation and High-Definition Map Localisation is critical for the safe and efficient operation of last-mile delivery autonomous vehicles. By employing high-definition (HD) maps that detail urban infrastructure, such as road types, curb locations, and traffic signals, these vehicles can navigate complex environments more effectively. The maps incorporate real-time data updates to reflect changing road conditions, enhancing navigational accuracy and safety. 1.3.2 Perception Implementation Perception in last-mile delivery autonomous vehicles integrates AI to identify and classify various objects within the vehicle’s vicinity. This involves Figure 1.4 Vehicles and scalability. Source: [7]. 16 Advancing Edge AI Perception Platforms and Sensor Fusion Figure 1.5 Perception and sensors fusion. Source: [8]. a fusion of data from multiple sources, where machine learning (ML) algorithms help predict potential obstacles and dynamic changes in the environment. The resultant data enhances situational awareness and informs decisionmaking processes. An overview of the sensors used in the perception and sensor fusion platforms for autonomous vehicles for last-mile delivery of goods is illustrated in Figure 1.5. Sensor fusion, AI processing and decision-making The overall system architecture of the autonomous vehicle comprises controllers for the perception, sensor fusion and actions for the vehicle’s actuators based on the sensed environment, objectives, and constraints. It is divided into three primary blocks: detection, perception, and decision policy, as illustrated in Figure 1.6 [9]. The main system management components consist of the Operating System (OS) based on Linux Ubuntu, and the middleware based on the ROS. ROS1 is a high-level API for evaluating sensor data and controlling actuators. The integration activities on sensor fusion, combines homogeneous and heterogeneous data from different sources like the perception sensors (LiDAR, cameras, ultrasonic sensors, etc.) to facilitate AI processing, decision-making, and planning. The perception workflow for image recognition, object detection and tracking are illustrated in Figure 1.7 [9]. Technology wrappers are used to integrate different protocols, data formats, and interfaces seamlessly. 1.3 Autonomous Vehicle Architecture for Last-Mile Delivery 17 AI processing and the autonomous systems are integrated using perception sensors for mapping the environment, HW/SW components for the acquisition, processing, aggregation, analysis, and interpretation of data, AIbased algorithms and methods for situation assessment, action planning, cognitive decision-making, and actuators for acting on the steering, braking and propulsion systems. Various AI frameworks, such as Python, PyTorch, Keras, and TensorFlow, as well as several machine vision libraries like OpenCV, SimpleCV, the Point Figure 1.6 Autonomous vehicle for last-mile delivery – Platform integration components. Source: Adapted from [9]. Figure 1.7 Perception workflow for image recognition, object detection and tracking. Source: Adapted from [9]. 18 Advancing Edge AI Perception Platforms and Sensor Fusion Cloud library, and YOLO, and AI-based computing platforms like NVIDIA Jetson AGX Orin, were evaluated for integration into different layers of autonomous vehicle architecture. These AI-based frameworks, algorithms, libraries, and platforms were utilised during various phases of the autonomous vehicle platform demonstrator’s development. The decision-making relies on sensor fusion and AI processing. Furthermore, the vehicle platform features several actuators and control units, including the electric steering servo and throttle/speed control for autonomous driving, speech information/recognition for use-case service/security purposes, NFC/mobile locking/unlocking systems, and wireless/wired emergency stops for safety reasons. Perception sensors and navigation The platform was integrated with the NVIDIA Jetson AGC Orin, which provides AI-based perception and sensor fusion capabilities [9]. To abstract the sensor brand and interface from Orin, parsing of the ultrasonic sensor electronic control unit (ECU) data were implemented and an interface provided (independent of ultrasonic system used). Having the vehicle control unit (VCU) interpret the sensor data also enables it to have an emergency brake function. This function is set so that if the vehicle is autonomous mode, the vehicle’s VCU sends a signal to disable drive to the CAN relay, which in turn triggers engaging of the electronic park brake. To get the ultrasonic sensors to have an impact on the vehicle motion there needs to be several interfaces defined where the data and information can flow. There are two distinct types of interfaces in form of CAN and ROS topics. An illustration of the interfaces is given in Figure 1.8. The ultrasonic sensor hardware abstraction layer (HAL) provides an interface that includes the CAN output from the VCU interface but provides it as an ROS topic that other nodes inside NVIDIA Jetson AGX Orin platform can make use of. The topic is described in a message and provides information for each sensor in a separate variable. Tracking System for Platooning Platooning of autonomous vehicles refers to a formation of multiple autonomous vehicles travelling closely together in a single-file line, with the lead vehicle controlling the speed and direction for the following vehicles. The group of connected autonomous vehicles exchange information, allowing 1.3 Autonomous Vehicle Architecture for Last-Mile Delivery 19 Figure 1.8 System overview. Source: Adapted from [9]. them to drive in a coordinated manner, with very small spacings, while still travelling safely at relatively high speeds [36]. The fleets of last-mile delivery autonomous vehicles utilise platooning coordinated driving style to enhance logistics efficiency, increase road capacity, improve traffic flow, reduce delivery times, and load goods from common warehouse hubs. Platooning requires the development of robust and reliable V2V communication, multi-sensor perception systems, control algorithms, and infrastructure, including dedicated lanes or specific road configurations, to function optimally. Information on the vehicles’ speeds, positions, accelerations, decelerations, and other relevant data for the vehicles in the platoon, as well as for those joining or leaving the platoon, is crucial, as all the vehicles in the platoon need to react efficiently and safely in real-time. Several information flow topologies (IFTs) have been traditionally used in the literature, such as predecessor-following (PF), two-predecessor-following (TPF), and bidirectional (BDL). The advancement of communication systems increased the use of more general schemes such as r-predecessor following (rPLF). The dynamic platoon nature, with vehicles changing their relative position over time, also adds complexity to the topology of communications [36]. The platooning style of driving can be implemented for autonomous vehicles equipped with V2X connectivity by conveying traffic information (e.g., 20 Advancing Edge AI Perception Platforms and Sensor Fusion GNSS, speed, or signal timing) and using either unlicensed V2X (ITS-G5) or cellular V2X (LTE-V2X/NR-V2X) [37]. The platooning of last-mile delivery autonomous vehicles can be implemented using the vehicle’s perception sensors (e.g., cameras, ultrasound) for areas with good visibility, and an alternative solution can complement the V2X system. The following section presents the implementation of a tracking system for platooning, as demonstrated in the ECSEL JU AI4CSM project [9], utilising an NVIDIA Jetson AGX Orin processing platform running the Ubuntu Linux operating system, which offers AI capabilities for the vehicle’s perception domain. The vehicle’s onboard unit communicates with its sensors through the ROS operating system as middleware. The autonomous vehicle, illustrated in the Figure 1.15, is equipped with multiple perception sensors, including LiDAR, a depth camera, and ultrasound sensors. For the implementation, the Intel RealSense D455 RGB-D depth camera was used as a sensor to detect the logo mark placed on the rear of the lead vehicle, serving as a target for the follower vehicle to follow. The logo in Figure 1.9 is attached to the rear of the vehicle. The logo design can make it difficult to distinguish it from other circular objects or signs with straight lines within the circle, such as no stopping/parking signs. To detect the logo, two different detection model frameworks were tested, YOLOv5 [38] and YOLOv8 [39]. YOLO is a computer vision model developed by Ultralytics and is part of the "You Only Look Once" (YOLO) family of models, known for their high inference speed, making them suitable for Figure 1.9 Tracking system logo and no stopping/parking sign. 1.4 Edge AI Platforms 27 1.4.1 Robot Operating System ROS is not a traditional operating system in the sense of Windows or Linux. Instead, it is a flexible framework of software libraries and tools that simplify the creation of complex robot applications [45, 46]. ROS is an open-source framework for writing robot software, providing a collection of tools, libraries, and conventions that aim to simplify the task of creating complex and robust robot behaviour across a wide variety of robotic platforms. It has also been recently used in autonomous vehicles, as seen in the case of this implementation for last-mile delivery autonomous vehicles and other autonomous systems. ROS is rather a middleware, a set of software frameworks for robot software development [35, 40, 47, 48]. There are two versions of ROS: ROS 1, which evolved with community contributions, and ROS 2, released in 2017. ROS 2 incorporates real-time capabilities, improved security, and better support for distributed systems by leveraging the Data Distribution Service (DDS) standard [45]. A table of key differences between ROS 1 and ROS 2 can be seen in Table 1.2 [11]. ROS provides services expected from an operating system, including hardware abstraction, low-level device control, implementation of commonly Table 1.2 Summary of ROS 2 Features Compared to ROS 1 [11] Category ROS 1 ROS 2 Network Transport Tailored protocol built on TCP/UDP Existing standard (DDS), with abstraction supporting addition of others Network Architecture Central name server (roscore) Peer-to-peer discovery Platform Support Linux Linux, Windows, macOS Client Libraries Written independently in each language Sharing a common underlying C library (rcl) Node vs. Process Single node per process Multiple nodes per process Threading Model Callback queues and handlers Swappable executor Node State Management None Lifecycle nodes Embedded Systems The ROSSerial client library used for small, embedded devices The micro-ROS stack integrates microcontrollers with standard ROS 2 Parameter Access Auxiliary protocol built on XMLRPC Implemented using service calls Parameter Types Type inferred when assigned Type declared and enforced 28 Advancing Edge AI Perception Platforms and Sensor Fusion used functionality, message passing between processes, and package management. The core of ROS is its anonymous publish/ subscribe messaging system. A process (called a “node”) that has information to share can publish it to a specific “topic”. Other nodes interested in that type of information can subscribe to the topic to receive the messages, which creates a modular, decoupled architecture where different parts of the system can be developed and tested independently [42, 43] . The integration of ROS into autonomous vehicles involves several key aspects, such as [41]: •Hardware abstraction, where ROS provides a standardised interface to a wide variety of sensors and actuators, meaning that a high-level autonomous driving algorithm can be developed independently of the specific hardware being used. In this context, a “LiDAR driver” node could publish data from a particular brand of LiDAR to a standardised topic, and a perception node could subscribe to that topic without needing to know the specifics of the LiDAR hardware. •Inter-process communication: Autonomous vehicles have a multitude of processes running concurrently: perception, localisation, planning, and control. ROS’s messaging system allows these processes to communicate with each other in a reliable and time-synchronised manner, even if they are running on different computers within the vehicle. •Ecosystem and tools: ROS has an ecosystem of tools for visualisation, simulation, and data logging. Tools like RViz enable developers to visualise sensor data and the vehicle’s state in 3D, while Gazebo provides a realistic simulation environment for testing algorithms without requiring a physical vehicle. ROS is used as a platform for developing perception and sensor fusion systems [49] in the implementation of the last-mile delivery autonomous vehicle presented in this chapter. As a result, several features of the platform were analysed, evaluated and integrated into the vehicle architecture. These elements are described below: •AI integration: The modular nature of ROS makes it easy to integrate AI and machine learning libraries. The typical approach used was to have a ROS node that utilises a library such as TensorFlow or PyTorch to perform object detection or semantic segmentation on camera images. The results of this process (e.g., the locations of other vehicles and pedestrians) were then published to a ROS topic for other nodes to use. 1.4 Edge AI Platforms 29 •Sensor fusion: The implemented autonomous vehicles rely on a variety of sensors, including cameras, LiDAR, ultrasound, IMUs, etc. ROS provided a framework for fusing the data from these different sensors to create a more accurate and robust understanding of the environment. A “sensor fusion” node could subscribe to topics containing data from the camera, LiDAR, and IMU, and then use a filter (like a Kalman filter) to combine this data and produce a unified representation of the environment. The original implementation of the functions for the last-mile delivery autonomous vehicles was implemented in the original version of ROS (ROS 1). The test results show that the system has some limitations in terms of real-time performance and security, and the newer version, ROS 2 [44, 45], is integrated into the new vehicle design due to the following [35]: •Real-time capabilities: ROS 2 is built on top of the DDS standard, which provides real-time, reliable, and scalable communication. The capabilities are necessary for autonomous vehicle implementation to reduce delays and increase the system’s robustness. •Security: ROS 2 includes a better security framework that provides features like authentication, encryption, and access control, which are essential for protecting the vehicle from cyberattacks. •Quality of Service (QoS): ROS 2 allows specifying QoS policies for each publisher and subscriber, which enables the control of aspects like reliability, durability, and latency, ensuring that critical data is delivered in a timely and reliable manner. A set of principles and specific requirements guides the design of ROS 2, including distribution, abstraction, asynchrony, and modularity, as well as several design requirements such as security, integration of embedded systems, use of diverse communication networks, real-time computing, and product readiness. The ROS 2 APIs provide access to communication patterns, such as services and actions, which are organised under the concept of a node. ROS 2 also provides APIs for parameters, timers, launch, and other auxiliary tools, which can be used to design a robotic system. ROS 2 issues a request-response style pattern, known as services. Request-response communication provides a clear association between a request and its corresponding response, which can be helpful when ensuring that a task was completed or received. A unique communication pattern of ROS 2 is the action. Actions are goal-oriented and asynchronous, providing communication interfaces 30 Advancing Edge AI Perception Platforms and Sensor Fusion with request-response capabilities, periodic feedback, and the ability to be cancelled. The middleware architecture of ROS 2 consists of several abstraction layers distributed across many decoupled packages. These abstraction layers enable multiple solutions for the required functionality, such as various middleware or logging solutions. Additionally, the distribution across various packages allows users to replace components or take only the necessary pieces of the system, which may be important for certification [11, 12]. Figure 1.14 displays the layers within ROS 2 as it is a set of software libraries and tools for building robot and autonomous systems applications. ROS2 builds upon DDS and contains a DDS abstraction layer. Users do not need to be aware of the DDS APIs due to this abstraction layer. This layer enables ROS2 to have high-level configurations and optimises the utilisation of DDS. Additionally, due to the use of DDS, ROS2 does not require a master process [7, 11, 12]. The client libraries provide access to the core communication APIs. They are tailored to each programming language to make them more idiomatic and take advantage of language-specific features. Communication is agnostic to how the system is distributed across compute resources, whether they are in the same process, a different process, or even a different processing unit. A Figure 1.14 ROS 2 architecture [7, 12]. 1.5 Future Considerations and Research 31 user may distribute their application across multiple machines and processes, and even leverage cloud compute resources, with minimal changes to the source code. ROS 2 can connect to cloud and edge resources over the internet. The client libraries rely on an intermediate interface that provides standard functionality to each client library. This library is written in C and is used by all the client libraries, although it is not required for their operation. The middleware abstraction layer, called RMW (ROS Middleware), provides the essential communication interfaces. The vendors for each middleware implement the RMW interface and are made interchangeable without code changes. Users may choose different RMW implementations, and thereby different middleware technologies, based on various constraints such as performance, software licensing, or supported platforms. The network interfaces (e.g. topics, services, actions) are defined, and ROS 2 defines these types using specific format files. Communications are agnostic to the location of endpoints within machines and processes. Nodes written as components can be allocated to any process as a configuration, allowing multiple nodes to be configured to share a process, thereby conserving system resources or reducing latency. 1.5 Future Considerations and Research 1.5.1 Deployment Considerations Autonomous vehicles for last-mile delivery can reshape the final step of the supply chain, from the distribution centre to customers’ doorsteps. The primary advantage of autonomous delivery is the potential for significant cost reduction and increased efficiency. These vehicles can operate around the clock, resulting in faster delivery times and improved fleet utilisation. They can also be designed to be more environmentally friendly, contributing to more sustainable logistics practices. The deployment of unmanned ground and aerial autonomous vehicles and mobile robots requires careful planning and consideration of various factors such as regulatory compliance, integration with existing logistics infrastructure, and public perception. Addressing these factors is essential for the successful adoption of autonomous delivery vehicles in urban areas. Partnerships with local governments and logistics providers can facilitate smoother integration and expansion of services. 32 Advancing Edge AI Perception Platforms and Sensor Fusion Figure 1.15 The autonomous vehicle [10]. The success of the technology depends on navigating complex urban environments, which include everything from busy streets to unpredictable behaviour of pedestrians and traffic participants. The path to widespread adoption of last-mile delivery autonomous vehicles is filled with challenges. The technology is still evolving, and ensuring the safety and reliability of autonomous systems is a top priority as the vehicles must be able to navigate a wide range of real-world scenarios, from inclement weather to unexpected road closures. Public acceptance and the impact on the workforce are also critical considerations. Gaining the trust of consumers and integrating these vehicles into daily life without causing disruption is key. Generative AI can enhance the interaction between autonomous vehicles and users through natural language processing, allowing users to communicate with delivery systems using voice commands, while facilitating seamless user experiences where customers can track orders or interact directly with service interfaces. Addressing future trends and overcoming technological, regulatory, and social challenges, while developing new technologies and AI-assisted tools, is key to unlocking the full potential of autonomous last-mile delivery. 1.5.2 Future research Autonomous last-mile delivery applications are leveraging advancements in edge AI platforms and sensor fusion technologies (camera, LiDAR, IMU, 1.5 Future Considerations and Research 33 ultrasound, V2X, etc.), specifically tailored to meet the unique requirements and constraints of these applications, including close-quarters navigation, interaction with VRUs, and severe cost and power limitations. Key architectural concepts, including various levels and strategies for sensor fusion, are employed alongside enabling algorithms (e.g., deep learning methods such as CNNs and Transformers) and hardware components (e.g., GPUs, specialised accelerators, and edge computing platforms). Significant integration challenges, encompassing sensor calibration, data synchronisation, computational load management, power consumption, cost-effectiveness, and fault tolerance, still need to be addressed. The pressing research challenges are achieving robust perception in adverse conditions, reliable VRU detection, handling sensor limitations, and ensuring the safety and validation of autonomous systems. Key future research directions include novel fusion architectures, end-toend learning, self-supervised methods, enhanced V2X integration, explainable edge AI (XAI) and interpretable edge AI (IAI) for fusion, lightweight models for edge deployment, and advanced simulation for validation, all aimed at advancing the safety, reliability, and efficiency of autonomous last-mile delivery. Future work includes the concept for a dedicated architecture and a multimodal AI-based autonomous vehicle platform for perception, automated control, and decision-making in delivering goods in controlled environments. The work addresses evaluating the integration of data from multiple sensors. Multimodal AI and generative AI enable real-time correlation and a more comprehensive understanding of the vehicle’s surroundings, allowing for vehicle control through voice and gesture commands. Future work plans to address the adoption of new concepts provided by Software-Defined and AI-Defined Vehicles (SDV/AIDV) architectures, where the vehicle’s features and functionality are determined by the holistic interplay between sensors/actuators, the hardware, software, AI platforms, and data and ROS is well-suited to this new paradigm, as it provides a flexible and modular platform for developing and deploying a wide range of applications. As last-mile delivery autonomous vehicles operate in fleets and are connected, there is a growing trend towards offloading the computational tasks to the edge. ROS 2’s support for DDS makes it easy to extend the vehicle’s communication system to include edge-based services. Further research is addressing the advancements of autonomous delivery vehicles, focusing on enhancing the human-machine interfaces between 34 Advancing Edge AI Perception Platforms and Sensor Fusion AI-driven vehicles and humans operating individual vehicles or fleets, improving security measures to protect against tampering, and refining the sustainability aspects of autonomous vehicle operations through various use cases and business models. Human-autonomous system interaction and the development of new human-machine interfaces require further investigation to enhance trust, acceptance, and the successful integration of last-mile delivery autonomous vehicles and IoRT into daily life. Future research combining last-mile delivery autonomous vehicles, IoRT, and edge AI needs to focus on advancing decentralised swarm intelligence, enabling fleets of autonomous vehicles to collaborate and make collective decisions to improve scalability and resilience, allowing the fleet to adapt to unforeseen events in real-time. Significant research should be directed towards developing more advanced and energy-efficient edge AI algorithms for predictive analysis, anticipating delivery demand, and optimising V2X communication. Robust security, privacy-preserving protocols, and trustworthiness within the IoRT framework are crucial for safeguarding sensitive delivery data and protecting autonomous systems from cyber threats, making this an important future area of research. Further research investigation includes the concept for a dedicated architecture and a multimodal AI-based autonomous vehicle platform for perception, automated control, and decision-making in delivering goods in controlled environments. The research also includes evaluating the integration of data from multiple sensors, combined with decision support that integrates small language models, vision language models, and agentic AI. Multimodal AI and generative AI enable real-time correlation and a more comprehensive understanding of the vehicle’s surroundings, enabling the implementation of vehicle control through voice and gesture commands. Another key focus is on standardisation and the development of AI-assisted tools that improve the capabilities of frameworks like ROS and increase the efficiency of designing last-mile delivery autonomous vehicles. 1.6 Conclusion Early implementations of last-mile delivery autonomous vehicle technologies demonstrate significant potential for cost reduction and efficiency enhancement in last-mile logistics. Nonetheless, lessons from these operations underscore the complexity of urban environments and the need for 1.6 Conclusion 35 continuous adaptation of technologies to meet real-world challenges. Key takeaways include prioritising safety, enhancing AI-based decision-making across diverse scenarios, and maintaining robust validation methodologies for autonomous systems. AI integrates data from multiple sensors, including cameras, LiDAR, IMU, ultrasound sensors, and GNSS, to create a comprehensive understanding of the vehicle’s surroundings. This fusion of sensory information enhances the vehicle’s decision-making capabilities, as it can analyse and interpret complex datasets to ascertain the safest and most efficient operation. AI serves as the backbone of autonomous delivery systems, driving their efficiency, safety, and user engagement. By harnessing advanced algorithms and technologies, these systems can provide faster, reliable, and more costeffective delivery solutions, revolutionising the logistics and delivery sectors. As AI technology continues to develop, its role in enhancing autonomous delivery operations is expected to grow, paving the way for even more innovative solutions in this area. Acknowledgements This publication has received funding through the projects ECSEL Joint Undertaking (JU) AI4CSM, Chips JU EdgeAI, and Chips JU MOSAIC. The ECSEL JU AI4CSM “Automotive Intelligence for Connected Shared Mobility” project was supported by the ECSEL Joint Undertaking and its members, including top-up funding from Austria, Belgium, the Czech Republic, Italy, Latvia, Lithuania, the Netherlands, and Norway under grant agreement No. 101007326. The Chips JU EdgeAI, “Edge AI Technologies for Optimised Performance Embedded Processing,” project is supported by the Chips Joint Undertaking and its members, including top-up funding from Austria, Belgium, France, Greece, Italy, Latvia, the Netherlands, and Norway, under grant agreement No. 101097300. The Chips JU MOSAIC “A Mosaic of Essential Electronic Components and Systems (ECS) for our Automated Digital Future in Industry and Mobility” project is supported by the Chips Joint Undertaking and its members including top-up funding by Austria, Belgium, Czech Republic, Denmark, France, Greece, Israel, Italy, Latvia, Netherlands, Norway, Poland, and Türkiye under grant agreement No 101194414. Funded by the European Union. Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union or Chips Joint Undertaking. Neither the European Union nor the granting authority can be held responsible for them. 36 Advancing Edge AI Perception Platforms and Sensor Fusion The authors would like to acknowledge the contributions of Espen Teigen, ISPAS AS, Norway, for the software implementation of the odometry and path planning modules described in this article during his tenure at NxTECH AS, Norway. References [1] M. G. Augusto et al., “Autonomous Van and Robot Last-Mile Logistics Platform: A Reference Architecture and Proof of Concept Implementation,” Logistics, vol. 9, no. 1, pp. 10–10, Jan. 2025, https://doi.org/10.3 390/logistics9010010. [2] BotPenguin, “Last Mile Delivery,” Botpenguin.com, 2024. https://botp enguin.com/glossary/last-mile-delivery. [3] Wise Systems, “The State of the Autonomous Last-Mile Market in 2023,” Wise Systems, Nov. 02, 2023. https://www.wisesystems.co m/blog/autonomous-last-mile-market-2023/ [4] V. Engesser, E. Rombaut, L. Vanhaverbeke, and P. Lebeau, “Autonomous Delivery Solutions for Last-Mile Logistics Operations: A Literature Review and Research Agenda,” Sustainability, vol. 15, no. 3, p. 2774, Feb. 2023, https://doi.org/10.3390/su15032774. [5] O. Vermesan et al., “4 Internet of Robotic Things -Converging Sensing/Actuating, Hyperconnectivity, Artificial Intelligence and IoT Platforms.” Available at: https://www.riverpublishers.com/pdf/ebook/chap ter/RP_9788793609105C4.pdf [6] O. Vermesan et al., “Internet of Robotic Things Intelligent Connectivity and Platforms,” Frontiers in Robotics and AI, vol. 7, Sep. 2020, https: //doi.org/10.3389/frobt.2020.00104. [7] O. Vermesan, “Embedding ROS and Edge AI-Based Perception Capabilities in AV Platforms,” European Conference on EDGE AI Technologies and Applications – EEAI 2023, 17-19 October 2023, Athens, Greece. https://edge-ai-tech.eu/wp-content/uploads/2023/11/2023-10 -19_Presentation_C03_2_Conference_EDGE-AI_Athens_EEAI_2023. pdf. [8] O. Vermesan, “Trustworthy Edge AI Solutions in Autonomous Vehicle Perception and Sensor Fusion,” European Conference on EDGE AI Technologies and Applications – EEAI 2024, 21-23 October 2024 Cagliari, Italy. https://edge-ai-tech.eu/wp-content/uploads/2024/11/202 4-10-23_Presentation_B03-3_Conference_EDGE-AI_Cagliari_EEA I_2024.pdf. 2.1 Introduction and Background 43 In order to be able to produce highly optimized and target-specific implementations of desired DNNs, we developed Aidge, a framework containing tools and methods that allow users to act both at the graph-level and at the operator level. Aidge is thus at the same time a neural network graph editor and a compiler that accepts high-level descriptions of DNNs (e.g. in ONNX) and produces low-level code (e.g. in C or assembly) optimized and targeted at chosen hardware back-ends. One of the great challenges in generating optimized code from high-level descriptions is the fact that different architectures manage operations, data and memory in different ways. For example, Deep Learning Accelerators (DLAs) [5] [1] [6] usually implement optimized tensor compute primitives, while GPUs [7] exploit their massive parallelism, and modern CPUs [8] [9] [10] contain vectorized instructions. Moreover, CPUs and GPUs automatically control pipeline dependencies to hide memory access latency, while for DLAs this has to be explicitly implemented by the developer. All these factors render the creation of a multi-target tool extremely complicated. 2.1.1 Related Work Although DL models have seen an incredible rise in multiple domains and applications, the same cannot be said about frameworks that allow to easily optimize and deploy them to a wide range of hardware targets. One way to represent and perform high-level optimizations is through computation graph domain-specific languages (DSLs). Examples of these are Tensorflow’s XLA [2], DLVM [11] and Glow [12]. Although these representations are well suited for high-level optimizations, they are not apt for low-level operator optimization. To do this, many frameworks resort to lowering procedures to directly generate low-level LLVM or utilize proprietary vendor libraries. Clearly, these methods require considerable engineering effort, considering that they have to be done for every combination of hardware backend and operator variant. An interesting solution has been proposed in the Halide language and compiler [13] where computing and scheduling are separated. This allows the authors to obtain considerable simplifications in programming and major speed-ups compared to hand-tuned C, intrinsics, and CUDA implementations. A different optimization method is proposed in Weld [14] where diverse functions can submit their computations in a simple but general intermediate 44 AIDGE: A Framework for Deep Neural Network Development, Training representation that captures their data-parallel structure. It then optimizes data movement across these functions and emits efficient code for diverse hardware. DnnWeaver [15] is a framework that automatically generates a synthesizable accelerator for a given (DNN, FPGA) pair from a high-level specification in Caffe. It uses hand-optimized design template to first translate a given highlevel DNN specification to its novel ISA that represents a macro dataflow graph of the DNN, then it tiles, schedules, and batches DNN operations to maximize data reuse and best utilize target FPGA’s memory and other resources. Finally, TVM [16] is a DNN compiler that has the capability of optimizing code by searching and combining the best tensor operators. This compiler provides end-to-end compilation and optimization stacks that allow the deployment of DNNs on CPUs, but also mobile GPUs, and FPGA-based devices. 2.2 Our Framework Overview In this work we present Aidge, a new end-to-end framework for training, optimizing and compiling DNNs especially for low power edge devices. This tool was designed to balance efficient compilation, flexibility, low level control and portability by combining insights from graph analysis and manipulation with methods from structured and functional programming languages. The platform integrates database construction, data pre-processing, network building or importing, manipulation, optimization, quantization, testing and hardware export functionalities (see Figure 2.1). It is particularly useful for DNN design and exploration, allowing simple and fast prototyping of different DNNs. With this tool it is possible to define and train multiple topology variations of a network and to automatically compare their performances (in terms of accuracy and computational cost). One distinctive aspect of Aidge is that it is based on the principle of “modularity”, i.e. there is a “Core Module” (see Figure 2.2) that can be extended by so called “plugins” that allow to add new functionalities and to meet needs not foreseen or implemented during the initial design of the framework. 2.2 Our Framework Overview 45 Figure 2.1 Schematic representation of the Aidge Framework with its main components and functionalities The Core Module is developed entirely in C++ (14) with bindings to Python (>3.7), and includes a set of functions that enable it to: • Create a computational graph representing a DNN. • Modify the computational graph (e.g. by deleting, replacing or adding a node). • Do graph querying/matching to find a specific sequence of operators in the computational graph. • Instantiate operators. • Instantiate data structures, such as Tensors. • Create schedulers (for now only sequential) to execute the computational graph • Apply graph optimization, such as fusion of operators Aidge separates the concepts of description and implementation. Operators and data descriptions are abstract, while implementations are targetspecific. For example, the software implementation of a convolution may differ on a GPU or CPU, but the definition of the convolution itself (i.e. its inputs and parameters) does not change. Moreover, the implementation might also change according to the utilized library, for example on an NVIDIA GPU, programming can be done either via CUDA or via TensorRT. For this reason Aidge introduces the notion of “Backend” to define both the hardware target 46 AIDGE: A Framework for Deep Neural Network Development, Training Figure 2.2 Aidge is built upon the concept of modularity with a “Core” component and several “plugins” that complete and extend the framework. and the library used for the implementation (with its data type and a number precision) Plugins allow developers and users to add or adapt functionalities of the platform. Different kinds of plugins can be developed (in C++ or Python) using the Aidge API, such as: • “Recipe plugins”, which may allow to load and save the network description in a specific format, or it may consist in a set of optimizer algorithms, for example to reduce the model’s cost in terms of memory and computing complexity, or to increase its robustness against external/adversarial attacks. • “Dataset plugins”, which add the capability to load data and labels from a specific dataset. • “Backend plugins”, which register to the Core compiled kernel libraries (e.g. C++, CUDA, HLS) allowing it to execute the computational graph. 2.2 Our Framework Overview 47 Figure 2.3 The image shows the constituent parts of an example Convolution operator. • “Operator plugins”, which adds the ability to define a new operator in C++ which is not available in the Core. • “Export plugins”, which define a set of rules and methods aimed at adapting the graph to the targeted hardware, and methods to produce source code corresponding to the optimized graph. 2.2.1 Internal Graph Representation Aidge’s low-level architecture is designed to allow the highest flexibility in DNN representation and computation, thus DNN models are represented using a directional “computational graph”. This graph is composed of a set of nodes, representing operations (Operators), connected with directed edges, representing the flow of data. Nodes in this computational graph are defined by three properties: the connectivity, the operation description, and the implementation. 1. Operation description: it describes the operation a node will do (e.g. Convolution, ReLu, Data Provider, etc.) and its attributes (see Figure 2.3). This description is agnostic of the implementation. The attributes are the following: ◦The sizes of the Kernel, Dilation (in case of convolutions), Stride, etc. ◦The number of inputs and their dimensions, datatype and precision; ◦The number of outputs and their dimensions, datatype and precision; 48 AIDGE: A Framework for Deep Neural Network Development, Training ◦A reference to a forward (i.e. inference) function implementation; ◦A reference to a backward (i.e. train) function implementation. 2. Connectivity: it describes which nodes (proper Operators or Data Providers) are connected to a given node. 3. Implementation: it points to the computational function/kernel used by the Operator for its forward and backward operations. The selection of the right implementation is made via a registrar system depending on the following attributes: ◦The Backend, defined by both the hardware target (e.g. CPU, GPU, ...) and available libraries (e.g OpenCV) ◦The DataType (float, int, . . . ) and Precision (8bits, 16bits, 32bits,...) of the inputs and outputs ◦The DataFormat (NCHW, NHWC, . . .) ◦The Kernel, the algorithm chosen to perform the computation This flexible computational graph description is paired with the ability to use a great variety of data representation (e.g. Tensors, Sparse Tensors, Event Based stimuli, etc.). 2.2.2 Platform interoperability Thanks to PyBind11, there is a seamless interoperability with Numpy arrays, achieved by defining a buffer_protocol in the binding of Aidge Tensors. This allows to use data imported from other frameworks that are compatible with Numpy. Aidge is interoperable with PyTorch and allows: • Creating an Aidge Tensor from a PyTorch Tensor • Running an Aidge (sub)graph within the PyTorch environment. • Running an Aidge computational graph within the PyTorch environment. Aidge allows interoperability with Keras by creating a wrapper from a Keras Model through a conversion step via an ONNX file. Similarly to PyTorch, Aidge can convert Keras tensors by using the Numpy interoperability. 2.2.3 Graph Regular Expression (GraphRegex) The proposed Aidge’s internal graph representation is a powerful tool that combines carefully chosen abstraction levels. The strategy is to adapt the internal representation to narrow the gap between a neural network and 2.2 Our Framework Overview 49 hardware devices. Aidge proposes an innovative way to facilitate the manipulation of the internal graph representation: the Graph Regular Expression or GraphRegex The Graph Regular Expression combines two main innovations: 1. A description of graph patterns. Taking inspiration from regular expressions from the formal language theory, we introduce a new language to describe a set of graphs starting from a sequence of characters. 2. Graph matching. Aidge provides a function that allows to extract a subset of the graph matching the provided GraphRegex description. Graph Regular Expression is complementary to other graph transformation methods such as adding and removing nodes or entire parts of the graph. With GraphRegex it is possible to work on two distinct levels in a graph: 1. At a conditional level, which corresponds to checking the presence of a node in a defined dictionary. 2. At a topological level, which allows to describe the interconnections between symbols. The topological description, can be compared to classical regular expressions, as it is a form of symbol sequence expression, but extended to the definition of graphs. At a practical level, this matching method can be subdivided into two distinct stages. First, we describe the desired pattern with the syntax of regular expressions, then we search for that pattern inside the graphs. These two steps together form the GraphRegex Query. After extracting all the subgraphs corresponding to a GraphRegex Query, it is possible to use an intersection resolution algorithm to obtain intersectionfree solutions. However, it is important to note that these algorithms can have a high complexity, which can make their execution time-consuming. 2.2.4 Network optimization We can define two categories of optimizations: topological ones, which change the structure of the computation graph, and parametrical ones, which change the parameters of the nodes. An example of topological modification is Tiling. This method splits convolutions in several ones (for example in 4 convolutions, as show in Figure 2.4). All of them are computed independently and concatenated at the end. This manipulation is mathematically exact (lossless). 50 AIDGE: A Framework for Deep Neural Network Development, Training Here is the practical implementation: Figure 2.4 Example of operator tiling/splitting: a Conv + Relu subgraph is split into a Slice + 4 Conv + 4 Relu + Concat. One of the key differentiators compared to other frameworks such as LLVM, is that Aidge applies directly graph modifications, which allows to make global topological changes as opposed to only focus on local ones. On the other hand, an example of parametrical optimization is quantization after training (PTQ) or during training (QAT). This is a well-established method for reducing memory usage and in most cases, accelerating the 2.2 Our Framework Overview 51 inference. PTQ is very useful when one does not have the time or the possibility to re-run the training and does not need to quantize to more than to 8-bits. If fewer bits are necessary, state-of-the-art QAT methods give very good results. These and other techniques (e.g. LSQ and FracBit QAT) are currently being finalized in Aidge. 2.2.5 Export phase One of the aims of Aidge is to produce an interpretable, explainable and auditable output. To do this Aidge produces/exports source code files and a number of related resource files that form a complete package. In Figure 2.5, which summarizes the export strategy, it is possible to see two phases: Export Mapping and Export Implementation. The first objective of the Export Mapping phase is to modify the computational graph to fit the target hardware by using several optimization techniques (e.g. hardware mapping optimization or graph transformation). The second objective is the generation of the graph scheduling constrained by the architecture rules of the target and additional project rules imposed by the developer or the user (e.g. the available memory, the available computer resources or the time allocated for the execution). Taking into account the architecture rules and the project constraints, the scheduler will generate a sequential list of nodes from the optimized graph that will determine how the forward process (i.e. the inference) of the exported DNN will run on the target. Figure 2.5 Schematic representation of Aidge’s export procedure. 52 AIDGE: A Framework for Deep Neural Network Development, Training The Export Implementation phase aims at producing a source code of the hardware-tuned graph returned by the scheduler. The typical steps for generating source code are the following: 1. Design and export the computation kernels. 2. Export the attributes of the nodes. 3. Export the parameters of the nodes. Each node of the graph must have an implementation of its forward method in order to use it in the export. Since only the hardware developers really know the characteristics and capabilities of their devices, it is their duty to provide the implementations of the computation kernels. These may be implemented as a kernel library, which is a collection of optimized functions developed by expert programmers targeting the architecture (computing functions, DMA programming, etc...). Together with the kernels, Aidge generates the configuration and parameter files, and also the files that contain the source code of the forward function of the hardware-adapted graph. The developer has also the possibility to add files to generate a whole Software Toolkit that will provide functionalities such as: • Compilation or project files to compile the export • Files to run a whole application of the export • Set of unitary tests (to test the kernels on board, ...) • Input data for tests • Third party libraries to use board functions • Resources to check other constraints like security rules or robustness directives • Memory map files indicating information about the static allocation of the resources used by the 2.3 Conclusion and future work In this article we proposed Aidge, a framework that allows end-to-end manipulation, optimization and compilation of DNN architectures and their deployment to a vast spectrum of hardware devices ranging CPUs to GPUs, MCUs, DSPs, FPGAs and neuromorphic architectures. Another aim of this framework is to develop and provide reusable hardware building blocks and methodologies that are transversal to all types of architectures. 3.4 Interconnect-based dataflow architecture 59 which is computed using the equation below: ofmap_rows = (((ifmaps_rows −(filter_rows +padding_start +padding_end))/stride) + 1) 3.4.1 NGC: Neural Global Controller The NGC is a five-stage Finite-State Machine (FSM) shown in Figure 3.1 (b). Following is the description of each stage: 1. IDLE represents the idle state of the NGC. 2. LOAD_config loads and decodes the configuration line for the current layer, including the size and number of filters and ifmaps for the current layer. 3. LOAD_filter starts sending the filter data (i.e., payload) and the control word from the Input GBs (IGBs) to the connected routers. Depending on the opcode, routers either unicast, multicast, or broadcast the incoming payload data to the AINoC. 4. LOAD_ifmaps starts sending the ifmaps data and the control word from the IGBs to the connected router. Depending on the opcode, routers either unicast, multicast, or broadcast the incoming payload data to the AINoC. LOAD_filter and LOAD_ifmaps states can be interchanged to provide some flexibility in the data loading order in the proposed architecture. 5. COMPUTE starts the MAC operations in the NPEs. Once a Partial Sum (PSum) is computed in the bottom row NPEs, the results are sent to their respective north NPEs along with their control words. The NPEs in the upper row then add the incoming results with their locally computed PSum and send the computed result to their north NPEs. This chain of operations is executed until it reaches the top NPE row, where the final PSum is stored in the Output GBs (OGBs) to be used in the next layer. 3.4.2 NPE: Neural Processing Element The NPE is a simple operator controlled by the FSM of the NGC, as shown in Figure 3.1 (c). It remains in the idle state until the NGC triggers the execution. It proceeds in this way according to the control signals sent by the NGC: •When the load_filter signal is received, it stores the incoming payload into the filter Register File (RF). Once all the NPEs have received their 60 A scalable and flexible interconnect-based dataflow architecture for Edge corresponding filter data, the top right NPE in the NPE array sends a signal to NGC to jump to the next state. •When the load_ifmap signal is received, it stores the incoming payload into the ifmaps RF. Once all the NPEs have received their corresponding ifmaps data, the top right NPE in the NPE array sends a signal to NGC to jump to the next state. When the start_compute signal is received, it begins the MAC operation on the filter and ifmaps data and generates PSum. Then, it either (i) sends the PSum to the north NPE (if bottom row NPE) or (ii) adds the incoming PSum from the south NPE with local PSum and sends the result to the north NPE or OGB (if top row NPE). In this phase, communication-computation overlap is also performed by the NPEs, which received their required data. At the end of the computation, an end_compute signal will be sent by the last NPE of the array to inform the NGC of the end of the execution, in order to move on to the next execution and the loading of new data. Figure 3.1 (a) The proposed interconnect-based dataflow architecture sub-system, (b) Neural Global Controller (NGC), (c) Neural Processing Element (NPE), (d) Router in Artificial Intelligence Network on Chip (AINoC). 3.4 Interconnect-based dataflow architecture 61 3.4.3 AINoC: Artificial Intelligence Network-on-Chip The AINoC [9] consists of routers optimised for parallel dataflow processing with minimal data transfer cost to achieve energy-efficient CNN processing without compromising accuracy and application performance. As shown in Figure 3.1 (d), the routing device is composed of several parallel routing paths, each including a buffer, a communication controller, an arbiter, and a switch. All these paths are designed to guarantee a large bandwidth and flexible communication. Indeed, through several buffering modules, e.g. First-In-First-Out (FIFO), different communication requests received in parallel can be stored without any loss. These requests are then processed simultaneously in several control modules. These modules ensure a deterministic control of the data transfer according to a static X-Y (X-direction priority) routing algorithm and management of different communications (unicast, multicast, and broadcast). Parallel arbitration of the processing order of incoming data packets according to the RoundRobin Arbitration (RRA) [5] based on scheduled access allows for better collision management, i.e., a request that has just been granted, will have the lowest priority on the next arbitration cycle. Parallel switching comes next to simultaneously route data to the right outputs according to the Wormhole switching [11], i.e. the connection between one of the inputs and one of the outputs of a router is maintained until all the elementary data of a message packet are sent and this in a simultaneous way through the different switching modules. The data packet format is shown in Figure 3.2. A data message consists of two packets: a control packet followed by a data packet. A packet is composed of a header (flit code) and a payload. In the control packet, the payload is a destination or source address, while in the data packet, the payload is a set of data flits. The packet size is 32-bit. However, the size of the header and the payload are variable. It depends on the size of the interconnection network, as the number of routing devices increases, more bits are needed to encode the addresses of the receivers or senders. Similarly, the flit size and number vary with the size of the payloads (filter weights, activation inputs, or PSums) to be passed through the network. The value of the header determines the communication to be provided by the router. There are three possible types of communication inter-PEs: unicast, multicast (horizontal, vertical, and diagonal), and broadcast. For memory access, the 62 A scalable and flexible interconnect-based dataflow architecture for Edge Figure 3.2 Packet format reading from the IGB is a multicast communication; however, the writing to the OGB is a communication type that processes a direct parallel unicast from the first NPEs rows, and the OGB. The routing device first receives the control packet containing the type of communication and the source or destination address. The routing device decodes this control packet and then allocates the communication path to transmit the data packet that arrives at the cycle following the control packet. Once the data flits are transmitted, the allocated path will be released for further transfers. 3.5 Execution Model 63 3.4.4 Global Buffers GBs are dual-port Random Access Memory (DPRAM) that are used to store the input data i.e., filter and ifmaps or output data i.e., PSum from top row NPEs. The size of each GB type is determined according to the data size requirement for each layer, such that the overhead due to GB is minimised. 3.5 Execution Model The data movement and execution pattern in the proposed architecture are presented in this section. Once the data is ready in IGB, the execution in the proposed architecture can be divided into three phases, i.e., (1) load ifmap data into their respective NPEs, (2) load filter data into their respective NPEs, and (3) perform execution on the available data in each NPE. These steps are explained below: 1. Load ifmap data: In this phase, ifmap data are loaded into their respective NPEs. Data from IGBs are diagonally loaded into the NPEs, which have connections with them through a single router, and then data reuse is performed by moving the data diagonally to the target NPEs. 2. Load filter data: In this phase, filter data are loaded into their respective PEs. Data from IGBs are horizontally loaded into the NPEs, which have connections with them through a single router, and then data reuse is performed by moving the data horizontally to their respective NPEs. During this phase, the overlap between communication and computation is also performed. The NPEs, which receive the required data to compute the partial sum, begin the computation phase. Particularly, the first column of the NPE array gets all the required data and jumps from the communication (i.e., data receiving) phase to the computation phase while other columns still wait for input data. 3. Execute MAC operation: When an NPE receives all required data, it jumps from the communication phase to the computation phase. Each column is locally synchronised, where the bottom NPE sends the computed PSum to the north NPE. Each NPE (except the bottom NPE) adds the PSum received from their south NPE with the locally computed PSum before sending the result to their north NPE. This chain of receiving, adding, and sending data is performed until the data reaches the top NPE, where the computed result is stored back into the OGB. NPE array is executing in the Globally Asynchronous Locally 64 A scalable and flexible interconnect-based dataflow architecture for Edge Synchronous (GALS) pattern to enable overlap between communication and computation in the proposed architecture. 3.6 Experiments and Results 3.6.1 Evaluation Methodology In this work, different CNN algorithms from state-of-the-art were used as case studies. They have different sizes and include different types of layers and shapes. LeNet5 [18] and MobileNetV2 [12] were chosen to have a collection of data resulting from a range of small to large CNN and using a set of layers including classical 2D convolution (CONV2D) and fully connected layers (FC) but also point-wise (PW) and depth-wise (DW) convolution layers in MobileNetV2. Table 3.1 details the characteristics of all these CNN algorithms, including the types of layers they have and the number of each layer type. The values in the proposed architecture configuration are obtained by following the calculation rule presented in section 1.4. In our experimental study, we chose to test the key convolution layers that emphasise different filter sizes and ifmaps and the fully connected layers that require a linear spatial representation of the proposed architecture. We also note that a configuration for the proposed architecture must be generated for each Table 3.1 CNN Layers type CNNs Layer Type ifmap size filter shape Config. of proposed architecture LeNet5 conv_1 1x32x32 1x5x5 5x28 conv_2 6x14x14 6x5x5 5x10 conv_3 16x5x5 16x1x1 1x5 fc_1 1x1x120 1x120x84 1x84 fc_2 1x1x84 1x84x10 1x10 MobileNetV2 conv_1 1x128x128 8x[3x3x3] 3x126 conv_2 8x64x64 8x3x3 3x62 conv_3 24x64x64 24x3x3 3x62 conv_4 36x32x32 36x3x3 3x30 conv_5 48x16x16 48x3x3 3x14 conv_6 96x8x8 96x3x3 3x6 conv_7 144x8x8 144x3x3 3x6 conv_8 240x4x4 240x3x3 3x2 conv_9 80x4x4 256x1x1 1x4 3.6 Experiments and Results 65 evaluated layer to respect the RS dataflow execution mode (section 1.3.2). However, the row width for the FC layer (i.e., 1000) of MobileNetV2 is too big for the proposed architecture, due to the limited space allotted to store the value of the number of channels in the configuration word, so this layer has been excluded from our experiments. 3.6.2 FPGA Implementation Results The evaluation platform used for all tests is the Versal ACAP VCK190 kit [20] featuring an “XCVC1902-2VSVA2197” FPGA partition containing 899840 programmable LUTs, 899840 Flip-Flops, 1968 DSP58, and 158Mb of URAM and BRAM. The software tools used to implement and test different configurations of the proposed architecture are: • QuestaSim or Questa Advanced Simulator (version 2021.4) from Mentor Graphics is provided to simulate and test the programming and debugging of FPGA chips. • Vivado Design Suite (version 2021.2) is a software suite produced by Xilinx to synthesise and analyse hardware description language (HDL) designs. 3.6.2.1 Area The different configurations of the proposed architecture include four main modules: the NGC, distributed memories (IGB & OGB), a given number of NPEs, and routers that are directly connected to the NPEs. All configurations of the proposed architecture are designed with the VHDL description language to be rapidly implemented on FPGA. The implementation results estimate the frequency of the proposed architecture, which is around 125 MHz. This frequency depends on the frequency of the longest critical path in the configuration. A good place and route for the modules of the proposed architecture is necessary to reduce the length of the critical path and accelerate the propagation of the signals. The synthesis results define the occupied area (logic elements, memory, Digital Signal Processing (DSP) blocks, etc.) and the hardware resources consumption of the proposed architecture according to the different configurations defined in Table 3.1. The synthesis results of the different modules constituting the proposed architecture are given in Table 3.2. Due to the simple structure of the different modules, the consumption of logic and memory resources remains low. This allows generating a configuration of the proposed architecture with a large 66 A scalable and flexible interconnect-based dataflow architecture for Edge grid of computing elements to process large convolution layers. For the GB memories, we opted for the use of Block Random Access Memory (BRAM) by forcing the synthesis tool to choose these memory blocks instead of the configurable logic blocks (CLB). We also notice that the size of the router is relatively larger than the NPE. This can be explained by NPE providing a simple convolution operation. At the same time, the router has multiple routing paths to provide parallel multicast and control of blocking areas in the communication network. These multiple routing paths mainly accelerate the data transfer and reduce the energy consumption during the execution of a convolution layer. It is then a trade-off between area and performance in the proposed architecture. Area can be treated as a small overhead to ensure a balance in the choice of the architecture and the objectives to be achieved. Table 3.2 Breakdown of Versal ACAP VCK190 FPGA resources used by the modules of the proposed architecture after synthesis Module CLB BRAM Area occupancy (%) NGC 12.21 0 0.02 GB 0 0.5 0.05 Router 76.78 0 0.13 NPE 39.10 0 0.07 Figure 3.3 Synthesis results of different configurations of the proposed architecture 3.6 Experiments and Results 67 Figure 3.3 shows the percentages of FPGA resource utilization when executing the different layers of Lenet5 and MobilenetV2 given in Table 3.1. The processing of each type of layer requires a particular configuration of the proposed architecture. A configuration of the proposed architecture depends on the number of filter rows, ifmap rows, and ofmap rows. We observe a correlation between the variation in the size of the proposed architecture and the consumption of the CLBs. The larger the configuration, the greater the resource consumption. The consumption of the BRAM memory blocks depends on the size of the input image or the ifmaps. This means that the memory size remains fixed for a fixed input image/feature-map size, and the size of the filter. Particularly, for the conv_1 of MobileNetV2, we notice that the number of CLBs exceeds the maximum number of CLBs available in the FPGA targeted in these experiments. This representation shows that the proposed architecture remains flexible to support all convolution layer sizes. We just need to aim for a prototyping platform that provides the necessary logic resources for mapping all layers. 3.6.2.2 Latency Figure 3.4 shows the latency performance of the proposed architecture for each convolution type. It can be observed that the proposed architecture is up to 71.2×(conv_1, Lenet5) faster w.r.t. single RISC-V CPU [2]. The total execution time for each convolution type for the proposed architecture is divided into ifmap loading time, filter loading time, data reuse, and overlap between communication-computation including time required for the PSum to traverse across their respective columns to store the computed ofmap. The breakdown of latency reports that data reuse and overlap between communication and computation significantly improve the overall execution time in the proposed architecture. For latency comparison of the proposed architecture with RISC-V CPU, the time required for access L2 to load data into IGBs is also considered for a fair comparison. The overall speedup of MobileNetV2 convolution layers is up to 2.07× w.r.t. Eyeriss v2 [16, 17]. Here, Eyeriss v2 executes all layers of MobileNetV2 while the proposed architecture executes convolution layers (Table 3.1). These results are obtained through RTL simulations. 3.6.2.3 Energy consumption Different hardware modules of the proposed architecture involved in different execution phases for each convolution type are shown in Table 3.3. The explanation of each phase is as follows: (1) Phase A represents data loading 68 A scalable and flexible interconnect-based dataflow architecture for Edge from all IGBs, (2) Phase B represents data reuse, (3) Phase C represents data loading from row IGBs, and (4) Phase D represents computation in NPE array. The results in this section are obtained through hardware emulation. Figure 3.4 Breakdown of latency (ns). For the proposed architecture, the convolution layer includes memory accesses and computations. WORK = This Work, RV32 = RISC-V CPU. Table 3.3 Different execution phases in the proposed architecture Execution Phase NPE array AINoC IGB row IGB column OGB NGC A X X X X X B X X X C X X X X D X X X X 4.2 Federated Learning and Related Work 75 In malware classification, effective and efficient detection models require significant amounts of training data. Although such amounts of data could be collected and made available through data sharing among organizations, there are intellectual property and privacy concerns and constraints that forbid or limit such exchanges. Federated Learning (FL) is a promising method for building effective and efficient detection models in a distributed fashion, using data of different organizations, because it does not require the exchange of source data [9]. FL is employed in two main configurations, cross-device and cross-silo. In cross-device configurations a large number of members (clients) with limited data samples each are coordinating in developing a model. Cross-silo configurations have significantly less members (clients), each with a large population of data samples. FL has been employed for malware detection in cross-device environments, focusing on IoT and Android devices [18, 19, 20]. However, FL in cross-silo configurations has not been explored. Our work focuses on cross-silo FL, considering the requirements of applications and services such as Edge or near-Edge devices and their coordination and collaboration in hierarchies that are being developed internationally. In this paper, we present cross-silo FL-based malware detection, where the detection model is constructed exploiting horizontal FL and employing a NN. Considering an analysis approach analogous to the one in Sisyfos, we measure the performance in malware detection and evaluate its dependence on several parameters, such as number of clients, repetitions of aggregation steps, dataset size and the percentage of common training data. Our results demonstrate that FL enables high accuracy in malware detection for all members of the federation, irrespective of the size of their own training dataset. This demonstrates an important advantage of FL in malware detection: members of the federation with small training datasets would never achieve independently the high accuracy which they achieve through their participation in the federation. The paper is organized as follows. Section 1.2 presents an overview of FL and the current state-of-the-art in its employment in malware detection. Section 1.3 presents our cross-silo FL system architecture. Section 1.4 presents our evaluation results and demonstrates the effectiveness of our approach. 4.2 Federated Learning and Related Work Federated Learning is an emerging machine learning approach that enables the training of AI models in a decentralized manner. Participating clients 76 Federated Learning for Malware Detection in Edge devices collaborate to train ML algorithms under the coordination of a central server, without sharing their private datasets with other parties [10]. To train a federated model a central server distributes to the participating clients an initial model and the training parameters. Then the following steps take place: 1. each participant trains the received model using their private dataset, producing a local model and then sends it to the server; 2. the server aggregates all local models into a global one; 3. the global model is distributed to all clients. Figure 4.2 illustrates this process which can be repeated for multiple learning steps and stopped when a designated criterion is met. FL is considered in two different configurations, in general [10]: •Cross-device: clients are computing systems with limited computing capabilities, varying device availability and small datasets, e.g. IoT devices or smartphones. •Cross-silo: clients are computing systems with high computational power, high reliability and large datasets (data silos), such as centralized and enterprise systems (typically 2-100). In traditional centralized machine learning environments, a device or an organization must train a model on its own self-collected data. In practice, these devices or organizations may not have access to sufficiently large data sets and the computing capabilities necessary to train an effective model. Additionally, data privacy concerns and intellectual property rights limit collaboration between parties. FL addresses these challenges by enabling collaboration between multiple parties to jointly train effective ML models with large, diverse datasets collected from all members of the federation [11]. As the produced models are the only information shared among federation members, local data never leave the participating devices enabling data owners to keep their data private. Importantly, FL scales well because additional members can contribute to model training without any burden to other members and with reduced data traffic among them. FL is employed in several operations of cybersecurity such as attack detection, anomaly detection, trust management, authentication and other IoT related tasks [12, 13, 14, 15]. FL is also effective in malicious URL and Denial-of-Service (DoS) attacks detection [16, 17]. In malware detection, research in FL employment is mainly focused on cross-device FL where federation members (clients) are smartphones [18, 19] or IoT devices [20], while limited effort has been spent on malware detection 4.3 Architecture 77 5 Figure 1.2 Federated Learning configuration 1.3 Architecture Our proposed architecture consists of a FL cross-silo configuration as shown in Figure 1.2. Multiple participating members, indicated as clients, collaboratively train a global malware detection model, and a server is responsible for all communications as well as the aggregation of the global model. Each client owns and trains with some large amount of local private data which it does not share with the other clients nor the server. In training, each client uses the same feed forward neural network, i.e. the same architecture and training parameters, an increasingly popular method for malware detection [8][9]. Specifically, we employ a neural network deployed in [21]. We adopt its architecture because it is versatile, widely adopted and can be fitted in devices with limited computing power such as near-edge or edge devices. The model consists of 3 linear layers and a dropout layer. The output layer performs binary classification using a SoftMax layer, classifying a sample as either malicious or benign. We adapt the model in [21] to accommodate our different dataset: EMBER v2 [22] instead of EMBER v1. EMBER v2 is an update on the original EMBER dataset and contains 2381 input features instead of the 2351 used in [21]. The dataset is discussed in more detail in Section 1.4.1. Our detection system, operates in two modes: (a) training, where multiple clients are using FL to collaboratively train the detection model and (b) detection, where each participating client uses the produced global model to detect malware. Figure 4.2 Federated Learning configuration. using cross-silo FL configurations. In our work, we propose a cross-silo FLbased malware detection method, where federation members are different Edge or near-Edge devices deployed to provide security to different organizational networks. The devices collect large amounts of data and have higher computational capabilities relatively to the devices considered in cross-device configurations. State-of-the-art malware detection approaches employ neural networks architectures to train models for sample classification as either malicious or benign [8, 9]. In our system, we employ a similar neural network [21] that can effectively learn to detect malware from the training data, while being able to fit in Edge or near-Edge devices. Some preliminary results of this work were presented in [26]. 4.3 Architecture Our proposed architecture consists of a FL cross-silo configuration as shown in Figure 4.3. Multiple participating members, indicated as clients, collaboratively train a global malware detection model, and a server is responsible for all communications as well as the aggregation of the global model. Each client owns and trains with some large amount of local private data which it does not share with the other clients nor the server. 78 Federated Learning for Malware Detection in Edge devices In training, each client uses the same feed forward neural network, i.e. the same architecture and training parameters, an increasingly popular method for malware detection [8, 9]. Specifically, we employ a neural network deployed in [21]. We adopt its architecture because it is versatile, widely adopted and can be fitted in devices with limited computing power such as near-edge or edge devices. The model consists of 3 linear layers and a dropout layer. The output layer performs binary classification using a SoftMax layer, classifying a sample as either malicious or benign. We adapt the model in [21] to accommodate our different dataset: EMBER v2 [22] instead of EMBER v1. EMBER v2 is an update on the original EMBER dataset and contains 2381 input features instead of the 2351 used in [21]. The dataset is discussed in more detail in Section 1.4.1. Our detection system, operates in two modes: (a) training, where multiple clients are using FL to collaboratively train the detection model and (b) detection, where each participating client uses the produced global model to detect malware. In training mode, the FL-based training process takes place in multiple steps. In each step the following process occurs: (i) each client trains a local model using its own private data, (ii) each client sends the produced local model to the server, (iii) the server aggregates all local models, producing a global model and (iv) the server distributes the global model to all clients. Then, the clients can use the global model to measure the model’s performance against their private datasets. The process can be repeated for multiple steps to improve the model’s performance further, until a satisfactory model is achieved, considering the time and processing constraints of the clients or until no further accuracy improvement is achieved. After training is complete, in detection mode, all participating clients have received a copy of the final global model from the server. Each client can use this model to detect malware in their own systems and networks, independently from all other clients. Finally, the clients can return to training mode to refresh and retrain their model with new data. 4.4 Experiments We evaluate the performance of cross-silo FL measuring the malware detection accuracy on a benchmark dataset containing features from malware and benign files. To further explore the effectiveness of FL, we consider multiple FL training setups measuring how the detection rate is affected by the number of learning steps, the number of participating clients and the commonality in 4.4 Experiments 79 the participating clients’ datasets. Furthermore, to demonstrate the benefits of FL for the participants, we also train a centralized model of the same architecture and parameters and evaluate its performance different training dataset sizes. To conduct the experiments, we employ flower [23], a popular FL framework, for training the federated models, in conjunction with Pytorch [24]. 4.4.1 Dataset In all our experiments we use the EMBER v2 dataset [22], a publicly available benchmark dataset, which contains features extracted from both malware and benign samples using static analysis. We use EMBER because there are no widely available standard datasets; this is a well-known problem in cybersecurity research. Although it does not distribute the sample binary files, due to privacy concerns, EMBER is common choice in malware detection and analysis because of three factors: (i) its sufficient size, (ii) its set of features and (iii) it contains features of malware and benign samples. EMBER v2 contains 2381 features per sample, extracted using static analysis from 1.1 million Windows Portable Executables (PE). More specifically for training, the dataset contains 600.000 samples labelled as either benign or malicious (300.000 benign and 300.000 malicious), and 300.000 unlabelled samples. The dataset also contains 200.000 samples labelled as either benign or malicious (100.000 benign and 100.000 malicious) to be used as a dedicated benchmark testing set. In our experiments we only use labelled samples, 600.000 for training the neural networks and 200.000 for testing the produced models. 4.4.2 Evaluation results In the first experiment we consider a FL setup where 2 participating clients train a common model for 10 learning steps using the entire dataset. Thus, each client trains each local model with 300.000 data samples. We measure the accuracy, precision. recall and f1 score of the model on the test set for each step. Table 4.1 summarizes the results of the experiment. For comparison we also train a centralized model on the full dataset, 600.000 data samples and we measure an accuracy of 0,9338 on the test set. Figure 4.3 plots the accuracy of the FL model for multiple learning steps. The orange line denotes the accuracy of the centralized model as reference trained with the entire EMBER dataset of 600K samples. The results show that the accuracy of FL increases with the increasing number of training loops 80 Federated Learning for Malware Detection in Edge devices Table 4.1 Accuracy on test set of FL model with 2 clients for multiple learning steps. Number of steps Accuracy Precision Recall F1 Score 1 0,8711 0,8419 0,9137 0,8763 2 0,9111 0,9012 0,9236 0,9123 3 0,9176 0,9066 0,9312 0,9187 4 0,9194 0,9091 0,9321 0,9205 5 0,9211 0,911 0,9335 0,9221 6 0,9232 0,9123 0,9365 0,9242 7 0,9242 0,9143 0,9361 0,9251 8 0,9261 0,9183 0,9355 0,9268 9 0,9247 0,9143 0,9373 0,9257 10 0,9251 0,9157 0,9365 0,926 8 Figure 1.3 Federated Learning model performance for variable training loops. Next, we evaluate whether the number of participants influences the accuracy of the produced model. We use the entirety of the EMBER dataset (600.000 samples) and keep the same total dataset size for all experiments, distributing it equally among the participating clients in every case (i.e. for 2 participating clients, each clients holds 300.000 samples and for 5 participating clients, each clients holds 120.000 samples). We run experiments for 2,5,10,15 and 20 participants and measure the accuracy of the produced models on the test set. Table 1.2 summarizes the results of the experiments for 2 learning steps. Table 1. 2 Accuracy on test set of FL model for different number of clients for 2 learning steps Number of clients Accuracy Precision Recall F1 Score 2 0,9103 0,9015 0,9212 0,9112 5 0,9201 0,9156 0,9255 0,9205 10 0,9144 0,9091 0,921 0,915 15 0,9139 0,9089 0,92 0,9144 20 0,9175 0,913 0,9221 0,9175 0.85 0.87 0.89 0.91 0.93 0.95 0.97 1 2 3 4 5 6 7 8 9 10 Accuracy Learning Steps Accuracy on test set Best centralized learning accuracy Figure 4.3 Federated Learning model performance for variable training loops. and importantly reaches the performance of the centralized model and is on par with the results presented in [21]. Additionally, we observe that we make most of the accuracy gains in the first 2 FL training steps for this dataset. Thus, in subsequent experiments we train all FL models for 2 training steps. Next, we evaluate whether the number of participants influences the accuracy of the produced model. We use the entirety of the EMBER dataset (600.000 samples) and keep the same total dataset size for all experiments, distributing it equally among the participating clients in every case (i.e. for 2 4.4 Experiments 81 participating clients, each clients holds 300.000 samples and for 5 participating clients, each clients holds 120.000 samples). We run experiments for 2,5,10,15 and 20 participants and measure the accuracy of the produced models on the test set. Table 4.2 summarizes the results of the experiments for 2 learning steps. Table 4.2 Accuracy on test set of FL model for different number of clients for 2 learning steps. Number of clients Accuracy Precision Recall F1 Score 2 0,9103 0,9015 0,9212 0,9112 5 0,9201 0,9156 0,9255 0,9205 10 0,9144 0,9091 0,921 0,915 15 0,9139 0,9089 0,92 0,9144 20 0,9175 0,913 0,9221 0,9175 Figure 4.4 plots the accuracy of the FL model for different number of clients. The orange line denotes the accuracy of the centralized model as reference trained with the entire EMBER dataset of 600K samples. We observe accuracy is effectively independent of the number of clients, suggesting that the malware detection system can scale to more and more participants without accuracy losses. 9 Figure 1.4 plots the accuracy of the FL model for different number of clients. The orange line denotes the accuracy of the centralized model as reference trained with the entire EMBER dataset of 600K samples. We observe accuracy is effectively independent of the number of clients, suggesting that the malware detection system can scale to more and more participants without accuracy losses. Figure 1.4 Federated Learning model accuracy for different number of clients Next, we consider a case where different organizations or different devices in the same organization that participate in a FL setup, have common data samples in their private data. As attackers and malware authors use the same malware samples to infect multiple targets and as organizations process a large amount of malware on daily basis it seems a likely scenario that participants will have some degree of commonality in their private datasets. Thus, we evaluate whether the presence of overlapping samples in the participants’ training sets influences the accuracy of the produced FL model. We consider different dataset overlap percentages between the participants for 2 and 10 participating clients. Tables 1.3 and 1.4 present the results of the experiments for 2 and 10 participants respectively for 2 FL training steps. Table 1. 3 Accuracy on test set of FL models for different dataset overlaps for 2 clients. 2 Clients 0.85 0.87 0.89 0.91 0.93 0.95 0.97 2 5 10 15 20 Accuracy Number of clients Accuracy on test set Best centralized learning accuracy Figure 4.4 Federated Learning model accuracy for different number of clients. 82 Federated Learning for Malware Detection in Edge devices Next, we consider a case where different organizations or different devices in the same organization that participate in a FL setup, have common data samples in their private data. As attackers and malware authors use the same malware samples to infect multiple targets and as organizations process a large amount of malware on daily basis it seems a likely scenario that participants will have some degree of commonality in their private datasets. Thus, we evaluate whether the presence of overlapping samples in the participants’ training sets influences the accuracy of the produced FL model. We consider different dataset overlap percentages between the participants for 2 and 10 participating clients. Tables 4.3 and 4.4 present the results of the experiments for 2 and 10 participants respectively for 2 FL training steps. Table 4.3 Accuracy on test set of FL models for different dataset overlaps for 2 clients. 2 Clients Overlap percentage Accuracy Precision Recall F1 Score 0 0,9111 0,9012 0,9236 0,9123 5 0,9166 0,9139 0,92 0,9169 10 0,9045 0,9039 0,9054 0,9046 15 0,9129 0,9057 0,9218 0,9137 20 0,9114 0,902 0,9231 0,9124 25 0,918 0,9117 0,9256 0,9186 30 0,9125 0,9063 0,9201 0,9131 35 0,9124 0,9098 0,9157 0,9128 40 0,9173 0,9127 0,9228 0,9178 45 0,9319 0,9267 0,9378 0,9322 50 0,9262 0,9197 0,934 0,9268 Table 4.4 Accuracy on test set of FL models for different dataset overlaps for 10 clients. 10 Clients Overlap percentage Accuracy Precision Recall F1 Score 0 0,9144 0,9091 0,921 0,915 5 0,9162 0,9108 0,9229 0,9168 10 0,9154 0,9086 0,9238 0,9161 15 0,9167 0,9096 0,9255 0,9175 20 0,9171 0,9121 0,9233 0,9176 25 0,9169 0,909 0,9267 0,9177 30 0,9171 0,9114 0,9242 0,9177 35 0,9178 0,9138 0,9227 0,9182 40 0,9146 0,9059 0,9254 0,9155 45 0,9179 0,9093 0,9285 0,9188 50 0,9174 0,9114 0,9247 0,918 4.4 Experiments 83 11 data even in the extreme case of a 50% overlap, meaning that the produced models do not overfit on the common data. Figure 1.5 Federated model accuracy for different dataset overlaps. Finally, to showcase the benefits of FL for organizations, we consider a scenario where a single organization or device is not participating in a FL setup but instead trains its own centralized model using its own private data. We consider organizations of different sizes that have different training data availability. We train a centralized model (no federation present) of the same architecture and parameters as in the previous FL setups with different dataset sizes. Table 1.5 summarizes the results. Table 1. 5 Accuracy on test set of centralized models for different dataset sizes. Number of samples Accuracy Precision Recall F1 Score 5000 0,8443 0,8299 0,8662 0,8477 10000 0,8482 0,8428 0,8828 0,8623 0.85 0.87 0.89 0.91 0.93 0.95 0.97 0 5 10 15 20 25 30 35 40 45 50 Accuracy Overlap percentage Accuracy on test set Accuracy - 2 clients Accuracy - 10 clients Best centralized learning accuracy Figure 4.5 Federated model accuracy for different dataset overlaps. Figure 4.5 plots the accuracy on the test set of the FL models produced as a function of the overlap (common subset) of the clients’ training data, i.e. x=5 indicates 5% common data in the client datasets. The blue and green lines depict the accuracy of the models trained by 2 and 10 clients respectively. The orange line denotes the accuracy of the centralized model as reference trained with the entire EMBER dataset of 600K samples. As we observe in both setups, accuracy seems to be effectively independent of the common samples present in the participants’ private data even in the extreme case of a 50% overlap, meaning that the produced models do not overfit on the common data. Finally, to showcase the benefits of FL for organizations, we consider a scenario where a single organization or device is not participating in a FL setup but instead trains its own centralized model using its own private data. We consider organizations of different sizes that have different training data availability. We train a centralized model (no federation present) of the same architecture and parameters as in the previous FL setups with different dataset sizes. Table 4.5 summarizes the results. 84 Federated Learning for Malware Detection in Edge devices Table 4.5 Accuracy on test set of centralized models for different dataset sizes. Number of samples Accuracy Precision Recall F1 Score 5000 0,8443 0,8299 0,8662 0,8477 10000 0,8482 0,8428 0,8828 0,8623 50000 0,8949 0,8791 0,9156 0,897 100000 0,9126 0,904 0,9232 0,9135 600000 0,9338 0,9271 0,9417 0,9343 Figure 4.6 plots the accuracy for the centralized (non-federated) system as a function of the dataset size. The blue line denotes the best FL accuracy we measured in our experiments for reference. The results indicate that a data set size of 600K is necessary in the centralized (non-federated) case for achieving high accuracy that reaches above 93% and matches the accuracy of the FL system. This result is the reference accuracy towards which we evaluate the performance of the federated system cases. Importantly, when considering Figure 4.3 as well, the plot demonstrates the benefit of FL for small organizations and near-edge devices with limited data availability that do not have access to large datasets to train centralized models. We also note that even that even organizations and devices that 12 50000 0,8949 0,8791 0,9156 0,897 100000 0,9126 »0,904 0,9232 0,9135 600000 0,9338 0,9271 0,9417 0,9343 Figure 1.6 plots the accuracy for the centralized (non-federated) system as a function of the dataset size. The blue line denotes the best FL accuracy we measured in our experiments for reference. The results indicate that a data set size of 600K is necessary in the centralized (non-federated) case for achieving high accuracy that reaches above 93% and matches the accuracy of the FL system. This result is the reference accuracy towards which we evaluate the performance of the federated system cases. Importantly, when considering Figure 1.3 as well, the plot demonstrates the benefit of FL for small organizations and near-edge devices with limited data availability that do not have access to large datasets to train centralized models. We also note that even that even organizations and devices that have access to large datasets can benefit from FL, as a model generated with contributions from multiple participants is trained on a potentially more diverse dataset with malware and benign samples coming from different networks. Figure 1.6 Centralized learning model’s performance for different dataset sizes. 0.8 0.82 0.84 0.86 0.88 0.9 0.92 0.94 0.96 5,000 10,000 50,000 100,000 600,000 Accuracy Dataset size Accuracy on test set Best federated accuracy Figure 4.6 Centralized learning model’s performance for different dataset sizes. 5.1 Introduction and Background 91 calls will be impractically slow. As a result, running f with a parameter set x is time consuming, and any solver for tuning problem should find optimal parameters with the smallest number of iterations. An example of optimization problem is Linear Programming (LP) where f, g, and h are linear function and define a polyhedron. Changing parameter x will move a linear function across the feasibility set as show in Figure 5.2. 5.1.4 Static and Dynamic Parameters in ISP Different algorithms utilized in ISP which have individual parameters. Each algorithm tries to attenuate artifacts from a specific source: a) Static Parameters: Algorithms responsible for improving artifacts originated from camera sensor have static parameters which should be tuned only once. b) Dynamic Parameters: Algorithm responsible for improving artifacts due to light condition and environmental phenomena have dynamic parameters which should be updated during runtime. The static parameters which are related to camera sensor characteristics can be tuned once for the specific camera. After tuning the ISP for the specific camera, the parameters can be fixed in configuration file for deployment. Tuning dynamic parameters improves the image quality in different environmental conditions. The dynamic parameters should be updated during runtime to guaranty best image quality performance. 5.1.5 State of Art The tuning process is done manually by experts. Each ISP algorithm is responsible for reducing specific artifact in image and the expert can measure the artifact intensity using a specific KPI. Then by changing the ISP parameters and try and error, the expert can find best parameters combination for specific camera sensor. For tuning dynamic parameters, expert do the same process, but for a range of environmental conditions. That means, first a set of input ISP images (Bayer Pattern) are captured from sensor in different environmental condition, then expert should tune the ISP for each one of the input images. ISP has an algorithm for gathering the statistical data for Bayer pattern image. Having the statical data for all images and corresponding optimal ISP parameters, one can make a “decision tree” which changes the parameters on flight based on statistical data provided by ISP. 92 Image Signal Processor (ISP) Tuning using Machine Learning (ML) methods 5.2 Automatic ISP Tuning The automatic tuning process should be done for both Dynamic and Static parameters. 5.2.1 KPIs for Artifact Attenuation For measuring ISP performance, various KPIs should be defined. Each KPI measures the intensity of a specific artifact in image. It should be noted that there is no one to one relation between artifacts and ISP algorithms. In many cases one KPI could be used for tuning multiple ISP algorithms. The list of KPIs is [4]: Table 5.1 KPIs for Measuring Image Artifacts. Artifact ISP block KPI Noise Noise Reduction Block(s) PSNR Loss of detail Sharpness Correction MTF50 Color Inaccuracy Color Correction Matrix (CCM) ∆E Color Casting White Balancing ∆E 5.2.2 Static Parameters The setup is done once in lab and a camera is attached to the capturing device. The captured Bayer pattern is fed to the ISP as input. The ISP generates an image. For measuring the performance of specific ISP algorithm in attenuating an artifact, corresponding KPI in Table 1 is used. ISP tuner can track KPI value for judging performance result of a set of parameters. ISP tuner changes the ISP parameters in multiple iterations and tries to optimize parameters based on KPI value. The process is iterative, and the iterative process is needed to be done only once for static ISP parameters as shown in Figure 5.3. The optimal value found by Tuner will be stored as fixed configuration for runtime. Figure 5.3 Tuning ISP Static Parameters. 5.2 Automatic ISP Tuning 93 5.2.3 Dynamic Parameters and Runtime Achieving optimal performance with dynamic parameters is harder. The parameters should be re-tuned to adapt environmental effects such as light condition, temperature, etc. There are limitations in using same iterative approach for tuning static parameters: 1. The iterative approach makes it impossible to have optimal parameters per frame or even per minute. 2. Measuring the KPI during runtime is challenging since there is no reference for the scene captured by camera. A proposal solution is to use a machine learning model which can map image statistical data to optimal parameters. All ISPs measure image statistical data and provide it per frame. The dataset can be created by utilizing same tuning procedure mentioned for tuning statistical parameters in a loop as demonstrated in Figure 5.4. E: All desired environmental conditions which ISP should operate in. For e in E { 1. Run tuner for optimizing ISP. 2. Store Optimal ISP parameters + ISP statistical data for the scene. } Figure 5.4 Dynamic Parameters Data Generation. In theory, a Neural Network (NN) should be able to predict optimal values for dynamic parameters after training; however, an NN creates a high computation load during inference, so a more efficient solution is needed. 94 Image Signal Processor (ISP) Tuning using Machine Learning (ML) methods In this paper, a gradient boost model is proposed for inferencing the optimal parameters. Gradient boosting models are trainable decision trees. Unlike NN which use Directed acyclic graphs (DAGs) as underlaying data structure for training and inferencing, gradient boost (GB) utilizes trees which are simpler data structures [5]. GB models are fast to inference and has similar performance as NNs for tabular data which exactly matches the use case and dataset we have for dynamic tuning application. Figure 5.5 Storing Optimal Parameters and ISP Statistical Data for Training ML model. 5.2.4 Test Setup Testing the proposed methods for tuning static and dynamic parameter are done with Solectrix SoftISP SXIVE1. The ISP runs on PC with dedicated graphic card. The tuner uses 16 cores CPU to speed up the process in lab for finding static parameters in ISP. Same tuner is utilized to create a dataset of ISP statistical data and optimal parameters for ten lighting color temperature. The dataset is then used for training a GB model named XGboost as runtime. The trained model then runs on single core with minimum load on the same machine to update White Balancing parameters. 5.2.5 Results The demo software (SW) is instantiated with init button. ISP with random parameters is run and the ISP output image is shown to the user. The user can select the boundaries of the Color Checker (CC) board inside the image then press “Tune”. The SW will crop the image to find cc board and samples 1sxive.com 5.2 Automatic ISP Tuning 95 patches from the board and shows the sampling areas to the user (Annotated CC). Then, tuning process begins. In Figure 5.6 the process for finding a better optimum point is illustrated as tuner progress. The measured ∆E for all color blocks in color checkerboard is calculated and aggregated as Mean Squared Error (MSE) of ∆E values: MSE(∆E) = 23 X k=0 (∆Ek)2 24 . The SW results in Figure 5.6 can be interpreted as: “ISP output” shows ISP output image for the current iteration, and “Best Config” shows the best result found by tuner until the current iteration. When tuner iterates over different configurations, it generates various “ISP outputs” Figure 5.6 Offline Tuning Results for Color Correction Matrix Tuning. 96 Image Signal Processor (ISP) Tuning using Machine Learning (ML) methods and corresponding KPI values; Based on observed KPI value, tuner guesses a better parameter set for the next iteration. As iterations go on, tuner can find better parameter sets which results in better KPI values, so as it can be seen, the “Best Config” image is improved when tuner progresses. White balancing (WB) is the algorithm chosen for dynamic tuning. Same tuner finds optimal parameter for various light conditions and intensities in No WB XGboost WB Figure 5.7 Runtime Result of Trained XGboost WB. References 97 lab in a loop as explained in Figure 5.4. The generated dataset maps image histogram to optimal WB configuration is used to train a XGboost model. The ISP has its own Automatic White Balancing (AWB), but we turned it off to show the effectiveness our of XGboost WB. The results are shown in Figure 5.7 without and with white balancing under blue light source. 5.3 Conclusion Tuning ISP was conventionally a cumbersome, costly, and suboptimal task. The iterative process should have been done for combination of numerous ISPs and cameras. In pursue of a more automated solution, the proposed method tries to distinguish parameters based on static/dynamic nature. The parameters which are related to specific camera, can be tuned in lab, and the optimal parameters will be fixed as static parameters. For ISP algorithms which attenuates environmental impact on image quality, a runtime should fine-tune the ISP in the field. The runtime algorithm should be light enough to run on a restricted HW processor. For tuning both static and dynamic parameters, the paper presents a tuning framework to automate the process. A tuner finds optimal parameters for static parameters. A data generation pipeline utilizes same tuner in a loop for various environmental conditions. The generated data maps the statistical data provided by ISP to optimal parameters found by tuner. In the next steps, the trained GB model based on the generated dataset is used as a lightweighted runtime for tuning dynamic parameters in changing environmental condition. References [1] D. Molloy et al., “Impact of ISP Tuning on Object Detection,” Journal of Imaging, vol. 9, no. 12, pp. 260-260, Nov. 2023, doi: https://doi.org/10.3 390/jimaging9120260. [2] S. Boyd and L. Vandenberghe, “Convex Optimization,” Mar. 2004, doi: https://doi.org/10.1017/cbo9780511804441. [3] https://en.wikipedia.org/wiki/Linear_programming. [4] https://www.imatest.com/support/docs/23-2/colorcheck/ [5] T. Chen and C. Guestrin, “XGBoost: a Scalable Tree Boosting System,” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’16, pp. 785-794, 2016, doi: https://doi.org/10.1145/2939672.2939785. 6 Using Edge AI in IoT devices for Smart Agriculture: Autonomous Weeding Christian Germain1,2, Barna Keresztes1,2, Aymeric Deshayes1, and Jean-Pierre Da Costa1,2 1University of Bordeaux, CNRS, France 2Bordeaux Sciences Agro, France Abstract This paper presents the evolution of a vision system dedicated to automatic weeding, initially implemented on a NVIDIA Jetson Xavier board. This evolution aims to take advantage of a new computing board able to implement efficient artificial intelligence oriented computations, keeping a low power consumption, and a low cost, developed in the ANDANTE project. The paper presents the automatic weeding tool, the existing vision system and the weeding data used to train the system. It also describes the specifications of the new board and the adaptation needed in order to integrate the previous algorithm in this new board. The results obtained during the first step of this integration are presented and compared to those obtained with the previous vision system. These new results are encouraging and rich in lessons for the future. Keywords: edge computing, precision agriculture, smart agriculture, automatic weeding, image processing, deep learning. 6.1 Introduction Agriculture has to face many challenges in the 21st century. With the increasing artificialization of land and the augmentation of the global population, we have to produce more food using less surface. Another challenge in order to 99 100 Using Edge AI in IoT devices for Smart Agriculture: Autonomous Weeding preserve our natural resources and soil quality for agriculture, is to produce differently, with less inputs (fertilizers, phytosanitary products, herbicides...). In addition, climate change has a huge impact on production, yields, water availability and many more aspects. To address those challenges, several solutions can be proposed, among which are smart farming, the usage of digital technologies in agriculture and precision agriculture, which can be summarised as applying to the crops the appropriate action, at the best moment, and at the right place and quantity. In order to make those improvements in crop management, two key technologies can provide a significant help: in-field connected sensors and robotic processing of the crops. In-field connected sensors have proven to be very useful for collecting data on the plots (vegetation index, soil composition, weather parameters...). These data are then processed and integrated in decision support systems to help the farmer manage the crops. Installing sensors in the field is not an easy task: outdoor conditions require robust material that can resist moisture, dust and shocks. Moreover, access to a power supply is not practical, so it is important to have low consumption devices, which limits the processing power available. Cloud computing could offer a solution: the sensor sends the data via internet in order to process it on a server. However, internet access is often limited in the fields and the amount of data can be large (images or videos). Another solution could be to use a long-range technology such as LoRa or SigFox. However, those technologies have a limited data rate and can’t support huge amounts of data to send in the cloud. Edge computing is a promising alternative, with the possibility to make computing on board, dramatically reducing the amount of data to be send (only the results), via LoRa for example. This makes it possible to use a connected sensor even in areas with poor network access. It also allows to reduce the power consumption involved in the communication in case of large amount of data. However, this latter advantage can be neutralized by the energy cost of the calculations carried out on board. In the case of robotic processing of the crops, embedded sensors are necessary to provide real-time data to the system so that it can implement the operations needed to process the culture. This real time constraint favours the edge architecture, avoiding loss of time in data transfer and reception of results, especially for significant input data quantity (image or video). In both cases, the constraints are similar: processing signals, images or videos of natural scenes require complex computations; the return on investment expected for the farmers limits the cost of the technologies used, 6.4 Work in Progress and Future Work 107 available development tools. The Mobilenet-based SSD detector was implemented using the Pytorch library, with separate classes for the backbone and the detection head. A simulator for the NeuroCorgi circuit was implemented on the N2D2 platform [14], which is a deep learning framework for creating artificial neural networks intended to work on constrained environments. The SSD network head was translated for the Kria KV260 FPGA using the VitisAI library. Figure 6.6 The adapted network architecture used for this application. The figure presents how the duplicated Mobilenet layers and the SSD head are connected to the NeuroCorgi backbone. Some modifications to the network structure were necessary: as the training database for the encoder contains few examples of plants, the initial detection results were inadequate. Duplicating some of the encoder layers on the FPGA and making them trainable improved considerably the detection accuracy. The resulting network architecture is presented in Figure 6.6. Table 6.3 shows the first results from the proposed architecture (these results are expressed in terms of loss function but the precision performances will be available soon). These results are promising, even if the SSD architecture is less precise that the reference Yolo V4 network. They also show 108 Using Edge AI in IoT devices for Smart Agriculture: Autonomous Weeding that it is necessary to use at least a partially retrained backbone. However, the improvement after retraining the first layers is marginal. Table 6.3 Detection performance (loss function) using the new architecture. Network training Loss on SSD Loss on SSD Lite Head only 3.8 4.6 Partially retrained backbone 1.8 1.9 Retrained backbone 1.5 1.7 Figure 6.7 presents examples of detection obtained by the reference system and by the proposed network. Figure 6.7 Results from the Yolo V4 network (left) and the proposed SSD network (right) on maize. Blue rectangles show the plants. Green rectangles show the stem locations. 6.4.2 Future work The two parts of the network are currently being transferred on the Platform 4.1a. The performances, in terms of accuracy, processing time and power 6.5 Conclusion 109 consumption will then be measured and compared to the reference (NVIDIA Jetson Xavier board). If the accuracy and computing time are adequate, the implementation inside the BIPBIP weeding system will then be possible, allowing field testing. 6.5 Conclusion In this paper we presented a vision system for automatic weeding (BIPBIP platform), and described the objectives and the progress of a project to evolve this vision system, through the integration of an Artificial Intelligence oriented computation board with low cost and low power consumption (ANDANTE project). The existing vision system (BIPBIP) should allow easy hardware integration by replacing the NVIDIA Jetson Xavier with the new circuit. However, an adaptation of the Convolutional Neural Network model appeared to be necessary. Encouraging simulations have shown the overall feasibility of the transfer, and have been very informative, particularly about the need to adapt the initial architecture of the circuit to achieve the expected precision performance expected for weed control applications. Furthermore, the availability of the new ANDANTE circuit makes it possible to address other “smart agriculture” use case such as a fixed vineyard monitoring vision sensor. The integration of the new ANDANTE board, even in its current architecture, should make it possible to improve the very simple vision processing algorithms carried out on board the existing prototype, while keeping a low power consumption inherent to this type of device, thus allowing to extend its uses. Acknowledgements The ANDANTE project has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 876925. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Belgium, France, Germany, The Netherlands, Portugal, Spain, Switzerland. www.andante-ai.eu. The BIPBIP project has been funded by the French Research Agency (ANR) (grant ANR-17-ROSE-0001 - BIPBIP) and has been supported by the organizers of the ROSE Challenge and all the partners of the BIPBIP project. 110 Using Edge AI in IoT devices for Smart Agriculture: Autonomous Weeding References [1] L. Lac, J-P. Da Costa, M. Donias, B. Keresztes, A. Bardet, “Crop stem detection and tracking for precision hoeing using deep learning”. Computers and Electronics in Agriculture, 2022, 192:106606. [2] L. Lac, “Méthodes de vision par ordinateur et d’apprentissage profond pour la localisation, le suivi et l’analyse de structure de plantes : application au désherbage de précision”, PhD Thesis, Université de Bordeaux, 2022. [3] Lemken. ‘IC-Weeder: Automatic intra-row hoeing machine for vegetables’. Accessed 21 June 2024. https://lemken.com/en-en/agriculturalmachines/cropcare/weed-control/mechanical-weed-control/ic-weeder. [4] Garford Farm Machinery. ‘Robocrop InRow Weeder’. Accessed 21 June 2024. https://garford.com/products/robocrop-inrow-weeder. [5] VisionWeeding. ‘Mechanical Robovator’. Accessed 21 June 2024. https: //www.visionweeding.com/robovator-mechanical/. [6] B. Jiang, J-L. Zhang, W-H Su, and R. Hu. ‘A SPH-YOLOv5x-Based Automatic System for Intra-Row Weed Control in Lettuce’. Agronomy 13, no. 12 (Dec. 2023): 2915. https://doi.org/10.3390/agronomy131229 15. [7] M. Pérez-Ruiz, D.C. Slaughter, C.J. Gliever, and S.K. Upadhyaya. ‘Automatic GPS-Based Intra-Row Weed Knife Control System for Transplanted Row Crops’. Computers and Electronics in Agriculture 80 (1 January 2012): 41–49. https://doi.org/10.1016/j.compag.2011.10.0 06. [8] A. Bochkovskiy, C.Y. Wang, H.Y.M. Liao, “YOLOv4: Optimal Speed and Accuracy of Object Detection”. in arXiv:2004.10934, 2020. [9] I. Miro-Panades, I. Kucher, V. Lorrain, A. Valentian, “Meeting the latency and energy constraints on timing-critical edge-AI systems”, in International Workshop on Embedded Artificial Intelligence Devices, Systems, and Industrial Applications (EAI), 2022. [10] I. Miro-Panades, E. Romay, L. Mateu Saez, M. Diaz Nava “Platform 4.1a : A Multi-Application Platform Supporting Several Uses Cases in the Domains Digital Farming and Transport and Smart Mobility”, European Conference on EDGE AI Technologies and Applications -EEA, 17_10 October 2023 Athens, Greece. [11] A.G. Howard, M. Zhu, B. Chen, D Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications”, https://arxiv.org/abs/1704 .04861, 2017. References 111 [12] W. Liu, D. Anguelov, D., Erhan, C. Szegedy, S. Reed, C.Y. Fu, A.C. Berg, “SSD: Single shot multibox detector”. In Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Springer International Publishing, 2016, Proceedings, Part I 14, pp. 21-37. [13] T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, C.L. Zitnick, “Microsoft COCO: Common Objects in Context”, in Computer Vision– ECCV 2014. Springer International Publishing D. Fleet, T. Pajdla, B. Schiele, T. Tuytelaars, Eds., vol. 8693, 2014, pp. 740–755. [14] CEA-LIST. “N2D2: Neural Network Design & Deployment”, in github.com, 2017, https://github.com/CEA-LIST/N2D2. Index A advanced driver assistance system (ADAS) 90 agentic AI 34 AI-defined vehicles (AVD) 1, 33, 39 Aidge xi, 41, 43, 45, 46, 48, 52 artifact attenuation 92 artificial intelligence network on chip (AINoC) 58, 60, 61 automatic weeding 99, 101, 109 autonomous vehicle xi, 1, 2, 3, 6, 11, 15, 16, 17, 32 C camera tuning 89 CNN accelerator 55 convolutional neural network (CNN) 54, 55, 71, 88, 103, 109 D data distribution service (DDS) 27 data reuse 44, 55, 58, 63, 69, 70 dataflow architecture xi, 55, 56, 57, 58, 60 deep learning 33, 42, 43, 56, 86, 90, 101 deep learning accelerators (DLAs) 43 deep neural network (DNN) xi, 41, 42, 71, 73, 118 denial-of-service (DoS) 76 deployment and compilation 42 DNN optimisation 42 dynamic driving task (DDT) 5 dynamic random access memory (DRAM) 57 E edge AI xi, 1, 3, 26, 55, 99 edge computing xi, 3, 15, 26, 87, 100 edge devices xii, 44, 73, 77, 84 electronic control unit (ECU) 18 embedded dynamic random access memory (eDRAM) 57 energy consumption 57, 58, 66, 67, 69 energy efficiency 26, 57, 69, 70 explainable edge AI (XAI) 33 F federated learning 73, 75, 77, 80, 81, 85 field-programmable gate array (FPGA) 26 G graph manipulation 42 graphics processing unit (GPU) 26 H hardware accelerator 54, 105 hardware export 41, 44 I image processing 54, 99 113 114 Index image signal processor (ISP) xii, 89, 90 inertial measurement unit (IMU) 37, 38 inertial navigation systems (INS) 14 interconnect xi, 3, 49, 55, 56, 58 Internet of Robotic Things (IoRT) 3, 36 Internet of Things (IoT) 7, 73, 87, 117 interpretable edge AI (IAI) 33 ISP 89, 90, 91, 92, 93, 97 K Keras 17, 48 L last-mile delivery xi, 1, 15, 17, 25, 32 latency 9, 14, 43, 57, 67, 68, 85 light detection and ranging (LiDAR) 1, 4, 8, 9, 35 long range radio (LoRa) 100 low power xii, 9, 44, 53, 85, 99, 101 M machine learning xii, 16, 23, 28, 41, 72, 73, 75, 85, 89 malware detection xii, 73, 74, 75, 77, 85 micro-electro-mechanical systems (MEMS) 12 ML 16, 73, 74, 75, 89, 94 N network optimization 49 neural global controller (NGC) 58, 59, 60 neural processing element (NPE) 58, 59, 60 neural processing unit (NPU) 26 O object recognition 2 odometry 12, 23, 24, 36 open neural network exchange (ONNX) 42, 43, 48, 53 OpenCV 17, 48 operational design domain (ODD) 4, 5, 37 operational technology (OT) 73 P path planning 23, 26, 36, 40 perception xi, 1, 4, 9, 12, 15, 16, 33, 101 platooning 13, 18, 19, 20 point cloud 8, 10 precision agriculture 100 pruning 53 Python 17, 42, 45, 46, 103 PyTorch 17, 26, 28, 42, 48, 79, 107 Q quality of service (QoS) 29 quantization 41, 44, 50, 53 quantization aware training (QAT) 53 R radar 1, 4, 7, 8, 9, 11 real-time kinematic (RTK) 14 real-time operating system (RTOS) 26 reduced instruction set computing (RISC) 53, 55, 67, 69 roadside units (RSUs) 13 robot operating system (ROS) 4, 27, 37, 39 ROS 1 27, 29 Index 115 ROS 2 27, 29, 30, 31, 33 ROS middleware (RMW) 31 S sensor fusion xi, 1, 4, 7, 12, 16, 23, 29, 33, 38 SimpleCV 17 simultaneous localization and mapping (SLAM) 12 small language models (SLMs) 34 smart agriculture xii, 99, 101, 109 software-defined vehicle (SDV) 1, 7, 33, 39 static random access memory (SRAM) 57 System-on-a-Chip (SoC) 26 T tensor processing unit (TPU) 26 TensorFlow 17, 26, 28, 42, 43 transformers 33, 53 tuning xii, 21, 42, 89, 90, 93, 95 V vehicle control unit (VCU) 18 vehicle-to-everything (V2X) 1, 7, 9, 13, 33 vision system 99, 101, 102, 103, 105, 106 vulnerable road users (VRUs) 8, 10, 33 W weeding system xii, 101, 102, 109 X XGboost 94, 96, 97 Y You Only Look Once (YOLO) 20