D2.11 - Report on Remotisation and Simulation tools

REPORT ON REMOTISATION AND SIMULATION TOOLS 2024-12-23 Ref. Ares(2024)9286182 - 31/12/2024 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 2 Project cofunded by the European Commission within the Digital Europe Programme Dissemination Level PU Public X CO Confidential, only for members of the consortium (including the Commission Services) □ CL Classified, as referred to in Commission decision 2001/844/EC □ Deliverable number: D2.11 Deliverable name: Report on Remotisation and Simulation tools Work package: WP2 Lead WP: POLIMI Lead Task: RISE 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 3 Contents Document Revision History ............................................................................................................................................... 5 Abstract ............................................................................................................................................................................. 6 Introduction ...................................................................................................................................................................... 7 Outline........................................................................................................................................................................... 7 AgrifoodTEF simulation and remotisation tool survey methodology: .......................................................................... 8 Simulation Software .......................................................................................................................................................... 9 Crop simulation ............................................................................................................................................................. 9 Unity .......................................................................................................................................................................... 9 Robotic simulation ...................................................................................................................................................... 13 Webots .................................................................................................................................................................... 13 Gazebo .................................................................................................................................................................... 15 4D-Virtualiz / 4DV-SIMULATOR ............................................................................................................................... 18 Unreal Engine 5 ....................................................................................................................................................... 18 ISAAC Sim ................................................................................................................................................................ 19 Esmini ...................................................................................................................................................................... 19 CoppeliaSim VREP ................................................................................................................................................... 21 Autonomous driving.................................................................................................................................................... 22 CARLA (UE4) ............................................................................................................................................................ 22 Farm simulation .......................................................................................................................................................... 23 Digital Future Farm ................................................................................................................................................. 23 Summary ..................................................................................................................................................................... 24 Remotisation Software ................................................................................................................................................... 24 Remote desktop access ............................................................................................................................................... 24 VNC (Virtual Network Computing) for AI Model training ....................................................................................... 24 Remote tool access ..................................................................................................................................................... 25 FarmBot ................................................................................................................................................................... 25 OpenAG ................................................................................................................................................................... 26 Summary ..................................................................................................................................................................... 26 Key characteristics to consider for choosing tool selection ............................................................................................. 27 Discussion and future work ............................................................................................................................................ 28 Timeframes and next steps: ........................................................................................................................................ 28 References ...................................................................................................................................................................... 29 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 4 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 5 Document Revision History Date Issue Author/Editor/Contributor Summary of main change 2023-11-29 V0.3 Editor: Victor Kardeby - RISE Document Draft 1 2023-12-20 V0.6 Editor: Victor Kardeby - RISE Document sent for internal review 2023-12-21 V0.9 Editor: Victor Kardeby - RISE Document sent for external review 2024-01-04 V1.0 Editor: Victor Kardeby - RISE Document published 2024-06-26 V1.1 Editor: Victor Kardeby - RISE Document revised based on feedback 2024-08-29 V1.2 Editor: Victor Kardeby - RISE Document reviewed before resubmission 2024-11-04 V1.3 Editor: Victor Kardeby - RISE Work with new content prior to next release 2024-11-25 V1.5 Editors: Henrik Abrahamsson – RISE, Nishat Mowla – RISE Added input from contributors 2024-12-13 V1.9 Review: Ankur Mahtani - LNE Internal review 2024-12-31 V2.0 Editor: AgrifoodTEF Year two submission for publishing 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 6 Abstract The AgrifoodTEF project is a collaborative network of test and validation infrastructures across Europe, designed to support Agri-Food technology companies in the near-product development of AI and robotics solutions within realworld agricultural settings. The project aims to bridge the gap between cutting-edge research and practical applications, enhancing efficiency and sustainability in agriculture while meeting the stringent usability and economic needs of end-users. Rooted in existing experimental farms and facilities, which are now operational, the project fosters a scalable approach to ensuring food security in the EU by engaging stakeholders and leveraging expert knowledge in AI and robotics. This document presents a catalogue of simulation and remotisation tools that facilitate the creation, visualization, and manipulation of virtual models of agrifood systems, as well as the remote control and monitoring of real-world processes. The report details the tools currently in use by project partners, providing insights into their applications and effectiveness. It concludes with a discussion of findings and outlines future work aimed at continuous improvement and adaptation of these tools to meet evolving agricultural challenges. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 7 Introduction The AgrifoodTEF project represents a significant initiative aimed at enhancing the agricultural sector through the integration of advanced technologies, particularly artificial intelligence (AI) and robotics. As a collaborative network of test and validation infrastructures across Europe, AgrifoodTEF supports technology companies in the near-product development of innovative solutions tailored for real-world agricultural applications. • The primary objective of AgrifoodTEF is to bridge the gap between cutting-edge research and practical implementation, fostering sustainable agricultural practices that meet the growing demands for efficiency and productivity. This initiative is particularly relevant in the context of increasing global food security challenges, as it leverages existing operational experimental farms and facilities. Standards for dataset creation, data sovereignty, and algorithm interoperability will be guided by GAIA-X, with representatives from various agricultural domains in Europe contributing to this framework. • Each TEF-node will operate independently with a sustainable business model tailored to regional needs, while collectively adhering to shared guidelines and standards that facilitate cross-regional collaboration. The aim is to position Europe as a leader in the adoption of innovative, technology-driven solutions for Agri-Food, enhancing efficiency in primary production akin to the advancements achieved during the Green Revolution. This report serves as the second annual update on the progress made in identifying and cataloging simulation and remotisation tools utilized by project partners. It introduces the concepts and tools that are evaluated and used within AgrifoodTEF. These tools are essential for creating, visualizing, and manipulating virtual models of agrifood systems, as well as for enabling remote control and monitoring of agricultural processes. In the second year of the AgrifoodTEF project, we continued a seminar series about simulation tools that was initiated in the first year of the project. The seminars explore the practical applications of the different tools, discuss upcoming and possible AgrifoodTEF services, and map the tools to these services. The seminars serve as a platform for a more comprehensive evaluation of the available tools and an opportunity to identify new tools that meet agricultural requirements. The insights gathered from this report will not only highlight the current landscape of tools in use but also provide a foundation for future developments and enhancements in agricultural technology, aligning with the objectives of other tasks within the project. This second version of the report includes ten simulation tools and three remotisation tools (compared to eight plus three in the first report). This second report includes more detailed and specific descriptions of the tools. For many of the tools there are now case studies of how the tool is used, in an agricultural example, with analysis of the tool's pros and cons. As part of dissemination activity related to the content of this report, an AgrifoodTEF seminar series is organized where the partners showcase various simulation and remotization tools for agricultural applications. To date, three seminars have been conducted: 1) Gazebo by Politecnico di Milano and ROS simulator by WUR, 2) 4D-Virtualiz, Isaac Sim, Carla, Simulink 3D Animation by LNE and ExtendSim by UdL, 3) Robot simulators by ILVO. Outline This report details the findings and methodologies related to the simulation and remotisation tools utilized within the AgrifoodTEF project. The Simulation Software section provides an overview of various tools, including their functionalities and applications, supported by case studies that illustrate their effectiveness in agricultural contexts. The Remotisation Software section similarly explores tools that facilitate remote control and monitoring of agricultural processes, detailing their integration and practical use cases. The Discussion section synthesizes the key findings from the previous sections, highlighting their implications for the agricultural sector and identifying areas for improvement. Following this, the Future Work section outlines the plans for ongoing evaluation and development of these tools, emphasizing the project's commitment to continuous enhancement and adaptation to meet evolving agricultural challenges. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 8 AgrifoodTEF simulation and remotisation tool survey methodology: As part of the activities to identify simulation and remotisation tools in T2.3, RISE conducted a comprehensive survey to understand the current landscape and future potential of these tools as key enablers in agricultural technology. The survey aimed to gather valuable insights from partners, focusing on evaluating the effectiveness, challenges, and reusability opportunities of the tools. The primary objectives of the survey were to: • Assess application areas: Identify where simulation and remotisation tools are currently utilized within agrifood technologies. • Evaluate adoption rates: Determine how widely these tools have been adopted by partners and their integration into existing agricultural practices. • Determine impact: Assess the impact of these tools on improving efficiency, productivity, and sustainability in agriculture. The survey was designed using a mix of quantitative and qualitative questions to capture detailed feedback on current tool practices, user experiences, and the number of tools in use per partner. This methodology provided a comprehensive understanding of how simulation and remotisation tools are concretely applied in the agrifood sector, guiding strategic decisions to enhance efficiency and sustainability in agricultural practices. Some questions were mandatory, requiring a ‘yes’ or ‘no’ response (e.g., ‘Do you use any simulation/remotisation software?’ as shown in Figure 1) to assess the current state of tool usage. Follow-up questions, which were not mandatory, aimed to gather additional information about specific tools if the answer was affirmative. Additionally, open-ended questions were included to identify types of simulation tools used and to collect feedback on any new types of simulation and remotisation tools available based on partner experiences. The survey also explored various aspects such as the extent of remote operation integration, benefits realized, tool preference on a scale of 1 to 5, tool accessibility, and partner plans regarding the tools. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 9 Figure 1 Survey question to identify the different purposes of simulations in AgrifoodTEF The survey link was distributed to a diverse group of partners in the agrifood sector, including researchers and industry experts, to assess their current use of simulation and remotisation tools. Our intention was to identify tools that are already in use by partners to evaluate their suitability for remote experiments in the agricultural landscape. The findings from the survey provide a comprehensive understanding of how these tools are practically applied in the agrifood sector, guiding strategic decisions aimed at enhancing efficiency, reducing costs, and improving productivity and sustainability in agricultural practices. Simulation Software Simulation software serves as a platform for creating virtual models and environments for testing, training, and analysis purposes. These tools allow to replicate real-world scenarios within a controlled digital space, facilitating experimentation, design, and training without the constraints or risks of physical testing. Common applications include engineering simulations, flight and driving simulators, medical training, and 3D modeling. Simulation software often includes realistic physics engines, graphical interfaces, and support for data analysis, catering to various industries like aerospace, automotive, healthcare, and entertainment. The advantage of using simulation software is that it enables users to refine designs, train personnel, and conduct research in a cost-effective and safe manner. Crop simulation Unity Unity is a cross-platform game engine developed by Unity Technologies. The engine can be used to create 2Dand 3Dgames, and interactive simulations. The engine has been adopted by industries outside video gaming, such as film, automotive, architecture, engineering, and construction 1 . Unity is proprietary software that offers free versions for individuals and companies. The Unity Perception package[unity-perception2022] provides a toolkit for generating large-scale datasets for computer vision training and validation. The package is open-source and licensed under Apache License Version 2.0. Case study: Sim2real flower detection towards automated Calendula harvesting This case study, from EV ILVO, demonstrates how synthetically generated data can aid deep learning and computer vision, and it exemplifies this in the context of automating the detection and localisation of Calendula flowers. To this 1 https://en.wikipedia.org/wiki/Unity_(game_engine) 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 16 shadows, and textures. Additionally, it can model a variety of sensors, including laser range finders, cameras (wideangle included), Kinect-style sensors, and more. Figure 6 Gazebo simulator with vineyard models from the BACCHUS project repository (https://github.com/LCAS/bacchus_lcas) Politecnico di Milano uses Gazebo to develop and test robotic navigation algorithms that exploit multimodal sensor input. A recent project at Politecnico di Milano serve as an example of how Gazebo can be used. Case study: Using Gazebo for development of an autonomous lawn mower In this case, we aimed to convert a robotic lawnmower prototype into a fully autonomous system capable of cutting grass in various outdoor environments. This process involved integrating sensors and developing a robust autonomous navigation system. Several key challenges emerged during the development. 1. First, we needed to identify the optimal sensor choice and configuration without prematurely investing in expensive hardware. 2. Second, we wanted to test our localization algorithms without immediate access to the physical robot, as deploying it in the field is time-consuming and logistically challenging. 3. Finally, we needed to safely test our navigation algorithms, avoiding the risks of damaging the robot, the environment, or harming people nearby. To address these challenges, we adopted Gazebo as one of the core tools in our development process. It allowed us to simulate the robot and its environment, facilitating sensor selection and algorithm testing in a risk-free virtual setting. Using this platform, we could experiment iteratively before transitioning to real-world trials. Our development workflow began with creating a simulated model of the robotic lawnmower. We also designed multiple virtual environments, including open fields and photovoltaic parks, to reflect the diverse outdoor settings where the robot would eventually operate. Once the model and environments were in place, we integrated various sensors into the simulated lawnmower. For example, we tested different configurations of LiDAR sensors by adjusting 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 17 parameters such as range, field of view, and placement on the chassis, aiming to enhance the robot's perception of its surroundings. Through these simulations, we measured the performance of the localization algorithms, using sensor fusion techniques to combine data from multiple sources and refine the robot's positioning accuracy. Additionally, we focused on testing and optimizing the autonomous navigation algorithms. By running these algorithms in the simulated environments, we could explore different parameter settings to fine-tune the system's behavior. This iterative approach helped us identify the most effective navigation strategies without risking real-world consequences. Once we achieved satisfactory results in the virtual tests, we transitioned to physical implementation. We integrated the chosen sensors into the real robotic lawnmower and deployed the software architecture. Confident in the system's reliability, we proceeded with live tests in actual agricultural fields. Gazebo was key to the success of this project. It provided critical features such as the ability to simulate sensor perception and run complex algorithms in real time. Its open-source nature allowed us to examine its code and extend its functionalities, if necessary, which proved valuable in addressing specific project needs. Moreover, Gazebo is lightweight and resource-efficient compared to other simulators, enabling us to use it on standard workstations without the need for specialized hardware. While Gazebo was highly effective for our robotic lawnmower project, we acknowledge its limitations when dealing with more complex environments that involve intricate physical interactions between the robot and the surroundings. For instance, simulating deformable objects, such as plants and the detachment of crops during selective harvesting with a robotic arm, would present significant challenges. Gazebo's capabilities in simulating such interactions are limited and would require more advanced or specialized simulators. Overall, Gazebo proved to be an indispensable tool in our development process. Its flexibility, accessibility, and ability to simulate real-world conditions allowed us to extensively refine our system before deploying it in the field. Despite some limitations in simulating complex physical interactions, it was highly effective for this project, and we achieved our results with it. Summary: • Used by: Politecnico di Milano, WUR • Link to the website: https://gazebosim.org/ • Platform features: Gazebo uses physics engines like ODE, Bullet and DART to simulate realistic physical interactions. It can support multi-robot simulations and is highly customizable. • Tool effectiveness: Gazebo enables physical designs in realistic virtual environments with a variety of sensor streams. The tool provides advanced 3D graphics, precise physics engine and multimodal sensors. Gazebo makes it possible to test control strategies in a safe virtual environment. As an example, Politecnico di Milano uses Gazebo to develop and test robotic navigation algorithms, and gave a score of 4 in the survey question of ‘how likely they would recommend Gazebo’ on a scale of 1 to 5 where 1 means ‘not at all’ and 5 means ‘very likely’. • Integration strategies: Gazebo can be integrated with software tools and robot operating systems (ROS) through its APIs and plugins to test robot operations in a safe virtual environment. • Cost analysis: The tool is open source. • Assumptions: Access to computer system with sufficient processing power and graphics capabilities to handle 3D simulations. • Skill requirement: Gazebo requires a basic understanding of robotics and simulation principles. There is documentation available online and a community for support (https://community.gazebosim.org) • Limitations: Performance may degrade with large-scale simulations, requiring high-end hardware. Also, environmental effects, such as rain, are not natively simulated. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 18 4D-Virtualiz / 4DV-SIMULATOR 4D-Virtualiz is a proprietary software used by LNE. The software simulates one or more robotic models into a 3D world. It has a physics engine which computes the world dynamics and produces accurate simulation of robot movements and its interactions with the environment. It can be used to assess the system robustness against challenging situations and produce stress tests, simulate sensor degradation, etc. LNE has developed a platform named Laboratory of Evaluation of AI (LE.IA) Simulation (purely numeric for virtual testing) to evaluate and assess agrifood robots using 4DV-Virtualiz to generate synthetic data of harsh environmental conditions such as foggy situations, and day/night images. A second infrastructure named LE.IA Immersion was developed for hardware-in-the-loop mixed testing of AI physical systems and models in their environment. Many scenarios including a digital twin of the Montoldre Farm (INRAE’s site) currently integrate the LEIA Immersion infrastructure. Summary: • Used by: LNE • Link to the website: https://www.4d-virtualiz.com/ • Tool effectiveness: 4DV-SIMULATOR is a real-time 3D simulation platform used for the development of robotic systems like vehicles, robots and drones. The tool enables realistic 3D virtual representations of operational use-cases. • Integration strategies: 4D-Virtualiz integrates with various robotics software and hardware to support realtime data exchange and co-simulation with other platforms, making it suitable for diverse research and development projects. • Cost analysis: 4D-Virtualiz requires a licensing fee. The cost varies based on the usage requirements and scale of the deployment. While the initial setup can be significant, the detailed simulations and robot testing capabilities can lead to cost savings by reducing the need for physical prototypes and minimizing risks in realworld testing. • Assumptions: Access to high performance computing resources to handle complex simulations. • Skill requirement: 4D-Virtualiz requires fundamental understanding of robotics, simulation principles and familiarity with programming. There is documentation and support available to assist practitioners. Unreal Engine 5 Unreal Engine 5's cutting-edge capabilities make it an ideal choice for applications where photorealism is crucial. However, the latest version still lags behind its predecessor (UE4) when it comes to seamless integration with AIrelated plugins. Despite this limitation, developers can leverage Unreal Engine 5's powerful features to create highly realistic environments, as demonstrated by LNE's project, which utilized UE5 to set up a photorealistic rural test site featuring a drivable vehicle surrounded by dynamic, living obstacles and traffic patterns. The use of blueprint systems and C++ event handlers allow the integration of AI systems, highlighting the potential of Unreal Engine 5 for complex simulations. Summary: • Used by: LNE • Link to the website: https://www.unrealengine.com/unreal-engine-5 • Tool effectiveness: Unreal Engine (UE) is a cutting-edge game engine that enables the creation of highly realistic 3D virtual environments. With its advanced capabilities, LNE has successfully developed a simulated farm scenario where navigation and obstacle detection algorithms can effectively integrate a moving vehicle. However, despite its impressive features, the latest version 5 may lack seamless integration with some plugins. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 19 LNE gives an average score of 4 in the survey question of how likely they would recommend UE5 on a scale of 1 to 5 where 1 means ‘not at all’ and 5 means ‘very likely’. • Integration strategies: UE5 can support ROS, however its integration is very recent and can be hard to configure. • Cost analysis: UE5 is free for academicals use. For development studios, royalty applies to gross product revenues of $1 million (USD) or more. Freelancers and small businesses (with gross annual revenue of less than 1 million dollars (USD)) do not pay any royalties. • Assumptions: Access to high performance computing resources to handle complex simulations. • Skill requirement: UE5 requires fundamental understanding of robotics, simulation principles and familiarity with programming. There is documentation and support available to assist practitioners. • Use case (under development by LNE): Obstacle detection tests, following the norm NF EN ISO 18497-4 (Tractors and agricultural equipment - Safety of partially automated, semi-autonomous and autonomous machines). ISAAC Sim ISAAC Sim is an open-source, robotic simulation software built to address many of the most common use cases, including manipulation, navigation, and synthetic data generation for training data. It includes advanced physics simulation, photorealism and MDL (Material Definition Language) material definition support for physically based rendering. LNE explored other simulation tools such as the ISAAC Sim to setup a robot navigation in its controlled environment. Besides a huge graphical consumption, ISAAC Sim proposes a standardized format to import robots (URDF). Summary: • Used by: LNE • Link to the website: https://developer.nvidia.com/isaac-sim • Tool effectiveness: ISAAC Sim is useful to generate synthetic data for training AI models, enhancing the development and validation process. It also provides useful simulation capabilities for robotic detailed manipulation and navigation tasks. LNE gave a score of 4 in the survey question of how likely they would recommend ISAAC Sim on a scale of 1 to 5 where 1 means ‘not at all’ and 5 means ‘very likely’. • Integration strategies: ISAAC Sim integrates with NVIDIA’s ecosystem, including GPUs and deep learning frameworks. It supports ROS and other robotics middleware. • Cost analysis: ISAAC Sim is freely accessible. However, optimal performance may require NVIDIA hardware, which could involve additional costs. • Assumptions: Effective use of ISAAC Sim requires access to compatible NVIDIA hardware and an understanding of NDIVIA ecosystem. • Skill requirement: Familiarity with robotics simulation, AI model training principles and NVIDIA’s tools and platforms. There is documentation and community support available for practitioners. • Use case (under development by LNE): Obstacle detection tests, following the norm NF EN ISO 18497-4 (Tractors and agricultural equipment - Safety of partially automated, semi-autonomous and autonomous machines). Esmini Esmini (Environment Simulator Minimalistic) is an open-source software tool to play “OpenSCENARIO” files. It is available both as a stand-alone application and as a shared library for linking with custom applications. In addition, 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 20 some tools have been designed to support design and analysis of traffic scenarios. AstaZero uses Esmini to visualize scenarios where the tool CARLA is used as rendering and early proof of concept simulation engine when working with perception and robotics. Case study: Using Esmini with CARLA for consistent test execution AstaZero AB utilizes Esmini to visualize scenarios within the context of autonomous driving vehicle simulation. This tool is often employed in co-simulation with the software tool CARLA. Below is an example demonstrating how AstaZero AB leverages Esmini for testing automation functionalities. AstaZero AB was engaged to investigate the realism required in a test environment to effectively validate and verify ADAS (advanced driver assistance systems) and AD (assisted driving) functions and to ensure that these functions could be exposed to complex and challenging traffic scenarios. This project involved conducting both physical and digital tests based on scenarios from the Euro NCAP test protocols for AEB (autonomous emergency braking) performance. Subsequently, a comparison and analysis of the test results were performed. The tests were designed based on pre-defined scenarios, each one consisting of a sequence of events outlined in OpenSCENARIO files. These files detailed the actors, event triggers, and the dynamic conditions that the scenario should follow and served as the baseline for both the physical and digital test executions. Esmini was employed to perform the simulated tests in co-simulation with CARLA. In this configuration, Esmini managed the scenario descriptions defined in OpenSCENARIO, while CARLA provided the simulation environment, handling more complex physics and rendering. Additionally, Esmini was utilized to validate the OpenSCENARIO files during development and to facilitate the execution of the physical tests. The validation of OpenSCENARIO files during development was conducted by visualisation. Within the co-simulation framework, Esmini acted as the scenario controller, overseeing the simulation’s progression. It was responsible for triggering scenario events at the appropriate times and updating the states of all actors in alignment with the scenario description. Figure 7 Scenario simulation via Esmini 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 21 The use of Esmini in this example facilitated consistent test execution across both simulated and physical environments, while also enabling the validation of predefined scenarios prior to the test executions. Esmini is an integral part of a comprehensive simulation toolchain, enabling performance and functionality testing at various stages of development. It plays a significant role in AstaZero AB's simulation toolchain and is frequently used in daily operations. Summary: • Used by: AstaZero • Link to the website: https://esmini.github.io/ • Tool effectiveness: AstaZero uses Esmini to visualize automotive scenarios in combination with the tool Carla. They gave a score of 2 in the survey question of how likely they would recommend Esmini on a scale of 1 to 5 where 1 means ‘not at all’ and 5 means ‘very likely’. • Integration strategies: Esmini integrates well with other simulation tools such as CARLA to enhance visualization and scenario analysis. • Cost analysis: Esmini is open-source, eliminating licensing costs. • Assumptions: Access to compatible hardware capable of running the simulations efficiently. • Skill requirement: It requires knowledge of traffic scenario design, software developer and python knowledge. There is available documentation and community support to help users. CoppeliaSim VREP CoppeliaSim (formerly V-REP) is a robotics simulator. It has an integrated development environment and is based on a distributed control architecture. CopeliaSim is used at ILVO to simulate kinematic models of agricultural robots. This enables early detection of errors in the kinematic modelling before actual deployment on the robot platforms. Figure 8 CoppeliaSim example. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 22 Figure 8 shows a screenshot from a video 5 showing the process for the CIMAT robot, having a 4WD4WS robot configuration. Summary: • Used by: ILVO • Link to the website: https://www.coppeliarobotics.com/ • Tool effectiveness: CoppeliaSim is useful for research and development projects in robotics as it allows simulating complex robotic systems and processes. Its distributed control architecture allows for flexible and scalable simulations. • Integration strategies: CoppeliaSim integrates with multiple programming languages including Python, C/C++, Java and robotics frameworks/ROS. It also allows seamless connectivity and data exchange with other tools and systems. • Cost analysis: CoppeliaSim has both free and paid versions. The free version provides all basic capabilities, while the paid version offers additional features and support tailored to different user needs and budgets. • Assumptions: Access to compatible hardware and stable development environment for optimal performance. • Skill requirement: Familiarity with robotics simulation and basic programming skills. The platform provides documentation and tutorials to assist users. Autonomous driving CARLA (UE4) CARLA is an open-source simulation software based on the game engine "Unreal Engine 4" to take advantage of the photorealistic rendering. It is mainly used for autonomous driving vehicle simulation, but it can also be applied on other domains where autonomy behaviour relies on the perception of the environment through cameras. CARLA supports development, training, and validation of autonomous driving systems. In addition to open-source code and protocols, CARLA provides open digital assets (urban layouts, buildings, vehicles) that were created for this purpose and can be used freely. The simulation platform supports flexible specification of sensor suites, environmental conditions, full control of all static and dynamic actors, maps generation and more. LNE used CARLA combined with Unreal Engine 4 to set up and integrate a robotic environment that includes various scenarios for autonomous driving. CARLA provides more photorealism for a better compromise of graphical resources needed than other evaluated tools. Case study: Using CARLA with Esimi for autonomous driving vehicle simulation AstaZero AB utilizes CARLA within the context of autonomous driving vehicle simulation, encompassing sensor perception, robotics, planning, control, and decision-making processes. This tool is frequently employed in cosimulation with the software tool Esmini. Below is an example demonstrating how AstaZero AB leverages CARLA for testing automation functionalities. AstaZero AB was engaged to investigate the realism required in a test environment to effectively validate and verify ADAS and AD functions and to ensure that these functions could be exposed to complex and challenging traffic scenarios. This project involved conducting both physical and digital tests based on scenarios from the Euro NCAP test protocols for AEB performance. Subsequently, a comparison and analysis of the test results were performed. The tests were designed based on pre-defined scenarios, each one consisting of a sequence of events outlined in OpenSCENARIO files. These files detailed the actors, event triggers, and the dynamic conditions that the scenario should follow and served as the baseline for both the physical and digital test executions. 5 https://www.youtube.com/watch?v=0MwMeYOQPlE 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 23 CARLA was utilized to conduct the simulated tests in co-simulation with Esmini. While Esmini managed the scenario descriptions in OpenSCENARIO, CARLA provided the simulation environment, handling the more complex physics and rendering. The use of CARLA in this example facilitated simulation within a realistic environment, where complex physics and rendering were applied to execute tests comparable to physical tests. Given that the simulation environment is a crucial component when testing and developing algorithms intended for real-world application, CARLA proves to be a suitable tool. It offers a photorealistic simulation environment with high-quality graphics, material characteristics, and realistic traffic and pedestrian behaviours. Additionally, CARLA supports a wide range of sensors, including cameras, lidar, and radar, which can be simulated in high detail, allowing for comprehensive testing of perception algorithms and systems. CARLA is an integral part of a comprehensive simulation toolchain, enabling performance and functionality testing at various stages of development. It plays a significant role in AstaZero AB's simulation toolchain and is frequently used in daily operations. Summary: • Used by: LNE, AstaZero • Link to the website: https://carla.org/ • Tool effectiveness: Carla is mainly used for autonomous driving vehicle simulation. LNE and AstaZero give an average score of 3 in the survey question of how likely they would recommend Carla on a scale of 1 to 5 where 1 means ‘not at all’ and 5 means ‘very likely’. • Integration strategies: Carla integrates with machine learning frameworks and supports ROS, with extensive customization and integrations with other tools and systems. • Cost analysis: Carla provides open-source code and open digital assets (urban layouts, buildings, vehicles) that can be used freely. • Assumptions: Access to high-performance computing resources capable of handling detailed simulations. • Skill requirement: Basic understanding of autonomous driving principles and simulation techniques. There is documentation and community support to help troubleshoot issues. Farm simulation Digital Future Farm This digital twin represents the nitrogen cycle of an arable farm or a dairy farm. It comprises of existing process-based and new data-driven models fed with real farm data to mimic reality. It can be used by farmers and researchers to reduce the nitrogen surplus of a farm while maintaining crop yields. Summary: • Used by: WUR • Link to the website: https://www.wur.nl/en/research-results/research-funded-by-the-ministry-oflnv/soorten-onderzoek/kennisonline/digital-future-farm-dff-1.htm • Tool effectiveness: Digital Future Farm is useful in modelling and simulating the nitrogen cycle within a farm. It provides insights into nitrogen use efficiency, helping to identify opportunities for reducing nitrogen surplus while ensuring optimal crop yields. WUR uses the tools as part of their design process. The tool employs modular components, representing a biophysical aspect of the farm (e.g. soil, crops, animals). Each component 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 24 can be made custom to a farm and linking these components results in a full-inclusive farm-specific digital twin. A machine learning approach is discussed to be used for the twinning process. • Integration strategies: Digital Future Farm integrates with data sources, including sensors and IoT devices to collect real-time data from the farm. It can also be connected with other farm management software to provide a comprehensive overview of farm operations. It also provides integration opportunity with other farm databases and external data sources. • Cost analysis: Digital Future Farm is a research institute funded project leveraging existing infrastructure and open-source models to minimize costs. The primary costs involve the setup and maintenance of sensors and IoT devices, as well as data processing and analysis. The ongoing operational costs are relatively low compared to the benefits in terms of improved nitrogen management and crop yields. • Assumptions: Access to accurate and reliable data from sensors and IoT devices. Adequate computational resources are required to run the models and simulations. • Skill requirement: Basic understanding of the nitrogen cycle and its impact on crop production. Also, familiarity with data collection technologies. There is training and support available through WUR to help users interpret the data and implement recommendations on the simulation results. Summary The above-mentioned software packages, currently in use by our partners UniMi, Politecnico di Milano, WUR, ILVO AstaZero and LNE, offer unique features tailored to specific types of simulation, ranging from robotic movement and interaction to autonomous driving and crop simulation. In the survey, LNE also mentioned that they have several other simulation capabilities such as MATLAB/Simulink: Embedded solution testing without 3D Rendering (Licensed) and SCANeR Studio: Close to CARLA Sim (Licensed). Universitat de Lleida uses ExtendSim to prototype and interact with end-users developing discrete event simulation models. ExtendSim is a general-purpose discrete event simulation tool. Remotisation Software Remotisation software and tools enable work from a remote location and include software and interfaces for remote access to desktops, networks and tools. Remotisation software and tools could enable remote sensing, remote monitoring, remote operation, remote tests in physical and simulated environments, remote dataset creation, or other remote services. Remote desktop access VNC (Virtual Network Computing) for AI Model training VNC is a remote desktop software that enables users to access and control a computer from another device regardless of the location. This feature is beneficial for various applications, including AI model training offering secure and convenient remote access to computational resources. VNC is a graphical desktop-sharing system that enables remote control of a computer over a network. It allows users to access and interact with a machine's desktop environment as if they were sitting directly in front of it. VNC is particularly valuable as a remotisation tool for headless devices— machines that run without a local monitor, keyboard, or mouse—such as Ubuntu-based AI computers, Raspberry Pi devices, and NVIDIA edge devices. In these scenarios, VNC can be used as a tool for managing, monitoring, and troubleshooting remote systems. A key application of VNC is in environments with limited bandwidth, where video and sensor data cannot be reliably streamed to the cloud or other remote servers. In such cases, VNC allows users to remotely access, update and test AI models and check critical data locally stored on the device, such as camera feeds, sensor data, graphs, and uptime statistics. This enables the monitoring of device and AI model performance, including factors like camera cleanliness and sensor health, without the need for continuous cloud communication by the remote device. By using VNC, users can efficiently perform remote diagnostics and maintenance on devices, especially 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 25 in remote or bandwidth-constrained environments, where storing and processing data locally is essential for optimal performance. VNC implementations used by EV ILVO are open source but there are also proprietary versions available. Summary: • Used by: EV ILVO • Link to the website: https://www.realvnc.com/ • Tool effectiveness: VNC is effective in providing remote access to computational resources required for resource intensive AI model training. EV ILVO give a score of 5 in the survey question of how likely they would recommend VNC on a scale of 1 to 5 where 1 means ‘not at all’ and 5 means ‘very likely’. • Integration strategies: VNC can be integrated with various operating systems and platforms. It can be used alongside other remote access tools and software to create a robust remote working environment. • Cost analysis: The open-source versions of VNC are free. • Assumptions: Access to stable and fast internet connection to maintain seamless remote access. It assumes users have necessary permissions and security protocols in place to access remote machines. Also, adequate hardware resources on both the client and server sides are required. • Skill requirement: Basic technical skills in setting up and configuring remote desktop connects. Familiarity with operating systems involved and basic understanding of network configurations. There is available documentation and communication support to help practitioners troubleshoot any issues. Remote tool access FarmBot FarmBot is an open-source CNC (computer numerical control) farming machine and software platform that allows remote control and management for precision agriculture. FarmBot is run by Raspberry Pi computers and Farmduino micro-controllers providing millimeter accuracy using the FarmBot genesis model. FarmBot can specifically be used by AgrifoodTEF partners for testing remote control operations of planting, watering, and weeding. It has a web-based interface for real-time monitoring and control. FarmBot also provides integration with sensors for soil moisture, light and temperature and can be customized with new tools and software updates. Summary: • Used by: Off-the-shelf remotisation platform. • Link to the website: https://farm.bot/ • Tool effectiveness: Effective for precision, agriculture tasks, offering precise control over planting, watering and weeding. Suitable tool for testing and optimizing farming practices. • Integration strategies: FarmBot has a modular design to support integration with additional hardware and software components. It can be connected to other open-source tools, platforms and various sensors for enhanced agricultural functionality. • Cost analysis: The platform is open source, but the initial setup costs can include purchasing hardware components (e.g., sensors, Raspberry pi). Ongoing costs are minimal, which mainly involve maintenance and potential upgrades. • Assumptions: Access to stable internet connectivity for remote control and monitoring. • Skill requirement: Basic technical skills and understanding of computer hardware, software, and web interfaces to set up and customize the system. There is documentation and support community available to assist practitioners.