D2.4 - Report on Remotisation and Simulation tools

REPORT ON REMOTISATION AND SIMULATION TOOLS 2023-12-22 Ref. Ares(2024)6128755 - 29/08/2024 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 2 Project cofunded by the European Commission within the Digital Europe Programme Dissemination Level PU Public X CO Confidential, only for members of the consortium (including the Commission Services) □ CL Classified, as referred to in Commission decision 2001/844/EC □ Deliverable number: D2.4 Deliverable name: Report on Remotisation and Simulation tools Work package: WP2 Lead WP: POLIMI Lead Task: RISE 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 3 Contents Document Revision History ............................................................................................................................................... 4 Abstract .............................................................................................................................................................................. 5 Executive summary ............................................................................................................................................................ 6 Introduction ....................................................................................................................................................................... 6 Background .................................................................................................................................................................... 6 Purpose .......................................................................................................................................................................... 6 Outline ........................................................................................................................................................................... 7 Simulation Software........................................................................................................................................................... 8 Crop simulation .............................................................................................................................................................. 8 Unity ........................................................................................................................................................................... 8 Robotic simulation ......................................................................................................................................................... 9 Webots ....................................................................................................................................................................... 9 Gazebo ..................................................................................................................................................................... 10 4D-Virtualiz / 4DV-SIMULATOR ............................................................................................................................... 10 ISAAC Sim ................................................................................................................................................................. 11 Esmini ....................................................................................................................................................................... 11 Autonomous driving .................................................................................................................................................... 12 CARLA (UE4) ............................................................................................................................................................. 12 Summary ...................................................................................................................................................................... 13 Remotisation Software .................................................................................................................................................... 14 Remote desktop access ............................................................................................................................................... 14 VNC (Virtual Network Computing) for AI Model training ........................................................................................ 14 Summary ...................................................................................................................................................................... 16 Discussion ........................................................................................................................................................................ 16 Future work:................................................................................................................................................................. 10 References ....................................................................................................................................................................... 17 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 4 Document Revision History Date Issue Author/Editor/Contributor Summary of main change 2023-11-29 V0.3 Editor: Victor Kardeby - RISE Document Draft 1 2023-12-20 V0.6 Editor: Victor Kardeby - RISE Document sent for internal review 2023-12-21 V0.9 Editor: Victor Kardeby - RISE Document sent for external review 2024-01-04 V1.0 Editor: Victor Kardeby - RISE Document published 2024-06-26 V1.1 Editor: Victor Kardeby - RISE Document revised based on feedback 2024-08-29 V1.2 Editor: Victor Kardeby - RISE Document reviewed before resubmission 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 5 Abstract AgriFoodTEF is a network of test and validation infrastructures in Europe that supports Agri-Food technology companies to do near product development of their AI and Robotics solutions in real-world facilities. The overall aim is to close the gap between excellent research in these fields and actual products that support an efficient and sustainable agriculture, while meeting stringent usability and economic requirements of their end-users. AgriFoodTEF foundations are solidly rooted in existing experimental farms and facilities for AI and Robotics in Agriculture, already operational in various regions highly representative of European Agri-Food production. These have been clustered to build scale and further enhance the EU role in guaranteeing world food security with testing and validation facilities that will engage all relevant stakeholders with the best experts in the AI and Robotics technology domains. While each TEF-node will promote independent operations with a sustainable business model that will be optimized for the specialties and territorial needs, all TEF-nodes will share common guidelines, standards and support each other with services that can be offered across the different regions represented by nodes and satellites. The agrifoodTEF digital infrastructure is the set of digital resources used to provide digital testing services which can be remotely accessed and are not tied to a physical facility. This report concerns a subset of these resources and provide an initial catalogue of the agrifoodTEF remotisation and simulation tools, which are software applications that enable the creation, visualization, and manipulation of virtual models of agrifood systems, as well as the remote control and monitoring of real-world devices and processes. This report describes a mix of tools and tool chains that is currently in use by the partners, and it finishes with a discussion of the results and future work. The report will be updated and extended in the following years, as the project progresses, and new results are obtained. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 6 Introduction Background AgriFoodTEF is a network of test and validation infrastructures in Europe that supports Agri-Food technology companies to do near product development of their AI and Robotics solutions in real-world facilities. The overall aim is to close the gap between excellent research in these fields and actual products that support an efficient and sustainable agriculture, while meeting stringent usability and economic requirements of their end-users. AgriFoodTEF foundations are solidly rooted in existing experimental farms and facilities for AI and Robotics in Agriculture, already operational in various regions highly representative of European Agri-Food production. These have been clustered to build scale and further enhance the EU role in guaranteeing world food security with testing and validation facilities that will engage all relevant stakeholders with the best experts in the AI and Robotics technology domains. Furthermore, standards for dataset creation, data sovereignty, algorithm or benchmark interoperability will be based on GAIA-X as the ambassadors of different GAIA-X agricultural-domains in Europe are part of the proposal. While each TEF-node will promote independent operations with a sustainable business model that will be optimized for the specialties and territorial needs, all TEF-nodes will share common guidelines, standards and support each other with services that can be also offered across the different regions represented by nodes and satellites. The final aim is to help Europe achieve leadership in fostering top quality, technology driven and massive adoption of innovative solutions for Agri-Food, bringing primary production to new levels of efficiency akin to what was achieved by the Green Revolution. The agrifoodTEF digital infrastructure is the set of digital resources used to provide digital testing services which can be remotely accessed and are not tied to a physical facility. This report concerns a subset of these resources and provides an initial catalogue of the agrifoodTEF remotisation and simulation tools. Purpose This report is the first annual report of the agrifoodTEF project, and it aims to showcase the outcomes of the work done on the simulation and remotisation tools during the first year of the project. The report is part of deliverable 2.4, which corresponds to task 2.3 of the project. The report will cover the following aspects: The description of the simulation and remotisation tool identification method, the existing simulation and remotisation tools most of which are used by the AgrifoodTEF partners, including software applications that enable the creation, visualization, and manipulation of virtual models of agrifood systems, as well as the remote control and monitoring of real-world devices and processes. The tools will be continuously designed and developed according to the specifications and needs of the end-users and the technical partners and they will be integrated into a common platform that facilitates their interoperability and accessibility in accordance with work from Task 2.1 and 2.2. The end-users are the stakeholders from the agrifood sector, such as farmers, processors, distributors, and consumers, who benefit from the use of the tools for improving their productivity, quality, and sustainability. The technical partners are the organizations from the research and innovation sector, such as universities, research institutes and companies, who are responsible for developing, testing, and deploying the tools. The report intends to provide a comprehensive overview of the simulation and remotisation tools of the agrifoodTEF project, as well as to demonstrate their value and potential for enhancing the efficiency, sustainability, and resilience of the agrifood sector. This will mainly be done by collecting information and statistics from the infrastructure that will be provided by task 2.1. The report will be updated and extended in the following years, as the project progresses, and new results are obtained. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 7 Outline This document gives an overview of the task and purpose of the report. It details the work done around simulation and remotisation tools in T2.3 including the AgrifoodTEF simulation and remotisation tool survey conducted by RISE to identify simulation and remotisation tools currently being used by the AgrifoodTEF partners. It then goes through the different categories of simulation and remotisation tools descriptions and short analysis that are mostly identified through the survey process and is made available through the agrifood TEF. It finishes with a discussion of the results and future work. As part of the activities to identity simulation and remotisation tools in T2.3, RISE conducted a ‘simulation and remotisation survey’ to understanding the current landscape and future potential of simulation and remotisation tools as key enablers in agricultural technology. The survey was designed to gather valuable insights from the partners, focusing on evaluating the effectiveness, challenges, and reusability opportunities of the tools. The primary objectives of the survey were to: • Assess application areas: What are the application areas where simulation and remotisation tools are currently being utilized within agrifood technologies? • Evaluate adoption rates: How widely have these tools been adopted by the partners and to what extent integrated into their existing agricultural practices? • Determine impact: What is the impact of these tools on improving efficiency, productivity, and sustainability in agriculture? AgrifoodTEF simulation and remotisation tool survey methodology: The survey was conducted using mixed methods for the survey questions by creating a mix of quantitative and qualitative questions designed to capture detailed feedback on current tool practices, relevant tool links, user experiences, number of tools already under use per partner, and main expert contacts for the simulation and remotisation tools. Some questions were made mandatory to answer ‘yes’ or ’no’ (e.g., ‘do you use any simulation/remotisation software?’ as shown in Figure 1) to understand the current state of simulation/remotisation tools usage followed by some more specific questions (not mandatory) if answered ‘yes’ to gain additional information about the specific tools. We also kept some questions open ended not only to identify types of simulation tools used but also to gain information about any new types of simulation and remotisation tools available to collect feedback based on partner experiences. Moreover, there were questions to understand various other aspects such as the extent of remote operation integration, benefits realized, tool preference on a scale of 1 to 5, tool accessibility, and understand partner plans with the tools. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 8 Figure. 1 Survey question to identify the different purposes of simulations in AgrifoodTEF. The survey link was forwarded to a diverse group of partners working in the agrifood sector, including researchers and industry experts, to identify partners' current use of simulation and remotisation tools. The intention for us this year is to identify tools that are already in use by partners to determine suitability for remote experiments in the agricultural landscape. The survey findings provided us with a bigger picture for understanding how simulation and remotisation tools are practically used in the agrifood sector, guiding strategic decisions to enhance efficiency, cost reduction, productivity, and sustainability in agricultural practices. Simulation Software Simulation software serves as a platform for creating virtual models and environments for testing, training, and analysis purposes. These tools allow for the replication of real-world scenarios within a controlled digital space, facilitating experimentation, design, and training without the constraints or risks of physical testing. Common applications include engineering simulations, flight and driving simulators, medical training, and 3D modeling. Simulation software often includes realistic physics engines, graphical interfaces, and support for data analysis, catering to various industries like aerospace, automotive, healthcare, and entertainment. The versatility of simulation software enables users to refine designs, train personnel, and conduct research in a cost-effective and safe manner. Crop simulation Unity Unity is proprietary software that offers free versions for individuals and companies. The Unity Perception package[unity-perception2022] provides a toolkit for generating large-scale datasets for computer vision training and validation. The package is open-source and licensed under Apache License Version 2.0. EV ILVO uses Unity for Sim2real flower detection towards automated Calendula harvesting. More information can be found in the paper by Vierbergen et al [Sim2real]. The paper presents a pipeline to generate synthetic data of agricultural processes with photogrammetry and a game engine. A Calendula flower detector based on a convolutional neural network is trained on synthetic data and validated on a test set of real Calendula images (sim-to-real transfer). This flower detector, combined with stereo vision, enables the localization of the flowers to automatically adjust the height of harvesters to increase harvest efficiency. All collected and generated data is made available on Zenodo.org [Sim2real dataset] as a contribution to future research on precision agriculture. The synthetic dataset follows the Synthetic Optimized Labeled Objects (SOLO) Dataset Schema, as defined in the Unity Perception package. For completeness, the data scheme is also included in the upload. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 9 Tool Assessment: • Used by: EV ILVO • Link to the website: https://unity.com/ • Tool effectiveness: Unity is effective for creating realistic simulations and generating synthetic datasets for training computer vision models. Its use in the Sim2real flower detection project demonstrates its capability to bridge the gap between simulated and real-world agricultural data, achieving an F1 score of up to 86% on test sets of real data. The detection model determined the 3D positions of flowers with an average error of 6 ± 5.1 mm in predicting flower height. Using the tool, it took only 20 minutes to generate a training set of 15,000 synthetic images on a laptop with an Intel i7-8550U CPU and a Radeon Pro WX 3100 GPU. EV ILVO gave a score of 5 in the survey question of how likely they would recommend Unity on a scale of 1 to 5 where 1 means ‘not at all’ and 5 means ‘very likely’. • Integration strategies: Unity integrates with various machine learning frameworks and tools. It can be used to generate labeled datasets that can be used to train and validate machine learning models, enhancing its applicability in precision agriculture. • Cost analysis: Unity is propriety, but it offers a free version for individual users and small companies. The major costs involve the initial setup and any potential licensing fees for advanced features (e.g., high resolution 3D objects, map integration with GAIA). • Assumptions: Access to high-performance computing resources to handle extensive processing required for simulation and data generation. It also requires basic understanding of game engine operations, computer vision and basic machine learning principles. • Skill requirement: Technical skills in game engine operations, scripting and understanding of computer vision and machine learning concepts. There is available documentation and community support available. Robotic simulation Webots Webots, developed by Cyberbotics, is an open-source, general purpose simulation software for robotics used by UniMi (University of Milan). It provides an environment for modeling, programming, and simulating robots, designed for professional use in industries, education and research featuring GUI, a physics engine (ODE fork) and an OpenGL 3.3 rendering engine. The software supports programming in C, C++, Python, Java, MATLAB, and ROS, offering a simple API for basic robotics needs. Tool Assessment: • Used by: UniMI, WUR • Link to the website: https://cyberbotics.com/ • Tool effectiveness: Webots is a general-purpose simulation software for robotics. UniMi and WUR give an average score of 4 in the survey question of how likely they would recommend Webots on a scale of 1 to 5 where 1 means ‘not at all’ and 5 means ‘very likely’. • Integration strategies: Webots can be integrated into existing project workflows by using its APIs to connect with software and hardware tools making it versatile for both project and research activities. It supports various programming languages including C, C++, Python, Java and Robot Operating System (ROS). • Cost analysis: Webots is free and open source. • Assumptions: Access to compatible computer hardware capable of running the software. • Skill requirement: Webots require basic understanding of robotics concepts and programming languages required for the simulation environment. Tutorials and user guides are available online. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 16 • Assumptions: Access to a reliable internet connection for remote control and monitoring • Skill requirement: Using OpenAg requires some technical skills, including familiarity with computer hardware, sensors, and web-based interfaces. However, extensive documentation and a supportive community are available to help users with various levels of expertise. Summary VNC is currently being used by EV ILVO to perform AI model training. The software’s ability to facilitate collaboration and resource optimization makes it an asset as remotisation tool, especially in the field of AI research and development. Farmbot and OpenAG are tools used for remote control and management. Other than that, Josephinum research is planning to introduce remotisation pipeline for their new mobile sensor platform (Infrastructure WP1). Discussion and future work In this report we describe the status of the survey on available tools in AgrifoodTEF. Early results show a mix of tools and tool chains that is currently in use by the partners. We have in this report identified eight simulationand three remotisation tools. This is still a work in progress and several tools may still be missing from this report. We see several avenues for expansion in the next yearly report. Besides extending the list of tools we see an opportunity to evaluate the performance, usability, and reliability of the tools, based on the feedback from the end-users and the technical partners, who have tested and validated the tools in different use cases and scenarios. This evaluation could involve the assessment of the technical, economic, and environmental impacts of the tools, as well as their compliance with ethical and legal standards. Another topic of interest is the identification of the challenges, limitations, and future improvements of the tools, as well as the lessons learned from the implementation process. The report discusses the main difficulties and issues encountered during the development and testing of the tools, as well as the possible solutions and recommendations for enhancing their functionality, quality, and usability. The report can thus reflect on the best practices and methodologies adopted for the successful execution of the project from a tool utilization perspective. Timeframes and next steps: • In year 1, our primary focus was to identify which tools are already in use by the AgrifoodTEF partners and to understand their effectiveness in agriculture-specific simulation and remotisation experiments. Based on the survey findings, our next step is to develop a series of seminars (that has already started) and workshops for AgrifoodTEF partners. These sessions will explore the practical applications of the different tools, discuss upcoming and possible AgrifoodTEF services, and map the tools to these services. • In year 2, we will continue the seminar series, which will serve as a platform for a more comprehensive evaluation of the available tools and an opportunity to identify new tools that meet agricultural requirements. This ongoing evaluation process will help us refine our approach and ensure we are utilizing the best possible tools for our needs. • Between years 3 and 5, we will use the insights gained from these evaluations to set up a dedicated simulation and remotisation tool knowledge base specific to agricultural practices. This knowledge base will serve as a resource for all AgrifoodTEF partners, providing detailed information on the tools’ capabilities, best practices for their use, and guidelines for integration into agricultural systems. 101100622/agrifoodTEF AGRIFOODTEF Deliverable D2.4 January 4th, 2024 17 References [Sim2real] Wout Vierbergen et al, "Sim2real flower detection towards automated Calendula harvesting", Biosystems Engineering, Volume 234, October 2023, Pages 125-139. https://doi.org/10.1016/j.biosystemseng.2023.08.016 [Sim2real dataset] Vierbergen et al, Sim2real flower detection towards automated Calendula harvesting [Data set]. Zenodo. https://doi.org/10.5281/zenodo.6945367 [unity-perception2022] Unity Perception Package, Unity Technologies, https://github.com/UnityTechnologies/com.unity.perception