D2.4 - HARTU-APP-BUILDER Open Tool

This project has received funding from the European Union’s Horizon Europe - Research and Innovation program under the grant agreement No 101092100. This report reflects only the author’s view and the Commission is not responsible for any use that may be made of the information it contains. D2.4 HARTU-APP-BUILDER Open Tool Deliverable ID: D2.4 Project Acronym: HARTU Grant: 101092100 Call: HORIZON-CL4-2022-TWIN-TRANSITION-01 Project Coordinator: TEKNIKER Work Package: WP2 Deliverable Type: OTHER Responsible Partner: TEK Contributors: TEK Edition date: 28 June 2024 Version: 03 Status: Final Classification: PU D2.4 HARTU-APP-BUILDER Open Tool 2 HARTU Consortium HARTU “Handling with AI-enhanced Robotic Technologies for flexible manufactUring” (Contract No. 101092100) is a collaborative project within the Horizon Europe – Research and Innovation program (HORIZON-CL4-2022-TWIN-TRANSITION-01-04). The consortium members are: 1 FUNDACION TEKNIKER (TEK) 20600 Gipuzkoa | Spain Contact: Iñaki Maurtua [email protected] 2 DEUTSCHES FORSCHUNGSZENTRUM FUER KUENSTLICHE INTELLIGENZ GMBH (DFKI) 67663 Kaiserslautern | Germany Contact: Vinzenz Bargsten vinzenz.barg[email protected] 3 ASOCIACIÓN DE INVESTIGACIÓN METALÚRGICA DEL NOROESTE (AIMEN) 36418 Pontevedra| Spain Contact: Jawad Masood [email protected] 4 ENGINEERING INGEGNERIA INFORMATICA S.P.A. (ENG) 00144 Rome| Italy Contact: Riccardo Zanetti riccardo.zanet[email protected] 5 TOFAS TURK OTOMOBIL FABRIKASI ANONIM SIRKETI (TOFAS) 34394 Istanbul | Turkey Contact: Nuri Ertekin [email protected] 6 PHILIPS CONSUMER LIFESTYLE BV (PCL) 5656 AG Eindhoven | Netherlands Contact: Erik Koehorst [email protected] 7 ULMA MANUTENCION S. COOP. (ULMA) 20560 Gipuzkoa | Spain Contact: Leire Zubia [email protected] 8 DEEP BLUE Srl (DBL) 00193 ROME | Italy Contact: Erica Vannucci [email protected] 9 FMI HTS DRACHTEN B.V. (FMI) NL-4622 RD Bergen Op Zoom, Netherlands Contact: Floris goet [email protected] 10 TECNOALIMENTI S.C.p.A (TCA) 20124 Milano | Italy Contact: Marianna Faraldi [email protected] 11 POLITECNICO DI BARI (POLIBA) 70126 Bari | Italy Contact: Giuseppe Carbone giuseppe.carb[email protected] 12 OMNIGRASP S.r.l. (OMNI) 70124 Bari | Italy Contact: Vito Cacucciolo [email protected] 13 INDUSTRIAL TECHNOLOGY RESEARCH INSTITUTE INCORPORATED (ITRI) 310401 Hsinchu | Taiwan Contact: Curtis Kuan [email protected].tw 14 INFAR INDUSTRIAL Co., Ltd (INFAR) 504 Chang-hua County | Taiwan Contact: Simon Chen [email protected] D2.4 HARTU-APP-BUILDER Open Tool 3 Document history Date Version Status Author Description 02/05/2023 01 Draft TEK Document template 12/06/2023 02 Draft TEK First version 28/06/2023 03 Draft TEK Version submitted D2.4 HARTU-APP-BUILDER Open Tool 4 Executive Summary This document describes the main concepts of HARTU-APP-MANAGER, the tool that HARTU makes available to end users and system integrators to create and control robotic applications, thus giving rise to two different functionalities: HARTU-APP-CREATOR and HARTU-APP-CONTROL. The core of HARTU-APP-MANAGER is based on the concept of behaviors trees and its implementation using BehaviorTree.CPP 3.8, a C++ library for building BehaviourTrees. The GUI is inspired and based GROOT, an advanced open IDE for creating and debugging BehaviourTrees. HARTU-APP-MANAGER is an open tool that will be made accessible through the GITLAB infrastructure created by HARTU. D2.4 HARTU-APP-BUILDER Open Tool 5 1 Table of contents 1 Introduction ................................................................................................................................. 8 1.1 Objective................................................................................................................................ 8 1.2 Codeless programming .......................................................................................................... 8 1.3 Node-RED vs Behaviour Trees ............................................................................................... 9 2 HARTU-APP-MANAGER: common entry point ........................................................................... 10 3 Robotic application builder: HARTU-APP-BUILDER ................................................................... 11 3.1 HARTU-APP-BUILDER: Main interface description .............................................................. 11 3.2 Icon bars .............................................................................................................................. 12 3.2.1 Simulation tools ........................................................................................................... 12 3.2.2 Programming by demonstration .................................................................................. 14 3.2.3 BT management ........................................................................................................... 14 3.2.4 BT Visualization ............................................................................................................ 14 3.3 Palette of components ........................................................................................................ 15 3.3.1 Action ........................................................................................................................... 15 3.3.2 Condition control ......................................................................................................... 22 3.3.3 Decorator ..................................................................................................................... 22 3.3.4 Subtrees ....................................................................................................................... 23 4 Control of robotic applications: HARTU-APP-CONTROL ............................................................ 26 4.1 Behaviour trees: workflow overview .................................................................................. 26 4.2 HARTU-APP-CONTROL: Main Interface description ............................................................ 26 4.3 HARTU-APP-CONTROL: Use Case dependent Interface description ................................... 27 5 HARTU-APP-MANAGER Deployment ......................................................................................... 28 List of figures Figure 1. Common access point of the HARTU-APP-MANAGER ........................................................ 11 Figure 2. HARTU-APP-CREATOR GUI .................................................................................................. 11 Figure 3. From left to right: Icons for adding custom nodes, loading a tree, saving the current tree and locking/unlocking BT editing ....................................................................................................... 12 Figure 4. Interface to define a scene ................................................................................................. 13 Figure 5. Interface to define the dataset of images to be generated ............................................... 13 Figure 6. Interface to configure the gripper for the LocalGraspPlanner ........................................... 13 Figure 7. Interfaces to select an existing gripper (left) or define a new one (right).......................... 13 D2.4 HARTU-APP-BUILDER Open Tool 6 Figure 7. BT arranged horizontally ..................................................................................................... 14 Figure 8. BT arranged vertically ......................................................................................................... 14 Figure 9. BT Original non-ordered BT ................................................................................................ 14 Figure 10. BT after reordering ............................................................................................................ 14 Figure 11. Basic elements of the GROOT pallete ............................................................................... 15 Figure 12. AcquireSensorData node .................................................................................................. 16 Figure 13. ChangeFrame node ........................................................................................................... 17 Figure 14. CircularMovement node ................................................................................................... 17 Figure 15. EstimateCADPoses node ................................................................................................... 17 Figure 16. EstimateGPCAD node ........................................................................................................ 18 Figure 17. EstimateGPPrimitives node .............................................................................................. 18 Figure 18. EstimatePrimitivesPoses node .......................................................................................... 18 Figure 19. ExecuteLearnedMovement node ..................................................................................... 19 Figure 20. GetCADGraspingPose node............................................................................................... 19 Figure 21. GetPrimitiveGraspingPose node ....................................................................................... 19 Figure 22. GetSensorData node ......................................................................................................... 20 Figure 23. LinearMovement node ..................................................................................................... 20 Figure 24. LocateBin node ................................................................................................................. 20 Figure 25. PathPlanning node ............................................................................................................ 21 Figure 26. PointToPointMovement node .......................................................................................... 21 Figure 27. SegmentObjects node ....................................................................................................... 21 Figure 28. SetOutput node ................................................................................................................. 22 Figure 29. SetRobotiqGripper node ................................................................................................... 22 Figure 30. Example of use of repeat decorator ................................................................................. 23 Figure 31. GraspingPointEstimation Subtree ..................................................................................... 24 Figure 32. MovePick Subtree ............................................................................................................. 24 Figure 33. MovePostPick Subtree ...................................................................................................... 25 Figure 34. Pick Subtree ...................................................................................................................... 25 Figure 35. Place Subtree .................................................................................................................... 26 Figure 36. Common interface to visualize the flow of actions (BT nodes) ........................................ 26 Figure 37. Colours for the different states of execution of the nodes .............................................. 27 Figure 38. Interface to be customize for each Use Case ................................................................... 27 List of tables No se encuentran elementos de tabla de ilustraciones. D2.4 HARTU-APP-BUILDER Open Tool 7 Acronyms List of the acronyms HARTU Handling with AI-enhanced Robotic Technologies for flexible manufactUring BT Behaviour Tree UI User Interface GUI Graphical User Interface CAD Computer Aided Design D2.4 HARTU-APP-BUILDER Open Tool 8 1 Introduction 1.1 Objective The objective of “Task 2.4 Application development support tool” is to provide a tool that facilitates the definition of a robotic application and its control by end users. Initially the use of Node-RED was considered, an open programming tool based on Node.js (nonblocking event-driven model), whose browser-based editor would facilitate the connection of flows using the wide range of nodes already available or by creating custom nodes using the available editor. Thus, it would be possible to reuse functions, templates or flows. However, after an analysis of other alternatives, it was decided to develop HARTU-APP-BUILDER based on the concept of Behaviour Trees, in particular the implementation based on BehaviorTree.CPP (see section 1.2). It uses a similar concept, so it is possible to create the set of Nodes for each of the components developed in the project and integrate with ROS2 based functionalities such as global and local trajectory planning and controllers. By means of the HARTU-APP-MANAGER, system Integrators / end users will create and control the application for the 7 use cases defined for the 5 industrial end users. IMPORTANT: Although the title of this document refers to HARTU-APP-BUILDER, it is more appropriate to name it HARTU-APP-MANAGER, as this software offers two different functionalities: • HARTU-APP-CREATOR, used to create and configure the robotic application • HARTU-APP-EXECUTOR, used to control and monitor the execution of the robotic application. 1.2 Codeless programming Codeless robot programming refers to the development and control of robotic systems using visual, intuitive, and user-friendly interfaces rather than traditional text-based coding. This approach aims to make robot programming more accessible to non-experts, allowing users to create, modify, and deploy robot behaviors and tasks without needing to write complex code. The main features of codeless robot programming are the following: 1. Visual Programming Interfaces: Users interact with a graphical user interface (GUI) where they can drag and drop predefined blocks or nodes that represent various actions, conditions, and control structures. These blocks can be connected to form a flowchart or diagram that represents the robot’s behavior and logic. 2. Pre-defined Components: D2.4 HARTU-APP-BUILDER Open Tool 9 Libraries of pre-built components or modules are available, covering common robot actions (e.g., moving, picking objects, sensing). Users can easily select and configure these components to fit their specific needs. 3. Simulation and Testing: Many codeless programming environments include simulators that allow users to test and validate their robot programs in a virtual environment before deploying them to the actual robot. This helps in identifying and fixing issues without risking damage to the physical robot. 4. Interactive Debugging: Tools for real-time monitoring and debugging of robot behavior are often integrated, allowing users to visualize the execution flow and identify problems interactively. 5. Cross-platform Compatibility: Codeless programming tools often support multiple robot platforms and hardware, providing flexibility and reducing the need for platform-specific expertise. Examples of codeless programming are Flowbotics Studio (https://wiki.lynxmotion.com/info/wiki/lynxmotion/view/ses-software/flowbotics/) and Intrinsic Flowstate (https://www.intrinsic.ai/) to name a few. Codeless robot programming democratizes the field of robotics by making it more accessible and easier to use. It leverages visual programming techniques to enable users to create, modify, and deploy robot behaviors without writing traditional code, thus accelerating development, reducing errors, and fostering collaboration. HARTU has adopted this approach to develop the HARTU-APP-MANAGER tool. 1.3 Node-RED vs Behaviour Trees As said above, a comparative analysis between Node-RED and Behaviour trees was carried out at the beginning of the project. Node-RED and BehaviourTrees are both tools used for building and managing logic flows, but they serve different purposes and have distinct differences in their design, application, and use cases. Here's a breakdown of the main differences between the two: Node-RED: Node-RED is a flow-based development tool for visual programming, mainly used to connect hardware devices, APIs, and online services. It is commonly used in IoT (Internet of Things) applications, home automation, and integrating different systems and services. Node-RED uses a directed graph where nodes represent operations or functions, and edges represent the flow of data between these nodes. The flow is built using a visual editor where nodes are dragged and dropped to create the logic flow. Each node performs a specific task and can be connected to multiple other nodes. D2.4 HARTU-APP-BUILDER Open Tool 16 Advanced users can create their own actions using BehaviorTree.CPP 3.8 3.3.1.1 AcquireSensorData Calls to the ROS2 node that implements the acquisition of an image. Currently it is available for Photoneo and ZEDi. When the data parameter is set to 1 it is called synchronously and doesn’t return until the image is available. With the data set to 0 is called asynchronously and returns the control immediately. In this later case, the image can be read using the GetSensorData function (see section 3.3.1.11). Figure 13. AcquireSensorData node 3.3.1.2 ChangeFrame Changes the frame of the camera to that specified in the node. D2.4 HARTU-APP-BUILDER Open Tool 17 Figure 14. ChangeFrame node 3.3.1.3 CircularMovement Calls the ROS2 node that generates a circular movement. Figure 15. CircularMovement node 3.3.1.4 EstimateCADPoses Calls the ROS2 node that provides the pose of the segmented objects in the image using their corresponding CAD model. Figure 16. EstimateCADPoses node 3.3.1.5 EstimateGPCAD Calls the ROS2 node that provides the pose of the grasping points on the segmented objects in the image, when the CAD is available. D2.4 HARTU-APP-BUILDER Open Tool 18 Figure 17. EstimateGPCAD node 3.3.1.6 EstimateGPPrimitives Calls the ROS2 node that provides the pose of the grasping points on the segmented objects in the image (the corresponding to the primitives), when there is not CAD is available. Figure 18. EstimateGPPrimitives node 3.3.1.7 EstimatePrimitivesPoses Calls the ROS2 node that provides the pose of the segmented objects (their corresponding primitives) in the image when there is not CAD model available. Figure 19. EstimatePrimitivesPoses node 3.3.1.8 ExecuteLearnedMovement Executes the learned movements during the learning from demonstration. D2.4 HARTU-APP-BUILDER Open Tool 19 Figure 20. ExecuteLearnedMovement node 3.3.1.9 GetCADGraspingPose Calls the ROS2 node that returns the best grasping points between all candidates in the scene, when there is CAD model available. Figure 21. GetCADGraspingPose node 3.3.1.10 GetPrimitiveGraspingPose Calls the ROS2 node that returns the best grasping points between all candidates in the scene, when no CAD model available. Figure 22. GetPrimitiveGraspingPose node 3.3.1.11 GetSensorData Retrieves the image acquired by the corresponding AcquireSensorData when in the latter the data parameter is set to 0. In this case AcquireSensorData is called asynchronously and returns the D2.4 HARTU-APP-BUILDER Open Tool 20 control immediately, otherwise, with data set to 1 it is called synchronously and doesn’t return until the image is available. Figure 23. GetSensorData node 3.3.1.12 LinearMovement Calls the ROS2 node that generates a linear movement from the current position to the target position. Figure 24. LinearMovement node 3.3.1.13 LocateBin Calls the ROS2 node that provides the location of either the work surface or the bin/container in the scene where parts can be found. Figure 25. LocateBin node 3.3.1.14 PathPlanning Calls the planner selected in MoveIt, to plan the trajectory given the initial and final points. D2.4 HARTU-APP-BUILDER Open Tool 21 Figure 26. PathPlanning node 3.3.1.15 PointToPointMovement Calls the ROS2 node that generates a point-to-point trajectory either in cartesian or joints. Figure 27. PointToPointMovement node 3.3.1.16 SegmentObjects Calls the ROS2 node that generates the masks of the segmented objects in the RGB image given the trained model. Figure 28. SegmentObjects node 3.3.1.17 SetOutput Calls the ROS2 driver (ROS2 node) that sets the digital output signal to true of false D2.4 HARTU-APP-BUILDER Open Tool 22 Figure 29. SetOutput node 3.3.1.18 SetRobotiqGripper Calls the RobotiQ gripper ROS2 driver (implementing MODBUS communication) that opens or closes the tool. Figure 30. SetRobotiqGripper node 3.3.2 Condition control The user has at his/her disposal the condition controls that are common in many programming languages. Details are available at https://www.behaviortree.dev/. • Fallback • IfThenElse • ManualSelector • Parallel • ReactiveFallback • ReactiveSequence • Sequence • SequenceStar • Switch1 to Switch6 • WhileDoElse A detailed explanation of this controls is available here: 3.3.3 Decorator Decorators are special types of nodes that modify the behaviour of other nodes. They act as intermediaries, adding additional control logic to the execution flow of the tree. Decorators can change the way a node or subtree is executed based on conditions or rules. For instance, the repeat decorator can be used to execute an action (or subtree) as many times as indicated: D2.4 HARTU-APP-BUILDER Open Tool 23 Figure 31. Example of use of repeat decorator Details are available at https://www.behaviortree.dev/. • BlackboardCheckBool • BlackboardCheckDouble • BlackboardCheckInt • BlackboardCheckString • Delay • ForceFailure • ForceSuccess • Inverter • KeepRunningUnitlFailure • Repeat • RetryUntilSuccessful • Timeout 3.3.4 Subtrees It is very common that a set of actions are repeated in different robotic applications. HARTU-APPCREATOR offers the possibility to define then as a subtree that can be then reused. Advanced users can create their own subtrees. The subtrees currently available are: D2.4 HARTU-APP-BUILDER Open Tool 24 3.3.4.1 GraspingPointEstimation It implements the various actions necessary to select the grasping point of the object that the robot will try to pick up in a given scene: acquiring the image, segmenting it and estimating the pose of the grasping point candidates. Figure 32. GraspingPointEstimation Subtree 3.3.4.2 MovePick It implements a set of movements to safely approach the pick position(just before activating the gripper). Figure 33. MovePick Subtree 3.3.4.3 MovePostPick It implements a set of movements to safely reverse the movements after the picking operation. D2.4 HARTU-APP-BUILDER Open Tool 25 Figure 34. MovePostPick Subtree 3.3.4.4 Pick It implements the complete picking operations as a combination of MovePick, SetRobotiquGrippper (or SetOutput) and MovePostPick Figure 35. Pick Subtree 3.3.4.5 Place It implements the actions to move to a destination and open the gripper to release the part held by the robot.