Knowledge representation and reasoning for perception-based manipulation planning

Universitat Politècnica de Catalunya Ph.D. Program: AUTOMATIC CONTROL, ROBOTICS AND COMPUTER VISION Ph.D. Thesis Knowledge Representation and Reasoning for Perception-based Manipulation Planning Mohammed Diab Thesis Advisor: Jan Rosell Gratacòs November 2020 Knowledge Representation and Reasoning for Perception-based Manipulation Planning Submitted in partial fulfillment of the requirements for the degree of Ph.D. in Automatic Control, Robotics and Computer Vision Supervised by Jan Rosell Gratacòs Institut d’Organització i Control de Sistemes Industrials Universitat Politècnica de Catalunya November 2020 This thesis is dedicated to the spirit of my father, my children, and my sisters’ children. Acknowledgments I would like to thank the Responsible of the Doctoral programme in Automatic Control, Robotics and Computer Vision Prof. Raúl Suárez for his efforts and guidance during the Ph.D. My external collaborators Prof. Micheal Beetz, Prof. John Bateman, Dr. Mihai Pomarlan and Daniel Beßler from Bremen University, Prof. Florentin Wörgötter, Dr. Tomas Kulvicius from Georg-AugustUniversität Göttingen, and finally Prof. Stefano Borgo from Laboratory for Applied Ontology (LOA), ISTC CNR, Trento, Italy, for their guidance during my research stays. A very special thanks should be presented to my supervisor Prof. Jan Rosell for his guidance, motivation, patience, discussion, and help in all aspects to complete this thesis. I would also like to thank Dr. Ali Akbari and Dr. Muhayy Ud Din for their help and useful long discussion. I would also like to thank all the members of the Service of Industrial Robotics research group for their help. I would like to thank Leopold Palomo for his guidance in the implementation stuff. A very special thanks to my mother, Wahida El-Husseiny, my wife, Fatema Mohamed, my sisters, and the rest of my family for there infinite support, it was not possible to complete the thesis without their support in all aspects. All praise and glory be to God who is the greatest benefactor, and Whose helps enabled me to complete this thesis. vii Contents Acknowledgments vii Abstract xix 1 Introduction 1 1.1 Problemstatement ................................... 1 1.2 Proposedsolutions ................................... 2 1.3 Contributions...................................... 3 1.4 Thesisroadmap..................................... 6 1.5 Motivationexamples.................................. 8 1.6 ListofPublications................................... 10 1.7 PublicationNote .................................... 11 2 Related Work 13 2.1 Manipulationplanning................................. 13 2.1.1 Taskplanning.................................. 14 2.1.2 Motionplanning ................................ 15 2.1.3 Combination of task and motion planning . . . . . . . . . . . . . . . . . . 17 2.2 Knowledge representation using ontologies . . . . . . . . . . . . . . . . . . . . . 19 2.3 The use of knowledge in different domains . . . . . . . . . . . . . . . . . . . . . . 20 2.4 Knowledge upper-level foundations efforts . . . . . . . . . . . . . . . . . . . . . . 21 2.5 Knowledge-based semantic perception in robotics . . . . . . . . . . . . . . . . . . 23 2.6 Use of ontologies to increase robot autonomy . . . . . . . . . . . . . . . . . . . . 24 2.7 Logic-basedplanning.................................. 25 3 Knowledge Guidance for Task and Motion Planning 29 3.1 Introduction....................................... 29 3.2 Problem statement and proposed solution . . . . . . . . . . . . . . . . . . . . . . 29 3.2.1 Problem formalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2.2 Proposed Framework overview . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3 A Knowledge Processing Framework for Physics-based Manipulation Planning . . 31 3.3.1 The Proposed Framework for Physics-based Manipulation . . . . . . . . . 34 ix LIST OF FIGURES 5.12 a) the robot checks the similarity of the current situation, it finds the same skill has been used with the same object (i.e., a drawer in the file cabinet), then adapts the skill with the same motion used in the database; b) the robot figures out the top-grasp is not feasible for pouring action, the top of the file cabinet is used as a placement room to change the grasp type; c) the robot changes the grasp type from the top-grasp to the side-grasp; d) the robot serves the contents of the can in the cup to a customer, the serve motion is adapted from the experience, according to the current pose of the robot and location of the cup. Video URL: https://www.youtube.com/watch?v=bTmWAkjC93c ................116 5.13 (a) Average time for openDrawer, pickUp from the drawer and serving skills using adaptation and planning with and without experiential knowledge. (b) Success ratefortheeachskill. .................................118 5.14 Adaptation process for serving skill with different poses. . . . . . . . . . . . . . . 120 5.15 The sequence of snapshots from planning process. The first image shows the manipulation example where the goal is to transfer the gray cylinder (labeled as A) to one of the trays w.r.t its color. . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.16 The conditional plan results from the planning process. a) flowchart describing the plan obtained by the contingent FF. b) flowchart added when executing the plan for monitoring and repair if necessary the action outcomes shown in red in theplan..........................................124 5.17Theexecutableplan...................................124 5.18 The integration of contingency plan with recovery knowledge. . . . . . . . . . . . 125 Page xvi of 151 List of Tables 3.1 Spatialreasoning. ................................... 56 3.2 List of relevant terms for the autonomous robotics domain, and their coverage in the different chosen works. Yes and No state for when the term is or not covered by the ontology of the specific framework. Note that in the cases when the term is needed and taken from the upper ontology used within the framework, and/or when the knowledge is captured using a similar term, it is considered that the term is covered. If the upper ontology contains the term but it is not used, we consider that the term is not included. . . . . . . . . . . . . . . . . . . . . . . . . 64 3.3 List of cognitive capabilities for the autonomous robotics domain and their coverage in the different chosen frameworks/ontologies. It is possible to find the reference to the articles in which the different reasoning capabilities are addressed using the ontologies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.1 Modeling the failure ontology under the DUL and SUMO foundations. . . . . . . 73 5.1 Test the skill openDrawer, pickUp and serving using adaptation method vs the planning system with and without experiential knowledge. . . . . . . . . . . . . . 117 B.1 DLnotation.......................................138 xvii Abstract This thesis addresses the perception-based knowledge representation and reasoning for a combination of task and motion planning to deal with different types of robotic manipulation problems, ranging from single or multiple collaborative mobile robots navigating among movable obstacles to complex higher-dimensional table-top manipulation problems carried out by dual-arm robots or mobile manipulators. For those problems, besides the combination of task and motion levels of planning, the integration of perception models with knowledge to guide both planning levels, resulting in a sequence of actions or skills which, according to the current knowledge of the world, may be executed, is necessary. This combination pursues the obtention of a geometrically feasible manipulation plan through a symbolic and geometric search space. It has emerged as a challenging issue as the failures due to geometric constraints lead robots to dead-end tasks. Manipulation tasks in which there may be interactions between robots and objects are considered. To cope with them, knowledge about the physics of the environment is integrated with a combination of task planning and physics-based motion planning, allowing to deal with push and pull actions and also to look for low-cost plans in terms of power. This integration enriches the planning process and to aid in providing ways of executing symbolic actions. Problems with uncertain information (in the initial state of the robot world or in the result of symbolic actions) and problems where humans and robots interact are also considered. To deal with such issues, a combination of contingent-based task and motion planner with knowledge for failure interpretation and recovery is proposed, which assumes the availability of a perception system (to evaluate the actual state of the environment) and the collaboration of the human operator (robots can ask humans for those tasks which are difficult or infeasible for them). For every-day tasks, experiential knowledge can play a significant role to make the robot capable to learn from its experience instead of repeatedly planning the same task with the same givens within the planning system, which could be computational and time-consuming process. To deal with such issues, a robotics framework is proposed which is equipped with a module with experiential knowledge (learned from its experience or given by the user) on how to execute a set of actions, like pick-up, put-down, or open a drawer, using workflows as well as robot trajectories. An implementation framework to combine different types of task and motion planners is presented. All the required modules and tools are illustrated, including the explanations on the flow of information between the different languages used, Prolog and C++. xix Chapter 1 Introduction This chapter describes the structure and contents of this thesis. It contains the problems that this thesis deals with, as depicted in Sec. 1.1, proposed solutions, as depicted in Sec. 1.2, and the thesis contributions, Sec. 1.3. The proposed solutions require approaches to be used such as planning, knowledge representation, reasoning, perception, as well as learning, as described in Sec. 1.4. These approaches have been used and integrated together in order to increase robot autonomy. Several challenges have been set to test the proposed solutions, as depicted in 1.5. These solutions have been published in robotics and Artificial intelligence (AI) journals, as well as conferences, they are mentioned in Sec. 1.6. These works have been done with collaboration with other research centers, as mentioned in Sec. 1.7. 1.1 Problem statement Indoor robots with autonomy, mobility and manipulation capabilities have the potential to act as robot helpers at home to improve the quality of life for various user populations, such as elder and handicapped people, or to act as robot co-workers at factory floors, helping for instance in assembly applications where collaborating with other operators may be required. In these semi/unstructured environments, the robot task may not be properly organized and the model of the environment completely known, and therefore abundant problems have to be taken into consideration, for example, the need of removing obstacles in order to have precisely access to a particular object, which requires the integration of symbolic and geometric planning levels, with skills such as pick-up, put-down or navigate. Moreover, perception systems using vision or depth sensors may be required to model the geometry and the pose of objects in the environment and regularly update their status. Different types of sensors have their own limitations, so perception systems based on multi-sensory data integration combining information from different sources are very useful to obtain information, which in some sense is better than the one obtained when 1 CHAPTER 1. Introduction the sources are used separately. To plan the tasks to be done by the robot in the above-mentioned scenarios, sophisticated planning mechanisms are required to adapt the actual state of the environment and comply with constraints both at task and at geometric levels. There are two dominant approaches in the manipulation planning domain, one based on classical task planning and the other based on knowledge and reasoning. The former mainly uses the Planning Domain Definition Language (PDDL) to describe the world. The main advantage of this way of description is that it can easily handle tasks with many actions, and integrate the geometric (motion) constraints. However, it makes the closed world assumption, i.e., if some facts about the world are not known or change, a planner may not be able to find a solution. This limitation means that robots are not able to begin a task until all objects in the environment are known and the actions the robot can do on them are completely defined. The latter has emerged as a new domain of planning, focused on making the robot able to flexibly perform manipulation tasks. The main advantage of this approach is that it can easily integrate the knowledge from the environment and adapt the action to be done accordingly. However, in complex manipulation problems that may have many task (geometric) constraints, such as the Towers of Hanoi problem, where task and motion levels are coupled, they may fail to compute a long sequence of actions with feasible motion solutions (Lagriffoul et al.,2018), (Beßler et al.,2018a). 1.2 Proposed solutions To tackle the aforementioned limitations of both approaches, some components are required that may facilitate the process of manipulation planning. First, a perception system to perceive the objects and their features, and a mechanism to capture the semantics of the actual scene and prepare the planning accordingly. Second, a mechanism to incorporate geometric knowledge and motion planning. Finally, a knowledge-based inference mechanism to reason on relative positioning, preconditions satisfaction and action feasibility. That is, the integration of perception and knowledge with manipulation planning approaches, may help to cover some missing components such as reasoning mechanisms able to analyze the feasibility of actions, the availability of placement regions, the reachability of grasping motions, and the satisfaction of manipulation constraints. These components play a significant role in robotic manipulation, specially for bi-manual robot tasks or for multi-robot cooperation, that require geometric reasoning, including physics-based motion planning with queries about how interaction with the objects is to be done. Also, knowledge-based sensing modules can be integrated with planners coping with uncertain scenarios, that require reasoning about the sensing system, the features of the environment entities, and sensor limitations. This way, the robot can be aware about the type of sensors that it has and how to use them. Hence, knowledge-based reasoning is proposed to facilitate the process of manipulation. This solution has the capability to make the world open to cope with environmental dynamic entities. It can be used for complex manipulation tasks that require the combination of both Page 2 of 151 1.3. Contributions symbolic and geometric levels of planning and the integration with a perception module can provide a rich semantic description for the robot whenever needed. In this sense, a well-structured knowledge representation plays a significant role. Many ways are used for knowledge representation, such as ontologies, that are concerned with structuring concepts and relations such that they are usable for reasoning tasks done by artificial systems (e.g., robots). Formally, an ontology is defined as “an explicit, formal specification of a shared conceptualization" (Gruber,1995). The conceptualization refers to the abstract models of entities in a certain domain. These models are achieved by defining their relevant concepts along with their relations. In this line, this thesis develops a series of modeling and reasoning tools for knowledgeoriented manipulation planning in semi/unstructured environments. The main idea is to use high-level knowledge-based reasoning to capture a rich semantic description of the scene, knowledge about the physical behavior of the objects, and inference mechanism to reason about the potential manipulation actions. Moreover, a multi-sensory module is proposed to perceive the objects in the environment and build the ontological knowledge. 1.3 Contributions Mobile manipulators acting as robot co-workers are required to work autonomously in human environments, and in the presence of human operators. Autonomy can be achieved with integrated task and motion planning capabilities, that are able to find feasible plans for the robot to execute complex tasks. Perception capabilities are, however, a key issue for the successful execution of tasks, because human environments are semi-structured and affected by uncertainty. Therefore, a perception module with different sensors, including those like RFID that can cope with non-line-of-sight (NLOS) situations, as well as sensory fusion mechanisms, is required. The availability of such a perception module may allow to consider sensing actions in the task planning procedure, to reduce the effects of uncertianty in the initial state and in the actions effects. Moreover, to be able to face difficult manipulation tasks in these semi-strucuted environments, some semantic knowledge on the objects of the environment and on the possible manipulation actions, is required. This knowledge may guide both the planning at motion level and at task level. This thesis contributes in this line by implementing and integrating the necessary module to increase the robot autonomy. The contribution are listed below. •Knowledge Guidance for Task and Motion Planning: A manipulation knowledge framework, called Knowledge-based Task and Motion Planning (KTAMP) is presented. The framework contains the tool called, Perception and Manipulation Knowledge (PMK) which is presented in terms of an ontology-based modeling and reasoning process. PMK aims at being shared and reused, and for this, PMK ontology relies on other upper and reference/domain ontologies: the Suggested Upper Merged Ontology (SUMO) (Niles and Page 3 of 151 CHAPTER 1. Introduction Pease,2001a) and the Core Ontology for Robotics and Automation (CORA) (Prestes et al., 2013). The reasoning mechanism includes some reasoning processes for autonomous robots to enhance Task and Motion Planning (TAMP) capabilities in the manipulation domain. A perception module can be integrated with the framework to capture a rich semantic description of the scene, knowledge about the physical behavior of the objects, and reasoning about the potential manipulation actions. The reasoning scope of PMK is divided into four parts: reasoning for perception (e.g. which perceptual features can be obtained and with which sensors?), the reasoning for object manipulation features (e.g. how can a given object be manipulated according to its characteristics and the current pose), the reasoning for a situation, (e.g. which are the spatial relations of the objects in the scene?), and reasoning for planning (e.g. can a given primitive be applied at the current scene?). The PMK tool is also integrated with physics-based motion planning that aims mainly to provide the way of interactions between a robot and objects holding specific manipulation constraints. Specially, PMK provides: 1. Standard representation: The knowledge modeling is proposed by adapting the available concepts provided by IEEE-1872 standards of knowledge representation for the robotic domain. Moreover, some uncovered concepts related to manipulation domains have been proposed, such as knowledge related to sensors. 2. Knowledge representation for perception: The perception ontology is proposed to include the perceived information from different sensors, e.g., the representation is workable for cameras or Radio Frequency Identification (RFID), and may include any implemented sensing library. 3. Situation analysis: Inference process predicates are developed based on Description Logic (DL) to evaluate the objects’ situation in the environment based on spatial reasoning, and to relate the classes entities and reason over them. Moreover, potential placement region and spatial reachability of the robot are introduced. 4. Planning enhancement: The use of PMK as a black-box allows any planner to reason about TAMP requirements, such as robot capabilities, action constraints, action feasibility, and manipulation behaviors. That includes a semantic extension to automatically construct and categorize the objects into different types according to the objects and task constraints. Also, the interaction dynamics extension to define a knowledge that allows the planner to deal with interaction dynamics. •Heterogeneous reasoning planning approach: An ontology-based framework for failure interpretation and recovery in planning and execution called FailRecOnt is proposed towards a more automated, reasoning and knowledge-driven approach to failure handling. Such an approach would need a concept of what "failure" means, what kinds of failures might happen and why, and concepts to define what an appropriate response might be. The reasoning must also be integrated into the geometric ontology that gives access to geometric reasoning such as collision check to check the actions’ feasibility as well as the perception action loop of the robot, and able to guide a plan repair process to resume or repeat a task after a failure. In more detail, FailRecOnt provides: 1. Ontology formulation: Formal definition of concepts to describe failures according to aspects such as causal mechanism, location, time relative to task performance, and Page 4 of 151 1.3. Contributions functional considerations e.g. resources, and concepts to describe recovery strategies according to the plan repair operations they require. 2. Modeling in different foundations: –For robotics domain: Modeling the absolute abstract concepts under robotics upper-level ontologies such as CORA, which uses the SUMO ontology as an upper level. –For engineering domain: Modeling the absolute abstract concepts under very generic foundational ontologies such as DUL. 3. Integration of geometric reasoning module within ontology: Integration of how to call the low-level geometric reasoning such as collision check, motion planning, inverse kinematic (IK) and object placement from ontology, which is a step toward more robot autonomy. 4. Use of the failure ontology: Description of how to use the failure ontology in a task and motion planning (TAMP) process using a knowledge-driven approach, resulting in an heterogeneous reasoning planning approach. •Skill-based task and motion planning: A Skill-based Robotic Manipulation Framework based on Perception and Reasoning framework, called SkillMaN is proposed, which is equipped with a module with experiential knowledge (learned from its experience or given by the user) on how to execute a set of skills, like pick-up, put-down or open a drawer, using workflows as well as robot trajectories. The framework also contains an execution assistant with geometric tools and reasoning capabilities to manage how to actually execute the sequence of motions to perform a manipulation task (which are forwarded to the executor module), as well as the capacity to store the relevant information to the experiential knowledge for further usage, and the capacity to interpret the actual perceived situation (in case the preconditions of an action do not hold) and to feedback the updated state to the planner to resume from there, allowing the robot to adapt to non-expected situations. Aiming at giving the robot more autonomy, specifically, SkillMaN framework provides the following services and modules: 1. Perception: An integrated multi-sensory module based on RFID sensors, with storage data capability, and RGB-D cameras, 2. Situation similarity check: Service to semantically check the situation similarity based on robot goals and perception outcomes. It helps the robot to decide whether to use its experience-based knowledge, 3. Planning: Service to interleave symbolic and geometric reasoning levels with the perception module to make the robot capable to partially figure out the environment and plan under partial information, and 4. Experiential knowledge: Module to store experiential knowledge on how to execute a set of skills, like pick-up, put-down or open a drawer, to be used for instance for adapting the motion of the robot in similar situations. All these services and modules are used through a task manager in a way that can be smoothly used in different tasks. Page 5 of 151 Chapter 2 Related Work This chapter focuses on state-of-the-art researches in manipulation planning, standardized knowledge representation, and perception-based semantic knowledge in the robotics domain, as well as the integration between them to increase robot autonomy. Moreover, a logic-based planning approach is described in more detail than other approaches because some of the ontologies proposed in this thesis are naturally integrated there. 2.1 Manipulation planning Manipulation problems are referred to as problems in which robots manipulate objects using a set of primitives, e.g., pushing, picking, or placing. Due to task constraints, the limitation of generic motion planning emerges, and the robot is required to displace objects when there is no feasible solutions between two robot configurations. A more general manipulation planning approach has been developed by (Siméon et al., 2004) that considers multiple possible grasps (that can be used for re-grasping the objects) and stable placements of the movable objects to solve the problem. In the related field of grasp planning, (Azizi et al.,2017) propose a geometric approach based on detecting a complete set of object subsurfaces in a cluttered scene, which allows the end-effector to safely approach and grasp the object. In a similar way, (Hertle and Nebel,2017) present techniques for sampling appropriate geometric configurations for object placements, grasping poses, or robot positions, in order to perform a specific action. And recently, a method for grasp planning in cluttered environments (Muhayyudin et al.,2018) has been proposed, which uses randomized physicsbased motion planning to account for robot-object and object-object interactions. This allows a robot to push obstructing objects away while reaching a target grasp pose. 13 CHAPTER 2. Related Work A manipulation planning framework with perception capability has been proposed (Migimatsu and Bohg,2020) that optimizes over Cartesian frames defined relative to target objects. The resulting plan remains valid even if the objects are moving and can be executed by reactive controllers that adapt to these changes in real time. The framework is applied to a torque-controlled robot in a pick and place setting and demonstrate its ability to adapt to changing environments, inaccurate perception, and imprecise control, both in simulation and the real world. As a complementary of this work, a learning technique has been proposed in (Shao et al.,2020) to endow a robot with the ability to learn manipulation concepts that link natural language instructions to motor skills. In (Englert and Toussaint,2018) learning manipulation skills from a single demonstration is proposed where a robot is demonstrated a manipulation skill once and should then use only a few trials on its own to learn to reproduce, optimize, and generalize that same skill. The manipulation problem of Navigation in indoor cluttered environment, at which a robot needs to manipulate some objects in the workspace in order to reach its goal region, has been addressed by in (Stilman et al.,2007;Stilman and Kuffner,2008) and (Hauser and Latombe, 2010;Hauser et al.,2010;Hauser,2014) considering the existence of the objects occluding the way between two robot configurations. The reviewed works concentrate on geometry challenges in manipulation with no highlevel reasoning, while there are other manipulation problems which require to search in a symbolic search space to find a sequence of actions. The approaches, furthermore, lack from underlying discrete symbolic relations (e.g. among robots and objects) which can break down the complexity of the problem, understood as the need to check the feasibility of the actions used in a certain task. Therefore, combining task and motion planning makes a significant role to deal with different challenges of manipulation problems for robots. 2.1.1 Task planning There has been a significant amount of studies in task planning, comprising a variety of different automated task planning approaches such as hierarchical, plan graph, and heuristic planning. Hierarchical approach (Wolfe et al.,2010), (Kaelbling and Lozano-Pérez,2011) establishes a network to assign possible preconditions of actions (primitive or compound). To find a plan, tasks are decomposed and grow in a search space until satisfying primitives actions. The plan graph (Bryce et al.,2006), (Blum and Furst,1997) constructs a search space of plans with the form of a graph and establishes a sequence of levels gradually expanded through the search space. The main merits of graph-based planners are the ability to analyze combinations of action to satisfy the task, and the ability to evaluate multiple ways to reach the goal. Regarding heuristic planning, one of the successful approach is FF (fast forward). It has two main components: RPG (relaxed planning graph), and state space search. RPG is the simplified version of the plan graph, i.e., delete list (the facts which are deleted by action) of actions are ignored. The latter is devoted to the search for the highest chance of success using the heuristic values, i.e. helpful actions (those actions that executed from a state have a high chance of being Page 14 of 151 2.1. Manipulation planning in the final plan) that it uses to guide the search of the state space (Dearden and Burbridge, 2014). To handle uncertainty in task planning (such as the uncertainty in the action effect), variants of FF such as contingent, and conformant FF have been proposed. Contingent FF is the task of generating a conditional plan given uncertainty about initial condition and action effects, but with the ability to observe some aspects of the current world state. Contingent planning can be transformed into an AND/OR search problem in belief space (the space whose elements are sets of possible worlds). The plan, which is a tree rather than a sequence of actions, can treat every possible outcome. It means that the search space is an AND/OR tree, and the plan is sub-tree where all leaves are goal (belief) states. The main advantage of contingent planner is minimizing time to find the target, and execute manipulation tasks efficiently (because there is a plan with observation) (Hoffmann and Brafman,2005). On the other hand, conformant planning is the special case of contingent planning where no observations are possible (Brafman and Hoffmann,2004), (Koenig et al.,2004). According to different applications, approaches and strategies may vary. Some researches have used FF planners for manipulation tasks, like (Srivastava et al.,2014), (Akbari et al.,2018a), (Lagriffoul et al.,2013). Besides the aforementioned approaches, some knowledge-driven approaches use ontologies in planning. For instance, a knowledge-based task and motion planning framework based on a version of the FF task planner is presented in (Akbari et al.,2016c). A reasoning process on symbolic literals in terms of knowledge and geometric information about the workspace, together with the use of a physics-based motion planner are used to evaluate the applicability and feasibility of manipulation actions and to compute the heuristic values that guide the search. The manipulation problem, involving knowledge about the world and the planning phase, is coded in the form of an ontology, and addresses a high-level and a low-level reasoning processes to appraise manipulation actions and prune the task planning phase from dispensable actions (Akbari et al.,2018b). Moreover, in (Beßler et al.,2018a) an assembly-based planner presented in Ontology Web Language (OWL) format using Description Logic (DL) is built upon an existing planner to assemble a kid toy plane called "BATTAT". This approach extends the proposed planner with a notion of action, and geometric reasoning capabilities. Actions are represented in terms of the action ontology which also defines action pre-conditions. Pre-conditions are ensured by running the planner for the action entity. This is used to ensure that the robot can reach an object, or else tries to put away occluding objects. To this end a geometric reasoner is integrated with the knowledge base. The interfaces of the geometric reasoner are hooked into the logic-based reasoning through procedural attachments in the knowledge base. Since this planner has been used in this thesis, we introduced in detail in Sec. 2.7. 2.1.2 Motion planning Motion planning deals with detecting collision-free paths to convey a robot to a configuration state. It is mostly done in the configuration space (C-space) (Lozano-Perez,1983). The C-space has as many dimensions as degrees of freedom the robot has, and therefore each point represents a configuration of the robot. The subspace corresponding to collision-free configurations is called Cfree and the subspace corresponding to collision configurations is called Cobs. Motion Page 15 of 151 CHAPTER 2. Related Work planning in C-space consists in finding a path in Cfree between two configurations. With regard to classical motion planning, researches have investigated how to deal with geometric constraints in planning (they do not impose differential dynamic constraints). Traditional methods comprise different techniques such as cell decomposition, roadmap-based methods, and potential fields. In motion planning based on cell decomposition techniques, the first step is to decompose free space, either in an exact or an approximated way, and represent the set of cells as a graph. Then, some search algorithms such as A* (Hart et al.,1968) or Dijkstra (Dijkstra,1959) methods can be employed to find a path of minimal cost. The roadmapbased approaches build a graph to connect the initial and goal states in free space and they usually find the solution paths based on the shortest distance. Regarding this case, when the obstacles in the C-space are represented as polygons, the visibility graph methods (De Berg et al., 2000) generates a graph whose nodes are the vertices of the polygons and whose edges are line segments with no interference of obstacles. The last approach uses potential fields computed in the C-space to provide attraction to the goal configuration and repulsion from obstacles (Khatib, 1986). The drawback of all these approaches is that for problems with more than two or three degrees of freedom, they are hard or impossible to implement and computationally prohibitive because they require the construction of the Cobstacle in the configuration space. Recently, much study is centered in sampling-based motion planning to provide efficient solutions for path planning by avoiding the need to compute the whole C-space. It results in probabilistic complete planning. The core of sampling-based motion planning is to use randombased techniques to sample the C-space and to interconnect those collision-free configurations as roadmaps or trees to capture the connectivity of Cfree. To cope with high dimensional degrees of freedom, it is found as an appropriate strategy to reduce the complexity of problems. Some sampling-based motion planners are investigated by (Elbanhawi and Simic,2014). The most popular approaches are tree-based techniques, which provide a single query, and roadmap-based techniques, which provide multiple queries as described below. If planning is carried out in static environments that remain constant at each manipulation step, roadmap-based approaches provide better solutions (even though they are costly to be implemented due to the need of exploring the whole C-space) because they make it possible to set multiple queries, like Path planning based on the Probabilistic Roadmap Method (PRM) (Kavraki et al.,1996) that works in two phases. The first phase is the construction phase, that spends a specific amount of time sampling Cfree and interconnecting samples with simple collision-free paths forming a roadmap. The second phase is the query process, which connects a start configuration to a goal configuration by using graph search techniques. In manipulation planning, however, since the status of the environment is altered after executing each action, tree-based algorithms have better solutions because they provide specific explorations of the C-space, e.g., focused on a given query. In this case, the environment is reconfigured and the C-space must be again explored. Here, some of the well-known single query sampling-based motion planners are reviewed as they fit better in manipulation problems and some of them will be used later in this thesis. Page 16 of 151 2.1. Manipulation planning Path planning based on the Rapidly Exploring Random Tree (RRT) (LaValle and Kuffner,2001) explores the configuration space by expanding several branches of a tree. In the generic RRT algorithm, a tree is initialized at the root where the initial state is placed and it incrementally grows towards the goal configuration along random directions biased by the less explored areas. The RRT-Connect planner (Kuffner and LaValle,2000) is a variant of RRT. There are two trees rooted at the start and goal configurations that grow to meet each other. The method provides substantial improvements in the search efficiency. The RRT* planner (Karaman and Frazzoli, 2011) has been developed that minimizes the cost of the returned solution. After growing the tree as done in the basic RRT algorithm, the RRT* captures the set neighbor nodes Nof each new added nodes to verify if it can be reached through them with a less cost and, in this case, edges are rewired. Then, this procedure is repeated for the nodes in N, which may to be rewired accordingly. Kinodynamic Motion Planning by Interior-Exterior Cell Exploration (KPIECE) planner is particularly designed for complex dynamical systems (¸Sucan and Kavraki,2009). KPIECE grows a tree of motions by applying randomly sampled controls for a randomly sampled time duration from a tree node selected as follows. The state space is projected onto a lower dimensional space that is partitioned into cells in order to estimate the coverage. As a result of this projection, each motion will be part of a cell, being each cell classified as an interior or exterior cell depending on whether the neighboring cells are occupied or not. Then, the selection of the cell is performance based on the importance parameter that is computed based on: 1) the coverage (the cells that are less covered are preferred over the others); 2) the selection (the cells that have been selected less number of time are preferred); 3) the neighbors (the cells that have less neighbors are preferred); 4) the selection time (recently selected cells are preferred); 5) the expansion (easily expanded cells are preferred over the cells that expand slowly). The cell that has maximum importance will be chosen, and a node of one of the motions of the cell will be randomly selected. The process continues until the tree of motions reaches the goal region. This approach is used while using physics-based motion planning, which is also used for grasp-in-the-clutter tasks, where the robot can interact with the obstacles obstructing the path towards the goal, moving them away. 2.1.3 Combination of task and motion planning The increasing emphasis on real world applications has led AI planning researchers to develop algorithms and system that more closely match realistic planning problems, in which the required planning activity is often distributed or continual. Distributed planning refers to situations where the planning activity is distributed across multiple robots, processes or sites. Continual planning refers to an ongoing, dynamic process in which planning and execution are interleaved. A set of actions should be considered in each task description. For instance, for a manipulation task, the set of actions may include transit (a robot moves without having an object), transmit (a robot moves an object to the goal), push, pull, pick and place. These actions should be organized in terms of the constraints that may exist between them, and an efficient strategy is required to select the sequence of excutable actions. For applying each action, a task Page 17 of 151 CHAPTER 2. Related Work planner has to evaluate whether the preconditions are satisfied or not in order to perform the required action. Complementary to motion and manipulation actions, sensing and reasoning actions can also be defined and used to reduce uncertainty and find a robust plan. In general, combination of task and motion planning needs to search in symbolic and geometric space. A task plan, including a sequence of symbolic actions, does not promise that actions become feasible in terms of geometry conditions. This requires the combined search and motivates the researchers to investigate efficient ways of the combination at both levels in order to acquire a feasible plan. Combining task and motion planning is an active research area in Robotics and AI. This section explains different strategies employed to combine task and motion planning with respect to different types of applications where the initial state and action effects are fully observable. There are two main ways of combining task and motion planning information: simultaneously or interleaved. The simultaneously approach like the work presented in (Akbari, 2018), accounts for geometric information by calling a motion planner while task planning is being pursued. The manipulation plan is then available after the task planning process is terminated. The interleaved approach like the work presented in Srivastava et al. (2014), decouples motion planning from the task planning part. Task planning first generates the sequence of actions, and then call a motion planner to evaluate the feasibility of the plan. Upon failure, geometric constraints are fed back to the task planner and the process resumes. This can be repeated several times until a feasible manipulation plan is achieved. Next, the TAMP approaches are presented in which task planning and geometric reasoning are more tightly intertwined based on the proposed ways of combination. FF-based TAMP: In this line, these studies are based on the simultaneously combination method. The studies in (Cambon et al.,2009a) present an algorithm which searches at symbolic and geometric levels, where a motion planner calls the task planner to guide roadmap sampling. Guidance is provided by a heuristic value based on the symbolic distance to the goal. In (Garrett et al.,2015) proposed an approach, called FFRob, which computes the heuristic value by analyzing the feasibility of actions with a Conditional Reachability Graph (CRG) based on a modification of PRM planner. It requires a pre-processing step to initialize the CRG by sampling objects poses and robot configurations, and determining conditions under which these samples are reachable or not. In (Akbari,2018), a simultaneous TAMP approach is proposed to efficiently deal with bimanual robot manipulation problems in constrained environments. The proposed approach is a heuristic-based planner, which searches for a plan in state space, and takes into consideration geometric constraints while computing heuristic values. Specifically, two types of geometric reasoning processes are used: a) geometric reasoning about placements of objects, grasping poses, and inverse kinematic solutions; b) geometric reasoning about motion. The former involves Spatial, Reachability, and Manipulation reasoning, which are used to account for geometric constraints in heuristic values. The proposed heuristic is able to cover a large range of tabletop manipulation problems. It figures out and excludes unfeasible motion planning queries Page 18 of 151 2.2. Knowledge representation using ontologies which have kinematic problems or collisions in their start and goal configurations, and guides state space search. The latter calls a motion planner for validating state transitions and either returns a path or provides feedback in case of failure. The state of planner is updated by the constraints which are detected using both geometric reasoning processes. Hierarchical-based TAMP: The works in this direction follow the simultaneously strategy. The work in (de Silva et al.,2013) focuses on a combination based on the HTN planner. It facilitates backtracking at different levels, also including an interleaved backtracking procedure. The application of combining hierarchical task and motion planning has been used by (Alami et al., 2014) and (Alili et al.,2010) for a teammate robot and robotic assistant. They use different geometric reasoning processes to find out a feasible plan. LTL-based TAMP: LTL-based motion planning is a hybrid approach that provides a framework to describe complex motion planning tasks in terms of temporal goals, and that plans in discrete and continuous spaces. The work in (He et al.,2015) applied TAMP using the LTL task planner. This approach is done following the interleaved approach. Motion planning evaluation launches after a task plan is provided. In the case of failure, task planning input can be updated by a set of constraints in order to find another plan. The authors claim the planner is capable enough in moving away objects that block desired executions without requiring backtracking. Constraint-based TAMP: In this direction, all the approaches are based on the interleaved combination method. The work in (Lagriffoul et al.,2012) and (Lagriffoul et al.,2014) introduce the concept of geometric backtracking, which denotes the systematic search process in the space of grasps and placements when instantiating a symbolic plan. They use linear constraints generated from symbolic actions and the kinematic model of the robot in order to prune the space of grasps and placements. The work in (Dantam et al.,2016a) proposed the Iteratively Deepened Task and Motion Planning method using the Satisfiability Modulo Theories (SMT). It incrementally detects constraints and keeps dynamically adding or eliminating a number of task constrains based on the feedback obtained from the RRT-Connect motion planner. The approach is able to find an alternative plan when an unfeasible one is identified. It first finds the task plan, and then motion planning is employed to evaluate its feasibility. The work presented by (Lagriffoul and Andres,2016) addresses TAMP by solving a culprit detection problem. In the case of failure at the geometric level, a logical explanation is computed. This explanation is fed back to the Answer Set Programming (ASP) task planner, which prunes entire families of plans leading to similar failures. The cycle repeats until a feasible plan is found. 2.2 Knowledge representation using ontologies Knowledge representation techniques are concerned with structuring conceptual knowledge such that it is usable for reasoning tasks done by artificial systems (e.g. robots). A brief introduction to ontologies and its languages is the following: Page 19 of 151 CHAPTER 2. Related Work Ontology: An ontology is a structured way of representing knowledge. Formally, it is defined as "an explicit, formal specification of a shared conceptualization" (Gruber,1995). In this definition, the term of conceptualization refers to the abstract models of an entity in a domain. These models are achieved by defining its relevant concepts along with their relations. The simplest example of an ontology is a taxonomy. It is a hierarchical structure expressing the universe of discourse based on relations, such as is-a and has-a between concepts and instances of a class. These concepts, instances and relations are expressed in formal languages. Formal languages for representing knowledge: Formal languages are used to express ontologies (in terms of concepts, relations and instances) and directly affects the reusability of the ontologies. There are three main categories of formal languages for representing knowledge: Frame based, First-Order Logic based and Description Logic (DL) based languages (Baader et al.,2017). 1. Frame based: It is a primary data structure used in artificial intelligence. Frame based language is originally developed for deductive databases. It focuses on explicit and intuitive representation of knowledge. The languages that are derived from namely frame based are: F-logic, Open Knowledge Base Connectivity (OKBC), and Knowledge Machine (KM). 2. First-Order Logic: It is a standard for the formalization of mathematics into axioms (facts). It allows the use of sentences that contain variables. The languages that are derived from First-Order Logic are: Common Logic, CycL, and Knowledge Interchange Format (KIF). 3. Description logic (DL): It provides an extension of namely frame based, without going so far as to first-order logic. The languages that are derived from DL are: KL-ONE, RACER, and OWL. A more detailed description of ontologies and DL can be found in Appendix B Abundant studies investigated the use of robot knowledge in order to connect low-level data (sensors) with high-level information (ontology). These studies cover several areas such as object recognition, task planning, or navigation. Recently, some ontologies standards in robotics have been established by integrating some of the works done in these areas, with the aim of making knowledge more general, and shareable across many domains. Ontologies standards will be explained in subsection 2.4. 2.3 The use of knowledge in different domains Many studies have investigated the use of knowledge in planning, like (Tenorth and Beetz, 2009) and (Ruiz-Sarmiento et al.,2017), that categorize knowledge about the world into terminological knowledge (TBOX), and assertional knowledge (ABOX). The former contains a hierarchy of concepts, such as interaction and action, and their relations, whereas the latter contains individuals that are instantiations of these concepts. In the navigation area, some Page 20 of 151 2.4. Knowledge upper-level foundations efforts works such as (Lim et al.,2011), (Gemignani et al.,2016) and (Krieg-Brückner et al.,2005) use a metric map and a topological map to define the robot environment. The metric map is used for the geometrical representation of the robot workspace in terms of free and occupied areas, while the topological map is used to capture the topology of the workspace. In the manipulation planning domain, works such as (Akbari et al.,2018c) propose an ontological framework to organize the knowledge needed for physics-based manipulation planning, allowing to derive manipulation regions and behaviors. Other studies have investigated the use of the robot knowledge in specific fields in order to connect the existing low-level data with the high-level information (Rai and Hong,2013) and (Chang et al.,2009). This connection can be used in several directions: object recognition and categorization (Johnston et al., 2008a), context modeling (Young Cheol Go and Joo-Chan Sohn,2005) and context reasoning (Anagnostopoulos and Hadjiefthymiades,2009), (Dargie,2009), task planning (Cambon et al., 2009b), locomotion (Chatterjee et al.,2005). For object recognition and categorization, a visual concept ontology that is composed of spatial concepts, spatial relations, color concepts, and texture concepts can all be used as an intermediate layer between domain knowledge and image processing procedures during the knowledge acquisition phase. Algorithms have also been developed for visual concept learning, feature selection, and training (Maillot et al.,2004). Moreover, symbol grounding, which occurs between sensory features and their symbolic representations (Johnston et al.,2008b), requires robust object recognition methods, combining many local features with several other visual features such as shape, color, and texture in a probabilistic and/or ontological approach. 2.4 Knowledge upper-level foundations efforts Nowadays, the use of human-robot or robot-robot collaboration is playing a significant role in robotics. One of the basic requirements for any type of collaboration is the need for a common vocabulary along with clear and concise definitions. Therefore, the need for a standard and welldefined knowledge representation is becoming more evident. The standards of ontologies are discussed in (IEEE-SA,2015). Ontologies for Robotics and Automation Working Group (ORA WG) is one of the recommended standard in robotics (Schlenoff et al.,2012), as explained below. ORA WG: The goal of this working group is to develop a standard ontology and associated methodology for knowledge representation and reasoning in robotics and automation, together with the representation of concepts in an initial set of application domains. The standard provides a unified way of representing knowledge. It allows to transfer knowledge among any group of humans, robots and other artificial systems. This group is comprised of four sub-groups entitled: Upper Ontology/Methodology(UpOM), Autonomous Robots (AuR), Service Robots (SeR), and Industrial Robots (InR). There are several general robotics standards that have been considered in the Page 21 of 151 CHAPTER 2. Related Work Planning Ontology The planner is driven by comparing goals, represented in the TBox, with believes, represented in the ABox, and controlled by meta knowledge called planning strategy. The planning strategy determines which parts of the ontology are of interest in the current phase, how steps are ordered, and how they are performed in terms of how the knowledge base is to be manipulated. Possible planning decisions are represented in a data structure called planning agenda. Planning agendas are ordered sequences of steps that each, when performed, modify the belief state of the robot in some way. The planner succeeds if the belief state is a proper instantiation of the goal description. Different tasks require different strategies that focus on different parts of the ontology, and that have specialized rules for processing the agenda. The strategy for planning an assemblage, for example, focuses on relations defined in the assembly ontology. Planning to put away parts, on the other hand, is mainly concerned with spatial relations. Strategies are associated to entities that should be planned with them. To this end, the relation needsEntity is defined that denotes entities that are planned by some strategy. Strategies assert a universal restriction on this relation in order to define what type of entities can be planned with them. For the assemblage planning strategy, for example, the axiom is asserted: ∀needsEntity.(Assemblage ∨AssemblyConnection)(2.5) Planning decisions may not correspond to actions that the robot needs to perform to establish the decisions in its world. Some decisions are purely virtual, or only one missing piece in a set of missing information required to perform an action. The mapping of planning decisions to action entities is performed in a rule-base fashion. These rules are described using the AgendaActionMapper concept, and are linked to the strategy via the relation usesActionMapper. Each AgendaActionMapper further describes what types of planning decisions should activate it. This is done with agenda item patterns that activate a mapper in case a pattern matches the selected agenda item. These are linked to the AgendaActionMapper via the relation mapsItem. Finally, the AgendaActionPerformer concept is defined which is linked to the strategy via the relation usesActionPerformer.AgendaActionPerformer provide facilities to perform actions by mapping them to data structures of the plan executive, and invoking an interface for action execution. They are activated based on whether they match a pattern provided for the last agenda item. Although these ontologies are useful for such manipulation applications which have a well-structured environment, they lack to show how it can be used in semi/unstructured environments which requires the integration of a geometric reasoning module to check the feasibility of the actions in such cases of combining two parts or even if the path toward a certain object is a collision-free. Part of the current thesis will try to fill this gap. Page 28 of 151 Chapter 3 Knowledge Guidance for Task and Motion Planning 3.1 Introduction This chapter introduces semantic manipulation knowledge for manipulation planning. The manipulation planning system includes the combination of task and motion planning (TAMP) levels. The use of abstract knowledge can enhance and facilitate the planning capabilities in both levels and give more autonomy to the robots to perform tasks. Many approaches exist for the representation of knowledge and one of them are ontologies, which are used to structure the knowledge in terms of concepts and relations. Reasoning processes over semantic knowledge can be then applied in order that robots infer the particular situation in their workspace. 3.2 Problem statement and proposed solution 3.2.1 Problem formalization In manipulation planning, dynamic interactions between the objects and the robots play a significant role. In this scope, dynamic engines, such as Open Dynamic Engine (ODE-http: //www.ode.org/) allow to consider them within motion planners, giving rise to physics-based motion planners that consider the purposeful manipulation of objects. In this context, on the one hand, at the geometric level, the representation of knowledge regarding how the objects have to be manipulated eases a semantic-based reasoning that reduces the computational cost of physics-based planners. In this work, an ontology framework is proposed to organize the 29 CHAPTER 3. Knowledge Guidance for Task and Motion Planning knowledge needed for physics-based manipulation planning, allowing to derive manipulation regions and behaviors. A semantic map is constructed to categorize and assign the manipulation constraints based on the robot, the objects and the type of actions. The ontology framework can be queried using Description Language to obtain the necessary knowledge for the robot to manipulate the objects in its environment. On the other hand, at the symbolic level, autonomous indoor service robots are supposed to accomplish tasks, like serve a cup, which involve sequences of manipulation actions. Particularly, for complex manipulation tasks that are subject to geometric constraints, spatial information is required, together with the way in which these objects can be manipulated. In this line, in this chapter an ontological-based reasoning framework called Perception and Manipulation Knowledge (PMK) is proposed as a guidance module for task and motion planning. The PMK includes: (1) the modeling of the environment in a standardized way to provide common vocabularies for information exchange in human-robot or robot-robot collaboration, (2) a sensory module to perceive the objects in the environment and assert the ontological knowledge, (3) an evaluation-based analysis of the situation of the objects in the environment, in order to enhance the planning of manipulation tasks. The work describes the concepts and the implementation of PMK, and presents an example demonstrating the range of information the framework can provide for autonomous robots. 3.2.2 Proposed Framework overview Figure 3.1: Knowledge-based reasoning framework for task and motion planning (KTAMP). With the aim of executing tasks automatically, the integration of several layers and modules is required, covering perception, knowledge representation and reasoning, and planning at symbolic and geometric levels. Page 30 of 151 3.3. A Knowledge Processing Framework for Physics-based Manipulation Planning The proposed framework Knowledge-based reasoning for Task and Motion Planning (KTAMP) is composed of three main layers, as shown in Fig. 3.1 planning and execution, knowledge, and assistant (low-level) layer. The planning and execution layer contains two modules, the task planning and the task manager modules. The former includes a task planner to compute a sequence of actions to be done, which requires a problem and domain description to set the initial scene, including the state of the world entities, and the goal state. The latter provides interfaces to communicate with the agents/operators (e.g., robots, humans or sensors). It also keeps monitoring the executed actions or atomic actions and it returns a failure signal to the recovery module if an error occurs. Moreover, it has a procedural structure for each step of a action. This structure is formally defined as a workflow that can be automatically executed through interfacing existing software components of the robot control system, e.g., executing a pickUp requires a call to the Inverse Kinematics (IK) module to check reachability for grasping the objects then finding a collision-free path towards a grasping configuration. A putDown action must search for available placement room. The knowledge layer contains the Awareness module to guide the planning and execution layer. It is composed of the PMK knowledge that consists of manipulation and perceptual knowledge. The former assists the robot to figure out the environmental entities, workspace and planning. The latter assists the robot to figure out which are the proper algorithms and parameters to be used for the available sensors in order to extract data. The PMK knowledge is described in detail below in Sections 3.3 and 3.4. Finally, the assistant layer provides the low-level modules that allow to deal with: perception issues, like finding out which sensors can be used for a sensing action in a given situation, which are dealt by the sensing module, and geometric issues, like determining if a configuration is collision-free or if an inverse kinematic solution exists for a gripper pose, which are dealt by the geometric module. 3.3 A Knowledge Processing Framework for Physics-based Manipulation Planning Description of OUR-K Knowledge Classes The ontology approach for physics-based manipulation planning proposed here is the first version of PMK. It is inspired by the Ontology-Based Unified Robot Knowledge for Service Robots in Indoor Environments OUR-K (Lim et al.,2011). The main contribution of the OUR-K framework is the establishment of an ontology-based unified knowledge for service robots in human environments by integrating low-level data Page 31 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning Figure 3.2: The structure of OUR-K ontology (Lim et al.,2011) (sensory data) with high-level knowledge (context information). This framework is described within a concept hierarchy through the use of an ontology. As shown in Fig. 3.2, it is composed of five main classes: feature, object, space, context, and action. Each class has three levels (sub-classes) (except feature class that has two), and each level has three ontological layers: metaontology, ontology, and ontology instance. Metaontology layer is used to represent generic information, such as the concept of physical object in the service robotics field. Ontology schema layer is a layer used for domain specific knowledge, for instance, in service robotics, ontology layer contains the knowledge of a particular domain such as kitchen. Ontology instance layer is used to store the information of the objects (such as their features). Because ontologies in general are an object-oriented and frame-based language the metaontology layer can provide a template for ontology layer to build terminology, while the ontology instance layer can be defined as an individual frame. The information of ontological classes, properties, and instances is transferred within unidirectional reasoning in the same knowledge level. Whereas, bidirectional reasoning relates several knowledge classes or knowledge levels. OUR-K classes are described below: Feature class is used to define how objects are perceived. It has two levels: perceptual feature and perceptual concept. In perceptual feature level, the real environment is described from the perceptual perspective, i.e. sensors, that perceive features of instances such as color, texture, and scale invariant feature transform (SIFT). In perceptual concept level, concepts are grounded to perceptual features. Page 32 of 151 3.3. A Knowledge Processing Framework for Physics-based Manipulation Planning Object class is used to define objects, their functionality and their parts. It is composed of three levels: part-object level, object level and compound level. Part-object includes parts of objects according to their functionality (for example, body and handle each has its own functionality, i.e. containing and grasping respectively). Object level includes object name and functionality, for instance, the composition of a cup is body and handle. In compound level, closely related objects that can be utilized together are linked (e.g. a cup and a saucer). Workspace class builds a semantic map and uses it to enhance the representation of the environment, which requires geometrical, positional information and qualitative features, to express relations. The construction of a semantic map depends on two types of maps: metric and topological. The former defines areas in the environment, which may be empty or occupied. The latter contains the information of the topology of the free region (e.g. with a graph extracted from a voronoi diagram). The relation between these maps and objects is stored in the semantic map. Context class is used to understand the situation of the objects in the environment. This situation is derived from two types of relations: spatial and temporal, which are defined in spatial and temporal levels, respectively, such as crowd, which means that the robot will encounter some obstacles. The spatial level contains the spatial relations (such as on, in, left, and right functions) and space classes (semantic map). Temporal level contains the temporal relations such as before, after, overlap, meet. Action class is used to define a task and the way of execution it. Action class consists of three levels: primitive behavior, sub-task and task. In primitive behavior, perceptual, motion and manipulation behaviors are defined (e.g. turn, goto, extractcolor, extractSIFT). In a sub-task level, a short-term sequence of behaviors are defined, such as gotoSpace, localization. A task such as navigate, delivery is defined in task level. It can be decomposed into sub-tasks, which can be decomposed into primitive behaviors. For example, recognizeObject is a task of finding an object within an image (e.g. extractcolor, extractSIFT). First Proposal of a Physics-based Manipulation Ontology In previous works (Akbari et al.,2016a), the authors proposed a manipulation ontology to represent knowledge to face the manipulation problems a motion planner has to deal with when the robot moves and encounters obstacles in the environment. Two classification of objects are proposed in this ontology, fixed bodies and manipulatable bodies (i.e. obstacles that can be pushed away). The former remains static during the whole planning process, even if collision happens with other manipulatable objects. The latter can be manipulated during the planning. The manipulatable bodies are classified as free manipulatable bodies and constrained-oriented manipulatable bodies. The free manipulatable bodies can move in any direction (according to the dynamics of rigid bodies) when collision occur. The constraint-oriented manipulatable bodies have some allowable motion directions and others are restricted. The manipulation constraints are modeled by defining the manipulatable region from where the object can be Page 33 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning pushed from (i.e. where the robot should be located to apply a pushing force). The knowledge is structured by defining six classes derived from a general manipulation class. These classes describe the initial state, goal state, action, region, object type, and object elements. Initial state class describes an initial location of the robot, while goal state class shows the final location of the objects. Action class contains different manipulation primitive actions, like pick, place and push. Region class defines three sub-classes: manipulation, object, and goal region. ManipulationRegion sub-class defines the region of manipulation, i.e. from where the mobile robot can interact with the object. ObjectRegion sub-class is defined as a bounding box of an object. GoalRegion sub-class is a circular region defined around the goal state. ObjectType class defines the manipulatable objects and free objects and their constraints. ObjectElements describe the feature of the objects. Although this proposal covers the manipulation constraints, it lacks to cover knowledge related to environmental entities, perception devices, agents, and planning. 3.3.1 The Proposed Framework for Physics-based Manipulation The representation of robot knowledge with an ontology has interesting properties like shareability and exchangeability (Alirezaie,2015). Knowledge is represented by defining the properties and relations between concepts that provide a rich semantic description to aid the solving of particular problems. The aim of this work is to formalize a framework following the structure of the OUR-K ontology to cope with manipulation planning problems by providing the necessary knowledge (e.g, perception, and planning) for complete scenarios. As shown in Fig. 3.3, the proposed ontology framework has six classes. The object class and the context class are the same as in the OUR-K framework. The other four are described in the following subsections. Feature class In the proposed framework, the feature class has two levels: physical concept and physical feature. The concepts of rigidbody and interaction are defined in the concept level, and their features in terms of material and interaction parameters in feature level. Fig. 3.4 describes the ontology schema of the feature class. For example, interaction is a concept defined as “Contact between two surfaces”, while its features depends on the type of material, i.e. each material has its properties like friction coefficients, density, etc. They are linked together (material and its properties) via axioms (facts). For example, to handle a cup (which is made from A), the interaction occurs between two rigid bodies, cup and robot (its end-effector is made from B), and each has its physical properties, as described below using DL: Interaction Page 34 of 151 3.3. A Knowledge Processing Framework for Physics-based Manipulation Planning Figure 3.3: Classes of the proposed physics-based ontology for manipulation, based on the OUR-K framework. Figure 3.4: Schema of feature class that explains the hierarchy of the concepts, features, and relations via axioms ∧∃hasSuperclass(Rigidbody, P hysicalObject) ∧∃hasInterP arameter(interactionP arameter, Rigidbody) ∧∃hasmaterial(material, Rigidbody) ∧∃ismaterial(A, material) ∧∃ismaterial(B, material) ∧∃hasfeature(friction, interactionP arameter) Axioms of the feature class define that the rigid bodies have a constituent material and that this material has properties, such as friction coefficient, density, slip, CFM (Constraint Force Mixing), and bounce. The physical features are used by the ODE simulator to accurately model the physical environment. Page 35 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning Figure 3.5: Schema of workspace class that describes the outcome of semantic map. The legend of different color shows that the classes information that is required to generate the semantic map. Actor class Actor class is a new-defined class that describes the properties of the robot in the environment. It consists of three levels: robot kinematics, dynamics, and constraints. Kinematic-level defines the kinematic structure of the robot and its location in the workspace, whereas dynamic-level contains the dynamic parameters of the joints of the robot, such as dumping, joint stiffness, and maximum efforts. In the constraints-level, based on kinematics and dynamics features, the robot working constraints are extracted. Workspace class In our framework, the Workspace class contains three knowledge levels: metric map, topological map, and semantic map. A metric map contains empty and occupied spaces (by either objects or the robots). Topological map defines the topology of the workspace, including where the objects are located in the workspace. Semantic map categorizes physical objects in the environment along the manipulation constraints. These constraints are defined as a function of the robot. It means that semantic map Sdepends on four parameters S= (O, R, A, T ), where: Ois the set of objects that is provided by object class, Ris the set of robot working constraints that is provided by the actor class, Ais an action that is provided by the action class, and Tis a topological Page 36 of 151 3.3. A Knowledge Processing Framework for Physics-based Manipulation Planning Figure 3.6: Schema of action class that is divided into three sub-classes: primitive behavior, sub-task and task. map that is provided by the Workspace class. The outcome of the semantic map is to assign the manipulation constraint by using DL reasoning, i.e. as shown in Fig. 3.5, the outcome of the semantic map due to reasoning is the type of object and its constraints. Action class The action knowledge class is composed of three levels: primitive behavior, sub-task, and task levels. A task is decomposed into sub-tasks, while sub-tasks involve a combination of primitive behaviors. The ontology schema of the layers of the three levels in action class is instantiated by a planner. When requested to plan, the planner checks on the topological map to know which object (or robot) is occupied with which space. Then, the semantic map is used to extract the constraints of the object (single or composite), and the robot w.r.t the action. Context class, particularly temporal level, can be used by the planner to set the sequence of action of a task. 3.3.2 Usage in Manipulation Table-top Problem To illustrate the proposal some simulation examples are performed using The Kautham Project (Rosell et al.,2014), which is a C++ based open-source tool for motion planning, that includes geometric, kinodynamic, and physics-based motion planners. It uses OMPL (Sucan et al.,2012) for the core set of planning algorithms. OMPL is a C++ based open-source library for sampling Page 37 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning Figure 3.11: Main parts of the system: The perception module, the Perception and Manipulation Knowledge (PMK) framework, and the TAMP planning module. PMK asserts the perceptual data, builds the IOC-Lab knowledge, and provides the reasoning predicates to the planning module. beverages on a table. The set-up is prepared by YuMi, while TIAGo is used to actually serve the beverages to the customers. The PMK framework with reasoning mechanisms is used to provide the reasoning predicates related to perception, object features, situation analysis, and geometric reasoning. These reasoning components, discussed in detail in this section, facilitate the planning process. The TAMP module is a combination of the FF task planner and physics-based motion planning (Akbari et al.,2016b). It is used to actually plan the task and provide a feasible sequence of actions to the robot to be executed. The execution module uses YuMi and TIAGo robots to execute the serving task. PMK Knowledge Structure A preliminary version of PMK structure has been presented in (Diab et al.,2017). It was inspired from an ontological schema proposed in (Lim et al.,2011) for navigation tasks in indoor environments, that describes concepts through the use a hierarchy of ontologies composed of three layers: metaontology layer, ontology schema layer, and ontology instance layer as shown in Figure 3.13 and described in Sec. 3.3. Moreover, examples of how information inferred from Page 44 of 151 3.4. A Knowledge Processing Framework for Aut. Rob. Perception and Manipulation Figure 3.12: A motivation example of a two-robot table-top manipulation task at the IOC-Lab. these layers are described in Sec. 5.2. Following this hierarchical schema, we proposed PMK framework 2for automated manipulation tasks where these layers are composed of seven classes: Feature, WSobject, Actor, Sensor, Workspace, Context Reasoning, and Action (Some concepts’ names have been modified here with respect to how they were presented in (Diab et al.,2017), in order to fit with the current standardized proposal). A preliminary version of this framework basically provided the concepts, relations, and inference mechanism for physics-based motion planning to teach the robot how to interact with the rigid bodies. However, the information regarding sensing was not considered, concepts that were proposed were not standardized concepts, and no knowledge regarding spatial and geometric reasoning for combined task and motion planning requirements was included. The PMK framework proposed here covers all the above missing issues. PMK Reasoning Mechanism The required inference for TAMP consists of: geometric reasoning to determine robot reachability and placement region, manipulation constraints analysis to determine how to 2https://github.com/MohammedDiab1/PMK Page 45 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning Figure 3.13: PMK ontology. interact with the objects, and the motion analysis to determine action feasibility. The reasoning mechanism of PMK is divided into four parts, as shown in Figure 3.11: reasoning for perception, reasoning for object features, reasoning for situation, and reasoning for planning. Reasoning for perception is related to sensors and algorithms to simply answer questions like which are the sensors the robot has? what is the corresponding algorithm to extract the perceptual data from the sensor?. Reasoning for object features copes with the features of the objects such as color and dimensions. Reasoning for situation analysis is used to spatially evaluate the objects relations between each other (i.e., cup inside box, and cup is reachable by left arm). Reasoning for planning is used to reason about the preconditions of actions, action constraints, and geometric reasoning (arm reachability, grasping pose reachability, and placement region). 3.4.3 Why PMK? This section covers the differences between PMK and other knowledge-based processing frameworks, such as KnowRob (Tenorth and Beetz,2017) and OUR-K (Lim et al.,2011), by highlighting the importance of PMK that is not covered by them. We divide the differences into two levels: modeling and reasoning process. At the modeling level, although these approaches provide frameworks that include a comprehensive way of representing the ontologies related to how to execute manipulation tasks, they lack the representation of the meta-level concepts using the common vocabularies provided by standardization such as (Niles and Pease,2001b) (i.e., they do not follow any standardization). This may lead to difficulties in incorporating/importing other ontologies under their terminologies because of the conflict in the meaning of the concepts for the robot. This may be a problem for collaborative tasks between robots that require some common vocabularies. At the reasoning process level, they do not fully cover the area of TAMP to facilitate the planning process. For example, in motion planning, to deal with rigid bodies in cluttered environments, there is the need to define the way to apply actions such as push/pull, requiring a rich semantic description to be fed to the planner, like the physics-based motion planner in (Muhayyudin et al.,2018). In task planning, the robot needs to reason on: (a) the feasibility Page 46 of 151 3.4. A Knowledge Processing Framework for Aut. Rob. Perception and Manipulation of an action at some instant of the manipulation planning process (according to object features, the the state of the object could change and be out of the robot capabilities, e.g., if the cup is empty the state is graspable and if full the state is pushable), (b) the selection of the placement where the robot must place the object, and (c) the current constraints. Moreover, reasoning about perception knowledge is required. PMK covers these gaps in the modeling and reasoning process levels. In the former, by following the standardized concepts presented by SUMO and CORA. In the latter, by providing reasoning predicates to cover TAMP needs. 3.4.4 Knowledge Formulation The classes mentioned above in Section 3.4.2 were shown in the metaontology layer in Figure 3.13. The concepts in these classes are formulated here according to the standardized concepts presented in the IEEE 1872 standard. The ontology schema and ontology instance layers are not discussed since they depend on specific domains (in the motivation example, these have been built for the domain of our robotic lab). The formulation according to the standard uses mainly SUMO, CORA, CORAX, and ROA (see Sec. 2.4). SUMO divides the entities into two groups: physical and abstract. The physical group describes the entities that exist in space-time, and is subdivided into object and process to represent, respectively, bodies and procedures. The abstract group describes the entities that do not exist in time and include mathematical constructs (IEEE-SA,2015). Following the same modeling strategy, as shown in Figure 3.14, PMK divides the knowledge into knowledge related to objects (manipulation world), knowledge related to processes (manipulation planning), and abstract knowledge (manipulation data). The manipulation world knowledge includes the description of objects, robots and sensors in the workspace. The manipulation planning includes the reasoning processes over PMK. These are detailed in the next subsections. The manipulation data knowledge represents the features of objects, such as color, mass, robot constraints (e.g., joint limits) and sensor constraints (e.g., maximum and minimum measurement ranges). Manipulation Data Knowledge It includes the Feature class, which is subdivided into three gradual levels to represent data from low-level to high-level (abstract). The standardized concept of SUMO: Quantity is used in level one to describe the quantities of the environment, such as object dimensions, color, position, or orientation. A new concept Quantity Aggregation has been introduced in level two to include aggregations of quantities, like wrench (force and torque) or pose (position and orientation). The standardized concept of SUMO: Attribute is used in level three to represent abstract data like matrices. These concepts are modeled and related to the ontology schema layer (which in our case describes the IOC-Lab domain) to retrieve the required information. For instance, as shown in Figure 3.15, the Quantity concept is related to the concept Pose to retrieve the information of the grasping pose of YuMi gripper. Page 47 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning Figure 3.14: Structure of the metaontology layer of PMK fit to follow the standard IEEE-1872. Concepts introduced in this work are shown in black, while those that inherit from the standard are shown in color: SUMO (blue), CORA (white), CORAX (red), and ROA (green). All the terminologies are defined in the Appendix. Figure 3.15: Description of some abstract manipulation knowledge concepts. Manipulation World Knowledge It includes four classes WSobject, Actor, Sensor, and Workspace. The main level of the first three classes is level two. In these levels, the standardized concepts are used to define the physical objects and their functionalities, while level one (component-level) represents the description of the components of the objects and level-three (grouping-level) represents the description related to the grouping of the objects in the world. For example, a cup has a body and a handle as components, and it can be grouped with a saucer. As shown in Figure 3.14, in the main level of WSobject, Actor, and Sensor classes, the standardized concepts of SUMO: Artifact, CORA: Robot and SUMO: Measuring Device have been used, respectively. Component-levels use the new concepts Artifact Component, Robot Component and Measuring Device Component. Grouping-level uses the standardized concepts of Page 48 of 151 3.4. A Knowledge Processing Framework for Aut. Rob. Perception and Manipulation SUMO: Collection and CORA: Robot Group for the WSobject and Actor classes, respectively, and the new concept of Measuring Device Group for the Sensor class. These concepts are modeled and related to the IOC-Lab ontology schema layer to retrieve the required information. Figure 3.15 shows an example of the relation between the component-level of the robot and the physical robot, where the YuMi robot and its gripper are defined under the Robot and Robot component meta-concepts, respectively. The main level of Workspace class uses the concept from CORAX: Physical Environment that describes the topology of the objects in the environment (i.e., which area is occupied by which physical object), the standardized concept SUMO: Region is used to describe the geometrical representation of the workspace in level-one, and the new concept of Semantic Environment is introduced in level-three to complete level-two with the data (features) of the physical objects. Manipulation Planning Knowledge The manipulation planning knowledge represents the part that is responsible for reasoning about the situation of the objects and the robot, and for planning the tasks. This reasoning process is done over the manipulation environment knowledge to facilitate the planning process of the tasks. As shown in Figure 3.14, this knowledge includes two PMK classes, Context reasoning and Action. Each class has three levels, from high-level to low-level concepts. Levelthree represents the symbolic information that depends on the first two levels. For example, in the Context Reasoning class, the standardized concept SUMO: Situation represents the status of the object or robot in the world with respect to space and time. To cover both, Spatial Context (such as left, right, on, in) and Temporal Context (such as before, after, meet, overlap) are introduced to describe them, respectively. These concepts are modeled and related to the IOCLab ontology schema layer to retrieve the required spatial information about the environment entities. For instance, Figure 3.15 shows the can object, which is sub-class of Artifact, that has the property Spatially Located to report the ontology with its spatial location with respect to other objects, e.g., the can is on the small table. For Action class, symbolic tasks such as serve are defined in level-three (Task), which are composed of short-term sequences of simpler actions such as move, pick up, moveholding, place, that are defined in level-two (Sub-Task). The standardized concepts of ROA:task and ROA:subtasks are used to describe these two levels. Level-one includes the new concept of atomic function to represent processes for motion, manipulation, and perception, such as task planners, motion planners, or perceptual algorithms. Moreover, it includes primitive actions, preconditions, and postconditions related to manipulation behaviors. Although ROA provides the concept of robot behavior (Olszewska et al.,2017), it does not fully cover the manipulation planning perspective that we need. On the one hand, for sensors, the corresponding suitable algorithms (depending the type of sensor) should be available to extract the features of the environment (see Section 3.4.4). On the other hand, for planning (e.g., task and motion), the corresponding algorithms should be available to plan according to the problem. So, the new concept of Atomic Function is introduced to define the algorithms related to planning in terms Page 49 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning of motion and task, and the perceptual algorithms. Knowledge Representation for Perception To perceive a robot environment, different sensors are usually used. Sensors provide data about the environment in the form of signals (one dimension) or images (multi-dimension), and to obtain the useful features from the perceived data the suitable algorithms have to be applied, for instance to detect an object pose some pose estimation algorithms based on image features can be applied, or alternatively algorithms based on tags identification can be used. In Figure 3.14, sensors are defined in sensor class as measuring devices (level-two). The sensors, which can be attached with the robot or fixed in the environment, may contain different parts, which are represented as sensor components (level-one), like the tags, antenna, and reader that contain a RFID sensor. Several sensors may also be grouped as a device group (level-three) for a given application or domain, like the grouping of a RFID sensor and a 2D camera for object localization. This grouping, for instance, allows the robot to understand that it has two types of sensors to locate objects, as well as their differences, e.g., type of data extracted from each sensor, algorithms to be used on the data provided, or the best environmental conditions for their use. The sensor grouping concept makes the robot aware of its available equivalent sensing strategies and allow the selection of the proper sensor to use in each case according to the situation. PMK provides the relation between the Feature, Sensor, and Action classes that allows to extract the information from sensors. For instance, as shown in Figure 3.15, the concepts of Camera and Algorithm, which are inherited form Measuring Device and Atomic Function meta-concepts, respectively, are modeled and related to the IOC-Lab ontology schema layer to retrieve the required algorithm to extract the image features. The inference mechanism required for the sensing procedure is detailed in Section 3.4.5. The main advantage of this way of representation is that it is workable for any type of sensor and any type of data processing algorithm (like pose estimation from tag detection in 2D images implemented in the ar–track–alvar library (http://wiki.ros.org/ar_track_alvar) and used in the case study shown below). 3.4.5 Case Study In this section, serving task is proposed to serve a beverage to a customer. The YuMi robot (bimanual robot) is used to prepare the beverages. It overcomes some manipulation constraints. To tackle the task, a set of different reasoning predicates regarding task and motion planning, perception, are proposed. The description of the task, its constraints, and the implementation tools are described below. Moreover, the reasoning mechanism to facilitate the planning process is introduced. The use of those predicates in the proposed task is explained in detail. The evaluation of the case study and the system flexibility are finally discussed. Page 50 of 151 3.4. A Knowledge Processing Framework for Aut. Rob. Perception and Manipulation Task Description Consider a manipulation problem including the bi-manual YuMi robot and a set of objects as depicted in Figure 3.12. The task is to serve the wineglass on the servingTable. Initially the wineglass is located inside the box. Due to reachability limitations, both arms have to collaborate with each other to solve the task. The responsibility of YuMi right arm is to pick up the wineglass and place it on the smallTable and then, using the left arm, pick up the wineglass up and place it on the servingTable. The challenge is that the placement region of the wineglass on the smallTable is already occupied by the can. Any planner to be used to solve this task needs a rich semantic description of the scene, able to answer questions such as what are the sensors the robot has?, what are these sensors detecting?,what is the associated algorithm to extract the object features?, what is the spatial situation of the objects?,what is the available regions to place the object?, or how can the obstacle be removed? and some other questions highlighted in Figure 3.16. PMK can provide answers to these kind of questions. The perception module consists of two 2D cameras and tags for all the objects (see Figure 3.12). One camera is fixed on the top of the main table to sense the main table entities, and the other is attached to the YuMi left arm to perceive the serving table. The tags are used to identify the world entities and semantically link them to the the properties of each object. Specifically, the purpose of the perception module is to detect the position of the objects and their IDs and assert them on the ontology to build the ontology schema and the ontology instance layers of the IOC-Lab environment (as shown in Figure 3.11). An expressive inference process helps to identify the hidden knowledge and increase the robots capabilities. The PMK framework uses an inference mechanism that consists of Prolog predicates. These predicates are used to query over the ontology to obtain the knowledge that the robot requires to manipulate the objects in the environment. The inference mechanism for manipulation planning domain includes the reasoning process related to sensing, task planning, and motion planning. The related generic predicates are explained in the following subsections. Implementation PMK can be integrated with any task and motion planner such as (Akbari et al.,2016a) to compute the sequence of actions to solve a manipulation task. The planner may ask the ontology questions about how to perform the actions, what are the objects’ poses, or which are the interaction parameters of the objects? The PMK handles the requests of the planner and answers by retrieving information, updating/deleting or reasoning over it. The request-answer relation is done using the service-client communication of ROS (Robot Operating System, www.ros.org). The PMK is designed using ontology web language (OWL) with Protégé ontology editor (http: //protege.stanford.edu/). Ontology instances can be asserted using information processed from low-level sensory data. The C++ library ar_track_alvar (http://wiki.ros.org/ar_track_alvar) has been used to detect the object pose and ID. These data are asserted in the PMK to extract a semantic description of the object. The 2D cameras have been used as a measuring device Page 51 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning Figure 3.16: The description of the constrains of the motivation example related to sensors, object geometry, interaction learning, and action feasibility. with two ROS nodes called FixedCam and AttachedCam, for the fixed and attached cameras, respectively. All the transformations of the objects and camera are calculated with respect to the world frame located at the YuMi base. Queries over the PMK are based on SWI-Prolog and its Semantic Web library which serves for loading and accessing ontologies represented in the OWL using Prolog predicates. PMK Ontology Representation In the metaontology layer, region is a concept linked to artifact and quantity, as shown in Figure 3.17 where the relation between the ontological layers that describe the IOC-Lab under the metaontology concepts is illustrated. In the ontology schema layer, the IOC-Lab region has artifacts such as box, table, wineglass, can, cup and quantities such as color and dimension. In the ontology instance layer, the instance box01 of the subclass box represents the storage area that contains the instance wineglass01 and cup01 of the subclasses wineglass and cup, respectively. These instances have perceptual properties such as pose and tagID, that are asserted with the following Prolog predicate: rdf_assert(instance, registerName:objectproperties, assertedvalue), Page 52 of 151 3.4. A Knowledge Processing Framework for Aut. Rob. Perception and Manipulation Figure 3.17: Taxonomy representation of the IOC-Lab domain in the PMK framework. The concepts of region, artifact, and quantity are inherited from Workspace, Wsobject, and Feature classes, respectively. The Quantity describes the features of each instance as shown in the properties part in the instantiated artifacts (the orientation information of the objects poses is not included in order to simplify the figure). and linked to the fixed properties stored on the ontology, so as to have a complete knowledge about the objects. Then, once the the robot figures out the object ID, all the features can be extracted. In order to know where is the reasoning done, who is doing the assertion, and how the assertion is done, see Fig. 3.19. Reasoning Process on Perception The inference process related to perception knowledge is basically the reasoning about feature extraction algorithms for perception, such as FixedCam or AttachedCam nodes used to extract the poses and IDs from images. The inference process related to the perception knowledge of PMK basically depends on the relation between three classes (Feature, Sensor, and Action). As shown in Figure 3.18, the device is a concept that has quantity and atomic function. The quantity contains the constraints and perceptual data. The constraints describe the sensor limitations, such as the fixed camera measuring only the mainTable with certain minimum and maximum ranges. The perceptual data describes the type of data and its properties, e.g., pose and IDs are the properties that are extracted from an image. These data are divided into two main parts Feature of Interest and Observable Property (names are inspired from the Semantic Sensor Network Ontolgy, https://www.w3.org/TR/vocab-ssn/). The former describes the type of data that a sensor senses, such as camera senses images. The latter is an observable characteristic (property) of a Feature of Interest, such as color is one of the image properties. As an example, the RunFixedNode predicate, shown below using DL, is applied to reason about the location and ID of the objects by being associated with the FixedCam Node atomic function (algorithm). Page 53 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning Figure 3.22: The sequence of snapshots for the execution of the serve a cup of coffee command. Video URL-link "https://sir.upc.edu/projects/kautham/videos/PMK-final.mp4". 3.4.7 Discussion Discussion about the Results The motivation example highlights the significance of using PMK to make the robot understand the current scenario and act according to the actual manipulation constraints. The manipulation constraints related to task execution can be listed as: 1. Manipulation constraints such as: From where the object can be interacted?, What are the interaction parameters? 2. Geometric constraints such as: What is the spatial robot reachability? Where can the objects be placed? 3. Action constraints such as: From where can the actions be applied? 4. Perception reasoning such as: What is the sensor attached to the robot? How does it work? What are its constraints, such as the sensor range measurement?. PMK provides the answers needed by combined task-motion planners and by physics-based motion planners. Some other frameworks proposed a knowledge-based processing and reasoning such as (Tenorth and Beetz,2009;Lim et al.,2011). Although there are some similarities with them in terms of spatial reasoning or reasoning about the object features which could be common in the working domains, the reasoning process related to planning (including geometric reasoning) and perception reasoning are newly proposed in PMK. For instance, these frameworks proposed Page 60 of 151 3.4. A Knowledge Processing Framework for Aut. Rob. Perception and Manipulation some spatial relations such as on, inside, right, left for the robot to spatially related between the environment entities. The extended spatial reasoning that includes some extra predicate such as spatial robot reachability and arm selection for bi-manual robots have been introduced. In the case study, the importance of these predicates appear, for instance, when the robot needs to execute the pick up action for wineglass. First, by asking about which arm is reachable, then by asking about the reachable placement region to place the wineglass on the servingTable with the left arm. Moreover, when using a physics-based motion planning, to apply the push action, a query is posed to know how to interact with the obstacle can, which needs, for instance, the interaction parameters like the friction coefficient. Moreover, for perception, PMK proposes the reasoning related to the perceptual features of the objects in the environment, like others, but also some reasoning related to the suitable algorithms that the sensor can run to extract the features, the sensors that are associated with the robot, and the sensor features and its limitations. The perception reasoning process makes the robot smarter and more flexible. This flexibility can be useful to cope with the failure of a sensor, by providing an alternative one, i.e., PMK has the flexibility to deal with multi-model sensory systems. Discussion about the System The execution of manipulation tasks with knowledge-based planning approaches not explicitly prepared for TAMP, like (Tenorth and Beetz,2009), can be a challenge because this would require, on the one hand, from the knowledge perspective, to provide all the components in a way that they match with their planning system. On the other hand, from the planning perspective, they would require the definition of the recipes (strategies) for executing the tasks (sequence of actions), including all possible strategies of execution and the way to switch between them when required, which can be a very expensive process, especially for tasks that need long sequences of actions, such as those involving manipulation in cluttered environments. Some other planning approaches rely on PDDL and on planning strategies best fit to cope with difficult task planning challenges, like those found in the manipulation domains, although the use of PDDL implies a closed-world assumption, which precludes their use in more dynamic environments that could require perception and knowledge-based geometric reasoning. The main role of PMK is to facilitate the planning process for TAMP by providing the necessary planning components, such as geometric reasoning, dynamic interactions, manipulation constraints, and action constraints. Moreover, it includes the perception knowledge and reasoning about the sensors that the robot has, the corresponding algorithms to extract the features, and the sensors limitations. All these components are required to automatically execute complex tasks, like those presented in (Lagriffoul et al.,2018;Quispe et al.,2018), and to make the system flexible enough to adapt to different situations requiring different manipulation actions, as demonstrated with the two motivation examples. Page 61 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning 3.4.8 Summary This study proposed the formalization and implementation of the standardized ontology framework PMK to extend the capabilities of autonomous robots related to manipulation tasks that require task and motion planning (TAMP). In terms of modeling, the aim has been to contribute with a unified framework based on standardization that provides common vocabularies in order to have the flexibility to incorporate PMK with other ontologies, to avoid conflict in the meanings of concepts. Moreover, in terms of reasoning process, some important components for task, motion, or combined task-motion planning are proposed, such as reasoning for perception, reasoning about the object features, reasoning about the environment, geometric reasoning, and reasoning for planning. To illustrate the proposal, two examples had been introduced to show the PMK framework abilities to query knowledge about the geometric reasoning and robot capabilities to solve TAMP problems. Most of the requirements for TAMP, discussed in this work, are met by the proposed framework. The knowledge is organized in a way to facilitate the planning process, so that the robot can easily access the concepts it needs for its tasks, i.e., PMK enables the robot to complete a manipulation task autonomously in spite of hidden or partial data. To substantiate the perceived information, knowledge related to perception has been considered, which allows analyzing the situation of the environment. Since TAMP queries the requirements to PMK, it automatically adapts to different scenarios. 3.4.9 Enhancement The PMK framework dose not cope with the navigation problems which requires the extension with more perception measuring devices, such as RFID sensors and 3D cameras, to be applied to more complex manipulation tasks involving assemblies. Also, an extension is planned to include concepts and reasoning related to failures, in order to make PMK useful in situations where the task planner may not be able to find a feasible sequence of actions to perform a given task, or may need to recover from an error. This enhancement has been done in Chapter five. 3.5 Comparison of PMK against other ontology-based approaches to robot autonomy A comparison between PMK and some projects that use ontologies to support robot autonomy is proposed. The systematic search for projects that fulfill a set of inclusion criteria has been done, and the comparison them with each other with respect to the scope of their ontology, what types of cognitive capabilities are supported by the use of ontologies, and which is their application domain. Page 62 of 151 3.5. Comparison of PMK against other ontology-based approaches to robot autonomy 3.5.1 Inclusion criteria Some frameworks or projects have been selected which are considered to be object of the analysis performed in this work. However, among them, the focus is only on the discussion and the comparison on the most influential approaches. Hence, this work provides a list of inclusion criteria to refine the list of surveyed projects. As presented in the Section 4.4 in (Olivares-Alarcos et al.,2019), authors briefly introduce the excluded approaches and provide some justification for our decision. Projects or frameworks are only considered in the scope of this work if they satisfy all of the following criteria: 1. Ontology scope: The project uses an ontology that defines one of the terms shown in Table 3.2, such as Capability, Skill, Plan and Function, etc. that are identified as particularly relevant for autonomous robotics (those terms are presented in Sec. 3.1 in (OlivaresAlarcos et al.,2019)); 2. Reasoning scope: It uses ontologies to support robots manifesting at least one of the cognitive capabilities (discussed in Section 3.2 in (Olivares-Alarcos et al.,2019)); 3. Transparency: It is transparent. Meaning that some material (e.g., websites, publications) is openly available that describes the overall goal of the project, what cognitive capabilities are considered, and how and what ontologies are used; 4. Curation: It is maintained. Meaning that recent developments or future plans are evident or at least possible; and 5. Accessibility: There exists –at least a prototypical –software that is accessible, and that demonstrates how ontologies are used to support a cognitive capability. 3.5.2 Comparison of abstract concept and domain category There are six frameworks/projects that satisfy the criteria from different robotics applications areas such as service and industerial, etc., and The PMK (Diab et al.,2019) is one of them. Beside PMK, others like KnowRob (Tenorth and Beetz,2009,2017), ROSETTA (Stenmark et al.,2018), IEEE Standard Ontologies for Robotics and Automation (IEEE-SA,2015), ORO (Lemaignan et al.,2010), CARESSES (Bruno et al.,2017) are discussed in this review. For each of them, their underlying principles and foundations are discussed, as well as what application domain the system was designed for. Also the description of how the frameworks evolved over time, and what impact they have had so far. As mentioned in Table 3.2, abstract concepts such as objects, environment map, action, capability, hardware, and software are proposed in PMK and some of the other frameworks. Uniquely, PMK proposed some terms like Function and Interaction which are not proposed by other frameworks. As a result of it, PMK is considered as a quite comprehensive and flexible enough to model several robotics domains through it. Page 63 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning Term KnowRob 1/2 ROSETTA ORO CARESSES OROSU PMK Object Yes/Yes Yes Yes No Yes Yes Environment map Yes/Yes No No No Yes Yes Affordance No/Yes No Yes No Yes No Action Yes/Yes No Yes Yes Yes Yes Task No/Yes Yes Yes No No Yes Activity No/No No No Yes No No Behavior No/No No No No No No Function No/No No No No No Yes Plan No/Yes No Yes No No Yes Method No/Yes No No No No Yes Capability Yes/Yes Yes No No No Yes Skill No/No Yes No No No Yes Hardware Yes/Yes Yes Yes No Yes Yes Software Yes/Yes Yes Yes No Yes Yes Interaction No/No No No No No Yes Communication Yes/No No No No No No Table 3.2: List of relevant terms for the autonomous robotics domain, and their coverage in the different chosen works. Yes and No state for when the term is or not covered by the ontology of the specific framework. Note that in the cases when the term is needed and taken from the upper ontology used within the framework, and/or when the knowledge is captured using a similar term, it is considered that the term is covered. If the upper ontology contains the term but it is not used, we consider that the term is not included. As shown in Table 3.3, PMK covers several categories in robotics domain. Decision making and choice category is covered because of the description of a robot system which enhances the execution of plans with the support of the PMK ontology. Based on the beliefs about the workspace (reachability of objects, feasible actions to execute, etc.), the system makes decisions about the distribution of actions among different robotic arms, and also about action’s parameters, slightly modifying the original plan. The Perception and situation assessment category is covered because of the integration of a tagged-based vision module within PMK. In this module, the tags are used to detect the poses and IDs of world entities and asserting them to the PMK to build the domain knowledge. Then, a reasoning mechanism is used to provide the reasoning predicates related to perception, object features, geometric reasoning, and situation assessment. Particularly, for situation assessment, an evaluation-based analysis is proposed which generates relations between the agent and the objects in the environment based on the perception outcomes, being, these relations used later to facilitate the planning process. The Problem solving and planning category is covered because of the ability of PMK to serve as tool for any planner to reason about task and motion planning inference requirements, such as robot capabilities, action constraints, action feasibility, and manipulation behaviors. The Reasoning and belief maintenance category is covered because of the ability of PMK to generate semantic maps of the robot’s workspace enhancing its belief maintenance. By means of computer vision methods, the robot detects objects and its properties (e.g. poses) and, using the ontology, Page 64 of 151 3.6. Summary of the chapter Cognitive Capability KnowRob ROSETTA ORO CARESSES OROSU PMK Recognition and categorization (Beßler et al.,2019)–(Lemaignan et al.,2010) (Menicatti et al.,2017)– – Decision making and choice – – – (Bruno et al.,2019)–(Diab et al.,2019) Perception and situation assessment (Beetz et al.,2015a)–(Sisbot et al.,2011)– – (Diab et al.,2019) Prediction and monitoring (Beetz et al.,2012)– – – – – Problem solving and planning (Beßler et al.,2018c)– – – – (Diab et al.,2019) Reasoning and belief maintenance (Beßler et al.,2018c)–(Warnier et al.,2012) (Bruno et al.,2019)–(Diab et al.,2019) Execution and action (Tenorth et al.,2014) (Stenmark et al.,2015)–(Sgorbissa et al.,2018) (Gonçalves and Torres,2015) (Diab et al.,2019) Interaction and communication (Yazdani et al.,2018)–(Lemaignan et al.,2011) (Bruno et al.,2018)–(Diab et al.,2017) Remembering, reflection and learning (Beetz et al.,2015b) (Topp and Malec,2018)– – – – Table 3.3: List of cognitive capabilities for the autonomous robotics domain and their coverage in the different chosen frameworks/ontologies. It is possible to find the reference to the articles in which the different reasoning capabilities are addressed using the ontologies. it stores a symbolic representation of the workspace. A reasoning process over those symbolic beliefs allows, for instance, to make assumptions about abstract spatial relations (e.g. cup on the table). The Execution and Action category is covered because of the ability of PMK to link the action representation in two forms. The former is the representation of the preconditions and effects of each action in an ontological form, then the planner can query over the PMK to retrieve the required information. The latter is to link the PDDL representation form with ontology. For each action, the planner can query over the PMK to retrieve the required information from a PDDL file. The Interaction and Communication category is covered because of the ability of physics-based motion planner to query over the PMK to reason about the interaction parameters with the physical objects in a certain environment. 3.5.3 Enhancement This study concludes that PMK gets an impact on modeling and reasoning levels. However, it needs to be tested in more planning approaches like knowledge-enabled approaches such as (Beßler et al.,2018c). Moreover, although PMK is an open-source library, it needs to be well-documented. 3.6 Summary of the chapter In this chapter, a ontological framework called PMK is proposed to provide useful knowledge to guide and facilitate the planning process within a classical-based manipulation planning framework. This planning framework facilitates the combination of task and motion planning (TAMP) approaches which includes Fast Forward (FF), a classical symbolic planning approach to compute the sequence of actions to be done in a certain task, and physics-based motion planning which deals with motions and possible interactions with the objects. The tools proposed to provide useful knowledge to the planning process called Perception and Manipulation Knowledge (PMK) is proposed. It provides, on the one hand, a standardized formalization Page 65 of 151 CHAPTER 3. Knowledge Guidance for Task and Motion Planning under several foundations such as the Suggested Upper Merged Ontology (SUMO) ,and the Core Ontology for Robotics and Automation (CORA) in order to facilitate the shareability and reusability when the interaction between humans and/or robots is done. On the other hand, the inference mechanism is proposed to reason about TAMP inference requirements, such as robot capabilities, action constraints, action feasibility, and manipulation behaviors. Moreover, PMK allows to break the closed world assumption of classical-based manipulation planning approaches. This proposal has been tested for serving task in a table-top manipulation problem. Page 66 of 151 Chapter 4 An Ontology-based Approach for Failure Interpretation and Recovery in Planning and Execution This chapter introduces knowledge-based failure interpretation and recovery tools for table-top manipulation problems, especially in the assembly domain. This tool provides a representation way under SUMO and DUL foundations. Moreover, a reasoning mechanism to reason on geometric components is proposed such as the feasibility of actions and motion constraints in a logic-based planning system. This reasoning mechanism requires the integration of low-level geometric planning modules (that include motion planning, collision check, inverse kinematics, and object placement) to feed back the planner on whether the preconditions of actions are met. Therefore, in this chapter, a geometric and failure interpretation and recovery ontologies are proposed, linked with a low-level geometric module, to provide a heterogeneous way of reasoning that helps any planning system. Some motivating examples have been introduced to illustrate the proposal, which has been tested with a logic-based planner. 4.1 Problem statement and proposed solution Autonomous mobile robot manipulators may not show a robust performance when placed in environments that are not tightly controlled. An important cause of this is that failure handling often consists of scripted responses to foreseen complications, which leaves the robot vulnerable to new situations and ill-equipped to reason about failure and recovery strategies. Instead of libraries of hard-coded reactions that are expensive to develop and maintain, more sophisticated reasoning mechanisms are needed to handle failure. This requires an ontological characterization of what failure is, what concepts are useful to formulate causal explanations of 67 CHAPTER 4. An Ontology for Failure Interpretation and Recovery in Planning and Execution Figure 4.1: FailRecOntThe combination of the planning system for assembly tasks with the proposed geometric reasoning and failure interpretation/recovery capabilities in orange. failure, and integration with knowledge of available resources including the capabilities of the robot as well as those of other potential cooperative agents in the environment, e.g. a human user. We propose the FailRecOnt framework as a step in this direction. We have integrated an ontology for failure interpretation and recovery with: 1) a logic-based planning framework in assembly domain, 2) a heuristic-based task and motion planning framework, such that a robot can deal with uncertainty, recover from failures, and deal with human-robot interactions. 4.2 FailRecOnt Framework In this section, an overview of the framework, the description of the proposed modules in the low-level and the knowledge-level, and how the data is managed are explained in detail. 4.2.1 Overview The proposed framework – FailRecOnt – is the continuation of KTAMP framework presented in chapter 3, and it is composed of the same three main layers (planning and execution layer, knowledge layer and assistant layer) although now the knowledge layer has been extended, as shown in Fig. 4.1. Page 68 of 151 4.2. FailRecOnt Framework The knowledge layer contains a set of knowledge to guide the planning and execution layer: 1. Awareness module: that contains a) PMK knowledge as described in detail in chapter 3; b) Geometric knowledge to provide the geometric reasoning responsible for checking the feasibility of the actions. 2. Recovery module, that provides knowledge to interpret the failures and proposes recovery strategies like: 1) asking a human for assistance for unsolvable tasks by the robot, 2) guiding the robot to autonomously recover itself, for instance by calling a sensing module to figure out the current scene of the world or 3) keep repeating the same action with another parameter (e.g., repeat a grasping action with a different angle). The next subsections are organized to facilitate capturing the contributions of this chapter. First, Sec. 4.2.2 describes the integration of the proposed geometric ontology to give access to the geometric reasoning module. The main advantage of this is that, instead of calling the module manually from the task planning module, as done in the previous chapter, the robot can query over the knowledge to retrieve the sequence of processes required to execute such actions in an automatic way. However, this requires the implementation of a mechanism that has the ability to figure out the failures that may occur. For this purpose, failure interpretation and recovery ontologies are proposed in Sec. 4.2.3 where the failure concepts, modeling and causes are described. 4.2.2 Geometric knowledge Concepts describing geometric queries An ontological module is defined to cover geometric notions used in robotics such as collision checking, placement feasibility, etc. This ontology also describes concepts like geometric querying, query status, and status diagnosis, and contains terms for particular queries such as IK (inverse kinematics) for a specific robot. Some important terms in the ontology are briefly detailed 1: •Geometric querying is defined as an Event in which some spatial reasoner– e.g. a collision checker– participates, and which is classified by/executes a GeometricReasoningTask. •GeometricReasoningTask is defined as a classification of the Geometric querying. It is represented as: GeometricQuerying v(∃isClassifiedBy.GeometricReasoningTask)u(∃hasParticipant.SpatialReasoner) u(=1hasStatus.QueryStatus). We also say that GeometricReasoningTaskv ∀ isExecutedBy.GeometricQuerying. 1(Description logic is used, see Appendix B for details.) Page 69 of 151 CHAPTER 4. An Ontology for Failure Interpretation and Recovery in Planning and Execution Recovery strategies A recovery strategy is a method a robot can apply to repair or reconstruct a plan whose execution resulted in a failure. Because replanning is a time-consuming operation, instead, repairing the plan could be an option. Some recovery strategies have been proposed such as: 1. Repeat last action: A robot could either employ a “repeat last action” strategy or instead have to address the underlying failure cause. We can axiomatically encode that a ”repeat last action" is not appropriate when the failure has a sustained cause. 2. Remove failure cause: Depending on the nature of the underlying cause of a sustained failure, we can use the ontology to formulate new subgoals as part of a RemoveFailureCause strategy by checking which of the robot-known situations get classified as UnrealizedPrecondition, and hence the unrealized preconditions become goals for new planning queries. 3. Ignore failure: A failure is not necessarily a problem for continuing a plan. If the expected outcome is an object located at X, but the object is at Y, this might not be a problem if the object is not needed again and not in the way for other actions. Geometric reasoning beyond the scope of the ontology decides whether an object is “not in the way”. We say an IgnoreFailure strategy may be appropriate only when the situation actually manifesting prevents no other expected situations. In order to check whether the prevents relations hold or not – so as to ascertain whether impedes holds or not – we would have to defer to other reasoning modules. Competency questions The proposal is to exploit the ontology to discover information helpful for the robot to evaluate and, when needed, overcome a failure. When a failure is detected, the ontology should be able to provide answers to the following failure questions (FQ) we imagine the robot raises: 1. Why was an Action classified as failure? 2. Is the failure causing a problem for subsequent activities? 3. If the failure is causing a problem (and so must be addressed), does it matter why it happened and if so what is the causal explanation for the failure? 4. If the failure is causing a problem, what are appropriate recovery strategies? 5. Assuming a recovery strategy is pursued, how can successful recovery be assessed? Page 76 of 151 4.3. The integration of the proposed ontologies with a logic-based planning Regarding FQ1, this ontology formalizes concepts to organize the robot’s information about the action executed and the expected vs. actual situation, and identify aspects of the mismatch e.g. failure location or functional aspects such as involved resources. The robot’s various reasoning modules, such as motion planning and navigation, cooperate in answering FQ2. This cooperation is guided by the ontology in that it formalizes queries about prevention relations between ongoing situations resulting from the robot’s past actions and future situations corresponding to goals being achieved, queries to be handled by mechanisms appropriate to the nature of those situations; e.g. navigation can check whether a particular object placement prevents a base movement. FQ3 reveals how the ontology converts information about prevention relations (in this case, between situations that have already happened and ongoing to situations that are postconditions of the failed action) into selection criteria for recovery strategies. In particular, it allows distinguishing between causes that must be addressed, and hence require defining new subgoals, from causes that can be ignored. FQ4 is answered by the ontology by the criteria it defines for recovery strategy applicability. These criteria are necessarily incomplete, in that we often can say when a recovery strategy is not appropriate, but this in itself is not proof that the recovery strategy actually is appropriate. Ultimately, it is the robot’s recovery attempt that is the final judge. However, reasoning guided by the ontology can filter out infeasible candidate strategies. Finally, for FQ5, the recovery strategies defined in the ontology also describe what counts as a success, because “success” is dependent on the strategy and the original transformation expectation. For example, a failed goal is simply abandoned by an IgnoreFailure, whereas new subgoals are defined for RemoveFailureCause. 4.3 The integration of the proposed ontologies with a logic-based planning In assembly applications in table-top manipulation problems, assembly recipes can be used and elegantly be represented in description logic theories (see Sec. 2.7 and Appendix B). With such a recipe, the robot can figure out the next assembly step through logical inference. However, before performing an action, the robot needs, on one hand, to ensure various spatial constraints are met, such as that the parts to be put together are reachable, non occluded, etc. On the other hand, if failures occurred while planning or/and during the execution phase, the robot must be able to recognize them, their sources and how to recover them. Such inferences are very complicated to support in logic theories, but specialized algorithms exist that efficiently compute qualitative spatial relations considering the possibility of fail, such as whether an object is reachable. Page 77 of 151 CHAPTER 4. An Ontology for Failure Interpretation and Recovery in Planning and Execution Figure 4.2: Proposed Integration. The brown ontologies provide by KnowRob group (Beßler et al.,2018a), while the blue and yellow boxes are proposed for this integration. In this chapter, as shown in Fig. 4.1, the FailRecOnt framework is proposed, which combines a logic-based planner, described in Sec. 2.7, with geometric reasoning and failure interpretation/recovery capabilities, to enable robots to perform their tasks under spatial constraints. The geometric reasoner is integrated into the logic-based reasoning through decision procedures attached to symbols in the ontology. Moreover, each assembly step is formally defined in a workflow that structures the step, and that can be automatically executed in a declarative knowledge base. As shown in Fig. 4.2, the integration of geometric and recovery module is proposed for the geometric failures and the behavior of the robot when encountering the aforementioned failures. This integration is useful to analyze the inconsistencies if they exist, using the feedback of the geometric computable predicates (a reasoning process that reports the failures) that are capable to evaluate those inconsistencies. The Geometric reasoner consists of two submodules; the failure solver and success. The former is used to reason over the Geometric analysis ontology about the the failure types. Moreover, it can provide solution(s) by calling the corresponding module from the ontology, if it exists, or provide different solutions to solve the failure. The latter is used to report the feasibility of actions. Page 78 of 151 4.3. The integration of the proposed ontologies with a logic-based planning Figure 4.3: Motivating example in assembly domain showing cases that need the planner to use the geometric, failure and recovery ontologies to interpret query and action results. Description of reachability and collision problems for the red objects (top and bottom wings). Motivation example We are mainly interested in assembly manipulation tasks for bi-manual robots, which often encounter complexity or failures in the planning and execution phases. Planning phase failures typically refer to failures of the planner itself, but we will use planning phase failures to also refer to situations where the planner reasons that some action would be infeasible, e.g. because objects block access to what the robot should reach. A correct selection of grasps and placements must be produced in such an eventuality. Depending on the type of problem, goal order must be carefully handled especially in the assembly domain; very large search spaces are possible, requiring objects to be moved more than once for achieving the goals. Execution phase failures refer to hardware failures related to the system devices e.g. robot or camera needs to be re-calibrated, or software failures related to the capabilities offered by specific software components, or failures in action performance such as an unexpected occluding object, or slippage. We selected a toy plane assembly, as shown in Fig. 4.3 targeted at 4-year-old children for the Page 79 of 151 CHAPTER 4. An Ontology for Failure Interpretation and Recovery in Planning and Execution experimental evaluation. The toy plane is made of 21 plastic parts that are mainly put together using a loose slide in connections, and fixed with bolts afterward. The parts are comparably huge such that grasping them is easier. We use a dual armed YuMi robot and further test the assembly planning in The Kautham Project (Rosell et al.,2014) which is an open-source project for motion planning. It provides the exibility to plan motions under geometric, and physics-based constraints. For the proposed experiments, we use the RRT-Connect motion planner (Kuffner and LaValle,2000), (Gillani et al.,2016), and the inverse kinematics approach developed by Zaplana (Zaplana et al.,2018). Kautham further provides a ROS-based interface, and the YuMi robot has also a set of existing software services that we use for controlling the robot. Ontologies are encoded using the Web Ontology Language (OWL) (Antoniou and van Harmelen,2004), and designed using the Protégé 2editor. Assembly Activities in Cluttered Workspaces Assembly tasks often have a fixed recipe that, if followed correctly, would control an agent such that available parts are transformed into an assembled product. These recipes can elegantly be represented using description logic. But inferring the sequence of assembly actions is not sufficient for robots because actions may not be performable in the current situation. This is, for example, the case when the robot cannot reach an object because it is occluded. A notion of space, on the other hand, is very complicated in a logic formalism, but specialized methods exist that efficiently compute qualitative spatial relations such as whether objects are occluding each other. The proposed solution is depicted in Fig. 4.2. We build upon an existing planner and extend it with a notion of action, and geometric reasoning capabilities. Actions are represented in terms of the action ontology which also defines action preconditions. Pre-conditions are ensured by running the planner for the action entity. This is used to ensure that the robot can reach an object, or else tries to put away occluding objects. To this end we integrate a geometric reasoner with the knowledge base. The interfaces of the geometric reasoner are hooked into the logic-based reasoning through procedural attachments in the knowledge base. More details will be described in Sec. 4.4.1 4.4 Modifications on knowledge representation of assembly Some modifications in the ontological relation have been introduced to enhance the logic-based planning presented in section 2.7 in order to smoothly reason over the proposed ontologies. 2(http://protege.stanford.edu/) Page 80 of 151 4.4. Modifications on knowledge representation of assembly To connect two parts, they must be in the correct fixture for the intended connection. It could be the case that a fixture blocks a required affordance. In that case, the part should be moved into another fixture that exposes the required affordance, such that the required affordance is exposed. To ensure this, we assert that required affordances must be unblocked: we add another axiom that restricts the assemblesConnection relation of the action: ∀assemblesConnection.(∀usesAffordance.FreeAffordance)(4.1) It asserts that any affordance used by the connection must not be blocked. This enforces the robot to use a fixture that exposes the required affordance. Finally, we define partOccludedBy ≡hasAffordance ◦occludesAffordance−which relates parts to parts occluding them, and assert that parts cannot be occluded by other parts when the robot intends to put them together: ∀assemblesPart.(≤0partOccludedBy.MechanicalPart)(4.2) This is used to make the robot put away parts that occlude other parts that provide required affordances for this action. 4.4.1 Heterogeneous reasoning process The proposed reasoning system is heterogeneous, which means that different reasoning resources and representations are fused into a coherent picture that covers different aspects, as shown in Fig. 4.3 and described in detail in Sec. 4.2. In this section, we will describe the two different reasoning methods used: geometric reasoning and knowledge-based reasoning. a) Geometric Reasoning The main role of geometric reasoning is to evaluate geometric conditions of symbolic actions. Two main geometric reasoning processes are provided: Reachability Reasoning: A robot can transit to a pose if it has a valid goal configuration. This is inferred by calling an Inverse Kinematic (IK) module and evaluating whether the IK solution is collision-free. The first found collision-free IK solution is returned, and, if any, the associated pose. Failure may occur if either no IK solution exists or if no collision-free IK solution exists. Spatial Reasoning: We use this module to find a placement for an object within a given region. For the desired object, a pose is sampled that lies in the surface region, and is checked for Page 81 of 151 CHAPTER 4. An Ontology for Failure Interpretation and Recovery in Planning and Execution Figure 4.4: The sequence of assemble the Battat toy. The scene has been modeled using PMK ontology. collisions with other objects, and whether there is enough space to place the object. If the sampled pose is feasible, it is returned. Otherwise, another sample will be tried. If all attempted samples are infeasible, the reasoner reports failure, which can be due to a collision with the objects, or because there is not enough space for the object. b ) Knowledge-based Reasoning In this project, knowledge-based reasoning refers primarily to checking whether an individual obeys the restrictions imposed on the classes to which it is claimed to belong, identifying an individual based on its relations to others, and identifying a set of individuals linked by certain properties (as done when identifying which parts have been linked, directly or indirectly, via connections). The geometric reasoner is integrated through computable geometric relations. The robot can then reason about them by asking questions such as “what are the occluded parts required in a connection?”: ?−holds( needsAffordance ( Connection , Affordance ) ) , holds( hasAffordance (Occluded , Affordance ) ) , holds( partOccludedBy (Occluded , OccludingPart ) ). Occluded=’ PlaneBottomWing1 ’ , OccludingPart=’ PlaneUpperBody1 ’ . Page 82 of 151 4.4. Modifications on knowledge representation of assembly Figure 4.5: YuMi plane assembly execution sequence. The robot can also reason about what action pre-conditions are not fulfilled, and what it can do to fix this. This is done by creating a planning agenda for the action entity that only considers pre-condition axioms of the action: ?−e n t ity ( Act , [an , action , [ type , ’ ConnectingParts ’ ] , [ assemblesConnection , Connection ]]) , agenda_create ( Act , Agenda ) , agenda_next_item (Agenda , Item ) . Item = " detach PlaneBottomWing1 partOccludedBy PlaneUpperBody1 " The robot can reason about what action it should perform that establishes a planning decision in its belief state. It can, for example, ask what action it should perform to dissolve the partOccludedBy relation between parts: ?−holds( usesActionMapper ( Strategy , Mapper) ) , property_range (Mapper , mapsItem , Pattern ) , individual_of (Item , Pattern ) , c a l l (Mapper , Item , Action ) . Action = [an , action , [ type , ’ PutAwayPart ’ ] , [movesPart , ’ PlaneUpperBody1 ’ ] , . . . ] . 4.4.2 Case study: Use of failure ontology in robotic assembly manipulation planning To illustrate the proposal the assembly of the Battat toy is proposed, as shown in Fig. 4.3, Fig. 4.4 shows snapshots of the resulting plan in simulation and real environment, respectively. The system is tested with different spatial constraints. Geometric reasoning about occlusions allows the robot to know when it needs to move parts Page 83 of 151 CHAPTER 4. An Ontology for Failure Interpretation and Recovery in Planning and Execution out of the way and change the action sequence provided by the logic-based planner. Some geometric situations are used for testing. The sequence to plan the assembly operations is: 1. call IK module to check reachability for grasping the objects; 2. call a collision checker to validate a path to grasp an object. Some failures can happen and get reported to the planner, where the failures are interpreted and a decision on the next action is made. Some situations in manipulation domain may happen often, such as the case where an object is blocking the chosen configuration to grap/place an object. This situation requires the selection of alternative feasible (or reachable) grasping poses and/or placements. For example, as shown in Fig. 4.3, if the object BottomWing has four grasping poses from each side g1-g4, the g1 and g4 are occluded by the holders of PropellerHolder and TopWingHolder respectively, that means g1 and g4 are not feasible and the robot is not able to reach object BottomWing through them. Meanwhile, the robot may not be able to grasp the object BottomWing through g2 and g3 because of the infeasibility of IK configurations. By querying over the proposed ontology, the robot will be able to analyze the cause and report to the planner. The failure symptom produced when planning to use g2 and g3 for grasping is a ReachabilityFailure, which is a CapabilityFailure, whereas a failure produced when planning to grasp using g1 or g4 is an OcclusionFailure, which is an AffordanceFailure. Both of these are InceptionFailures which prevent the planned task from even being undertaken. CapabilityFailure can be addressed by generating new capabilities, e.g. selecting new grasping poses that are reachable. Affordance failures, meanwhile, can be addressed by manipulating the environment to better expose its affordances, so by generating intermediary goals of moving the occluders out of the way, the robot can eventually grab the BottomWing. To interpret the causes of those situations presented in Fig. 4.3, the geometric ontology is integrated with the failure ontology as described in Fig. 4.6. This ontology describes the failure symptoms, as well as the FailureNarratives which make use of these symptoms to classify failures. The narratives may include other information to enhance the diagnosis process, such as the initial goal and participating objects in the agent’s task, and a diagnostic to indicate which component failed. Failure symptoms are ontologically characterized also in terms of what failure diagnostics they are compatible with; for example, a ReachabilityFailure can only be used by a failure narrative where the explanation role is played by a failure diagnostic that names some IK component as the failure cause. These modules (i.e, IK and collision check) are low-level modules that the symbolic level considers to be spatial reasoners. For more explanation on how to use the FailRecOnt framework, another example is proposed Page 84 of 151 4.5. Summary of the chapter Figure 4.6: An interpretation of blocking object using the proposed failure ontology. The concepts in blue belong to failure ontology, meanwhile the ones in yellow are from the geometric ontology. (see Sec. 5.9.4 Using recovery module within SkillMaN) in the next chapter using the results achieved in this chapter. 4.5 Summary of the chapter This chapter proposes: 1. The formalization and implementation of the failure interpretation and recovery ontologies as well as geometric ontology to extend the capabilities of autonomous robots related to manipulation tasks that require task and motion planning along with execution. 2. The integration of geometric ontology with failure interpretation and recovery ones, which is required for the combination of geometric and symbolic levels of planning to interpret such failures. 3. The heterogeneous way of reasoning which improves the robot capability to execute complex manipulation tasks. In the modeling level, the absolute concepts of the aforementioned ontologies are modeled under DUL and SUMO foundations to facilitate the usability for roboticists community. A case study is introduced to illustrate the use of failure ontology in automated planning and workflow execution phases by proposing the common situations that could be encountered by such planner. The aforementioned ontologies have the interface to access the low-level geometric modules to feed them back the required qualitative spatial reasoning. Page 85 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning 5.3.1 Sensing module We extend our work presented in chapter three (PMK), by enhancing the sensing module. The sensing module integrates different types of sensors, RFID, with one-dimensional output data, and RGB-D camera, with multi-dimensional output data. The purpose of the multi-sensory integration is to cover the non-line of sight (NLOS) by using RFID technology as exploited in (Deyle,2011), and line of sight (LOS) by using a camera. This integration allows the robot (especially the ones that have navigation capabilities) to figure out where the objects are located in an indoor environment (even if these objects are hidden, like cans inside a drawer), and the status of the objects (e.g., a can is full or empty). a) RFID technology An RFID technology is composed of three main parts: reader, tags and antenna. The tags, like the ones used in (Deyle et al.,2014), have a physical storage medium that allows a robot to store relevant data related to the status of the object or its relative location or spatial relationships, information that can be automatically updated from the result of the robot actions. The use of RFID technology has appeared from the beginning of this century and most of the related works are focused on localization, like the works presented in (Deyle et al.,2014) and (Li et al.,2010). However, few efforts have been done to utilize the memory inside the tags for autonomous manipulation tasks, which requires implementing robust strategies to store and update the data in memory. Here, in this work, we make use of associated memory to store the dynamic data and update them accordingly to support the planning system by extracting the relevant information. The purpose of using RFID in SkillMaN is: •To partially localize the objects in the indoor environment. This allows the robot to start planning under partial information of the environment instead of discovering the entire environment which increases the computational cost of the planning process. This includes figuring out the hidden objects that the other sensors like camera can not detect. Then, the integration with other sensors like a camera can precisely recognize the objects. •To store relevant data regarding the status of the objects in the environment, such as a can is full or empty, in order to adapt the manipulation behavior of the robot. b) Camera Using the camera and visual tags attached to the objects, the object poses are obtained (more complex untagged-based pose estimation algorithms could be used). Then, spatial relationships are extracted geometrically to understand the state of the physical world. Relationships Page 92 of 151 5.3. Assistant modules currently supported by the framework are in,on,inside,right and left as presented in (Diab et al.,2019). The tags are used to identify the world entities and semantically link them to the the properties of each object. Specifically, the purpose of integrating the sensing module with a camera is to precisely detect the position of the objects and their IDs and assert them on the relevant ontology. Then, evaluate the spatial relations of the world entities with respect to each other and the robot (e.g., object A is located on the right side of the robot and on the left side of object B.) 5.3.2 Geometric module The geometric module provides several services that help a planner to evaluate the feasibility of the skills, as described in chapter four. It can be summarized into four main services: 1. Inverse Kinematics (IK) used to compute the robot configurations for a given gripper pose, 2. Collision Check (CC) used to check the feasibility of a single configuration or a trajectory, 3. Motion Planning (MP) used to generate a sampled-based trajectory to be executed, and 4. Object Placement (OP) used to sample the placement region. In SkillMaN, these services may be required in many situations in the manipulation domain, such as the case where an object is blocking the chosen configuration to grasp/place an object. This situation requires the selection of alternative feasible (or reachable) grasping poses and/or placements. Specifically, these services are used to: 1. Compute an alternative grasp or an alternative placement pose for the object. 2. Compute the IK for the new grasp or the IK for the new object placement according to the current grasp. 3. Compute a collision-free path for the new goal configuration. These services may be required during planning to find a feasible solution, or to generate recovery strategies to recover a plan whenever a failure occurs. 5.3.3 Adaptation module This is a module that adapts the robot motions to the actual scene based on the perceived poses of the objects in the environment. Broadly, there are two sources from where to adapt the Page 93 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning motions, one is from human demonstrations, the other is from collision-free motions computed by a motion planner. This is similar to what human do, i.e., we intuitively know how to perform the motion primitives, although our exact movements are only produced when we see the objects and adapt them to the scene context while we perform the skill. The technique used for imitation motions has been the Dynamic Movement Primitives (DMPs), that implements a set of differential equations that allow describing any motion. Complex motions have long been thought to be composed of sets of primitive actions that are executed together. DMPs are a mathematical formalization of the motions of the primitives using dynamical systems theory. These dynamical systems have stable behaviors using the basic set of parameters provided by the DMP tool, although extra parameterization according to the task at hand may improve the results. 5.4 Knowledge modules Formally, knowledge is divided into knowledge representation and inference mechanism. The former copes with how the knowledge is represented, the latter copes with how to infer the relevant knowledge. In SkillMaN, the main goal is to capture knowledge about 1. how the planning system can be guided by the knowledge, 2. how the similarity of the situations can be checked, 3. how to manage the perception system, 4. whether a path is feasible or not, 5. how can robots perform skills and what skills are needed to achieve certain goals, 6. how can a situation be interpreted as a failure, and 7. which are the available recovery strategies for a given failure. The knowledge modules are first introduced and then the heterogeneous inference mechanism will be detailed at the end of the subsection. 5.4.1 Experiential knowledge module The knowledge-based experience, or experiential knowledge, is divided into two main parts: knowledge for planning and situational knowledge. Page 94 of 151 5.4. Knowledge modules Figure 5.4: The situation modeling in SkillMaN. The blue and orange are abstract concepts, while the red referred to the instances of the classes. The multi-sensory module is used to build the description of the environment. A) Knowledge for planning: One of the significant requirements for the planning system is to reason about how to perform skills in semi/unstructured environments. This requires having geometric information, based on the current situation of the environmental entities, of how to manipulate the objects, e.g. which is the experience-based feasible grasp. This is what we call "geometric skills experience". Currently, in SkillMaN, there are two sources from where to build the geometric skills experience: humans and robots. For humans, the user can build manually the geometric skill experience including the description of the task constraints (either programmed or included in an ontology). For the robots, if the robot starts exploring the way of executing an action and finds a feasible solution, it stores this solution to be later used if required. For example, let’s consider the side-grasp is used for picking an object from a table and there is an obstacle occluding the path from a certain angle, several angles could be applied to explore the feasibility of the grasping configurations. Once found, the robot stores these configurations to be used in similar situations. This knowledge is required to guide the planning system especially when some motion constraints exist. In the proposed case study, several constraints have been introduced to show the importance of using geometric skills experience within a planning system as shown in Sec. 5.8. B) Situational knowledge: In SkillMaN, experiences are thought to be situations that provide a relational context on a set of events that occurred, and objects that were involved. This includes, e.g., what roles an Page 95 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning object plays during an skill, and what the diagnosis is in case a failure was raised during skill execution. Situations in our knowledge base, as described in Fig. 5.4, can be written as a tuple < S, O, E >, where Sis the skill that was executed, Othe set of objects that were involved, and Ethe set of events that occurred. The representation strategy of the Descriptions and Situations ontology (Maass et al.,2007) has been followed where descriptions are used to create views on the relational context of situations. In particular, the proposed framework associates skills to the situations where the skill was executed, and exploits this information for the realization of a set of inference mechanisms. The environment’s description is used to semantically link low-level perception data with high-level knowledge, and to analyze the situation of the environment entities in order to enhance the task execution. The tagged-based sensors, i.e., RFID and camera, are used to identify the world entities and semantically link them to the properties of each object. Specifically, the purpose of the sensing module is to detect the position of the objects and their IDs and assert them on the ontology to build the description of the environment and its relevant instances following the Perception and Manipulation Kowledge (PMK) presented in chapter 3. C) Experiential data: Our system records sensory data over time and associates it to situations during which the data was acquired. This is mainly to capture the trajectories that were executed by the robot, and to associate them with task, environment, and execution. This is useful for machine learning applications where expressive queries can be answered at higher levels of the knowledge base, and results of such queries may serve as filter for the lower-level data to gather only the data matching a semantic situation. In SkillMaN, two types of storage mediums are used, the memory of RFID and knowledge database. The former is used to store the dynamic data such as objects’ positions and their status (e.g., a can is full or empty). The latter is used, beside guiding the planning system, to store the static data such as objects’ features. Data management is detailed in Sec. 5.4.5 5.4.2 Awareness module This module contains low-level knowledge related to perception, to geometric issues required for the evaluation of the actions feasibility, and to the way of executing skills. A) Perceptual knowledge To perceive a robot environment, different sensors are usually used. Sensors provide data Page 96 of 151 5.4. Knowledge modules Figure 5.5: The representation of perceptual knowledge in SkillMaN. Page 97 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning about the environment in the form of signals (one dimension) or images (multi-dimension), and to obtain the useful features from the perceived data the suitable algorithms have to be applied, for instance to detect an object pose some pose estimation algorithms based on image features can be applied, or alternatively algorithms based on tags identification can be used. The complexity increases when the integration between the sensors exists. Perceptual knowledge is the knowledge related to the robot sensors or to sensors associated to the environment. This knowledge is used to guide the proposed multi-sensory module. A first version of the perceptual knowledge was presented in chapter 3, where two cameras worked in parallel to perceive the table-top environment. Here, we enhance the representation of the knowledge to be capable of working with different types of sensors including RFID and camera. The perceptual knowledge for multi-sensory integration in SkillMaN, as shown in Fig. 5.5, is represented as a tuple < D, C, A > where D is a measuring device (sensor), C is the sensor constraints or limitations, and A is the corresponding algorithm to extract the features from the sensor signals. This knowledge is responsible to answer three main questions which are the sensors attached to the robot?, what type of data the sensors perceive and what are their limitations?, how to extract the relevant data?. To answer the first question, a description of the sensors is proposed to make the robot understand which are the group of sensors it has. Moreover a description of the components of each sensor, like the tags, antennas and reader of the RFID is included. To answer the second question, a description of the perceptual features is proposed to clarify to the robot which type of data (i.e, one or multi dimensions) the sensors are perceiving, and what are the constraints or limitations of each sensor. To answer the third question, a method to call the corresponding algorithms is proposed to extract the relevant data. Using Description Logic (DL, (Baader et al.,2017)), the knowledge of RFID sensors is expressed as: RF IDKnowledge :− ∃hasSuperclass(RF ID, Sensor) ∧∃Sense(RF ID, RSSI) ∧∃hasSensingComponents(RF ID, T ag) ∧∃hasSensingComponents(RF ID, Reader) ∧∃hasSensingComponents(RF ID, Antenna) ∧∃hasID(RF ID, taggedID) ∧∃hasConstraints(RF ID, minRange) ∧∃hasConstraints(RF ID, maxRange) ∧∃hasAlgorithm(RF ID, readT ag) And the knowledge of camera is expressed as: Page 98 of 151 5.4. Knowledge modules CameraKnowledge :− ∃hasSuperclass(Camera, Sensor) ∧∃Sense(Camera, Image) ∧∃hasConstraints(Camera, minRange) ∧∃hasConstraints(Camera, maxRange) ∧∃hasAlgorithm(Camera, tiagoCam) B) Geometric knowledge The geometric knowledge has a structure for sequential access to the geometric services in the assistant layer. The main advantage of this ontology is that, instead of calling the module manually from the task and motion planning client, the robot can query over the knowledge to retrieve the sequence of processes required to execute such actions in an automatic way. C) Skill knowledge A skill, in SkillMaN, is a description of what the robot can do. The SkillMaN provides some methods of teaching new skills to the robot inspired by the work presented in (Munawar et al., 2018): 1. Primitive skills consist of a sequential list of atomic actions, which refers to a single action or gesture, including its preconditions and effects. For example, an openDrawer skill is composed of the sequence of actions: move to the handle position, close the gripper, and finally pull the drawer. 2. Rule-based skills consist of a set of “if A then B rules" to issue appropriate gestures according to sensors outcome. Both methods, however, cannot be executed on their own. They require a structure, such as a workflow, that contains the abstract steps that are usually required for task execution. This structure is described at a symbolic level and grounded to be attached to each skill using the assistant layer. The main difference between the both aforementioned methods is a perceptionbased conditional node in the structure. That means the structure includes some branches of a given value that should be sensed. Page 99 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning 5.4.3 Recovery module Knowledge for recovery is a module that provides an interpretation of the failures that occur. In SkillMaN, an interpretation failure ontology described in chapter 4covering several sources of failures, is used both during planning and execution. It offers recovery strategies for: 1. Geometric failures, that may appear when e.g. the robot can not reach to grasp/place an object, there is no collision-free path or there is no feasible Inverse Kinematic (IK) solution; 2. Hardware related failures that may appear when e.g. the robot in a real environment requires to be re-calibrated (gripper or arm), or it is sent to a non-reachable configuration; 3. Software agent related failures, that may appear when e.g. the robot has software components that fail like when an algorithm is not able to extract the proper features. 5.4.4 Heterogeneous inference mechanism This section presents a heterogeneous way of reasoning that includes symbolic reasoning over the knowledge module and geometric reasoning. The former includes filtering the situation from the database, situation similarity check, skill reasoning, semantic reasoning regarding the environment and its entities, manipulation constraints and perception. The latter includes the geometric reasoning to check the feasibility of the generated skills. They are discussed below. Symbolic reasoning A.1) Filtering situation: Filtering situation is a process of finding those situations that satisfies a skill description. It means the robot has to detect the situations that use a specific skill in their description, e.g. using a Prolog predicate (WIELEMAKER et al.,2012) the reasoning on “which are the situations that contain a certain skill?” is: ?−f i l t e r S i t u a t i o n ( h as Skill ( Situation , S k i l l ) ) , ?−f i l t e r S i t u a t i o n ( hasParticipant ( Situation , Object ) ) , ?−f i l t e r S i t u a t i o n ( hasPart ( Situation , Event ) ) . Situation=[S kill , Object , Event ] . A.2) Situation similarity check: The similarity of situation scenes is computed using taxonomic information from the situational ontology together with information about what makes up the compared entities. Note that scenes are compound entities – that is, a scene has objects and agents as participants. Page 100 of 151 5.4. Knowledge modules Figure 5.6: An example of similarity check between two scenes. This example is also a part of the experimental scenes of scenario one in Sec. 5.8 (storage task). Objects and agents themselves are compound entities; an object or agent may have other objects as parts. Also, the description of an agent includes the skills to be executed, geometric-skills experience, and agent goal. Entities that are considered simple – the parts of objects or agents – are compared using WuPalmer similarity (Wu and Palmer,1994), although for numerical stability reasons, the logarithm of this similarity score is used here, i.e., for two individuals x and y: Sim(x, y) = log depth (lca C(x), C(y)) 0.5(depth C(x)) +depth C(y)) (5.1) where C(x)is the class to which individual xbelongs, depth(A)is the depth of class A in a taxonomy, and lca(A, B)is the lowest common ancestor of classes A, B in that taxonomy. The intuition behind Wu-Palmer similarity is that similar classes should be close to each other in the taxonomy. To compare individuals x, y that are compound entities, their parts are matched such that for every part xpof x, we find the part ypof ythat maximizes Sim(xp, yp). Then, the sum of the similarity scores obtained from these matching is added to Sim(x, y). The intuition here is that we want to have the similarity of complex objects such as situations or scenarios to depend on the nature of those scenarios as well as their participants. We only compare the “tree” of part-hood relations for efficiency reasons. In principle, there may be many stored scenes one could compare the current situation to, and filtering out most of them so that only a few relevant candidates remain. Once some candidate similar scenes are selected, the more intensive procedures of adapting robot motion from the stored scene to the current one can be used to ascertain the usefulness of the stored experience for the current task. Page 101 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning 5.7 Implementation and set-up 5.7.1 Implementation tools A) Perception The C++ library ar–track–alvar (http://wiki.ros.org/ar_track_alvar) has been used to detect the object pose and ID. Moreover, the C++ library ThingMagic Mercury API (http://www. thingmagic.com/manuals-firmware) of RFID technology has been used to detect the objects, including the hidden ones, and to store the relevant dynamic information. Some services are implemented to read the tagID, read the data from memory and write/update the data on the memory. These IDs are asserted in the knowledge to extract a semantic description of the object. All the transformations of the objects and camera are calculated with respect to the world frame. B) Planning and adaptation A planning system consists of two main phases: task planning and motion planning. The first is implemented using the Fast Forward (FF) task planner to generate a sequence of actions. The latter is implemented using The Kautham Project (Rosell et al.,2014). The Kautham Project is a C++ based open-source tool for motion planning, that enables to plan under geometric and kinodynamic constraints. It uses the Open Motion Planning Library (OMPL) (Sucan et al., 2012) as a core set of sampling-based planning algorithms. In this work, the RRT-Connect motion planner is used to generate a path between two configurations. The main technique onto which the imitation motions have been implemented is the DMP (Ijspeert et al.,2002). The DMP experiments are performed first in simulation and afterwards using the real robot. The experiment consisted in learning by recording the execution of the planned-base motion and then changing the initial and final points to see how the planned gestures are imitated. An interface has been implemented to command the arm to perform such motions that require imitation gestures. This interface can be divided into three main parts, the data acquisition process, the DMPs generation, and the execution of the motion. The data acquisition is performed by recording the motion. The DMPs generation is done using the motion recorded as input to learn how to perform the DMP primitive in a new situation. The execution of the motion uses the initial configuration and goal state required to adapt the motion in similar situations. The integration between the planning and adaptation tools is done automatically together in the preparation phase. Page 108 of 151 5.7. Implementation and set-up In SkillMaN, the proposed abstract primitives are described: 1. releaseGripper: an atomic action used to open the gripper. 2. closeGripper: an atomic action used to close the gripper. 3. pick-place: an skill that contains the atomic actions move, hold and put-down; it is used for transferring the objects between two locations. 4. openDrawer: is an skill that contains the atomic actions move, hold and pull; it is used for opening/closing the drawers. 5. serving: is a skill that contains the actions move and pour; it is used for serving the beverages to a customer. The motions that are adapted are initially either computed by the motion planner, e.g, in case 3 and 4, or copied from human demonstrations, e.g, in case of 5. In case of 1 and 2, the motion of closing and opening the gripper is predefined. These abstract primitives correspond to the basic functions of the robot manipulator, which can be implemented in many different ways. Our way of implementing such primitives is at the lowest motor control level. The focus of our work is, however, not a specific implementation, but rather we would like to propose a way to combine them to seamlessly perform skills. C) Knowledge The knowledge is designed using ontology web language (OWL) using the Protégé ontology editor (http://protege.stanford.edu/). Ontology instances can be asserted using information processed from low-level sensory data. Queries over the knowledge to reason or check the similarity are based on SWI-Prolog and its Semantic Web library which serves for loading and accessing ontologies represented in the OWL using Prolog predicates. A ROS (Robot operating System) interface has been implemented in order to facilitate the query-answer process as a client-service communication. The PMK approach, as presented in chapter 3is used in this work. It is explicitly implemented to enhance Task and Motion Planning (TAMP) capabilities in the manipulation domain. It is integrated with the multi-sensory module allowing the instances to be asserted to the ontology using information processed from low-level sensory data. Page 109 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning Figure 5.9: Navigation experiment: (a) the plan view of the indoor environment, (b) the real scene of how the robot detects the tables used in the indoor environment, and (c) path planning, obstacle avoidance capabilities, and navigation poses on the map. D) Navigation and mapping Fig. 5.9 describes the navigation strategy proposed in SkillMaN. In (a), the plan view of the indoor environment has been shown. By using the mobile capacity of TIAGo, it plans toward the navigation position of the objects (e.g, TIAGo navigate toward the picking and serving tables) until it detects the labels attached to them, as shown in (b). During the navigation, TIAGo has the capabilities of path planning with obstacle avoidance and localization of the objects in the map, as shown in (c). All the objects in the environment are localized with respect to the reference frame. D.1) Navigation: TIAGo has autonomous navigation functionalities implemented using the ROS 2D navigation stack (http://wiki.ros.org/navigation). This package is one of the most commonly used to implement mapping and autonomous navigation solutions in robots running on ROS. It takes in information from odometry and sensor streams and outputs velocity commands to send to the mobile base. This navigation software is composed of several different ROS nodes, services and topics that are able to perform SLAM (Durrant-Whyte and Bailey, 2006). Using the information stored on the map and the data of its surroundings provided by different sensors, this package is capable of computing a suitable path to lead the robot to a certain goal position without hitting any obstacle. D.2) Mapping: The mapping and pose generation process starts by creating the occupancy Page 110 of 151 5.7. Implementation and set-up Figure 5.10: Task management and the communication with the ROS-based services from symbolic and low-level modules. grid map of the environment of the robot. To obtain it, the gmapping (http://wiki.ros.org/ gmapping) package installed in the robot has been exploited. This map is necessary for the navigation to successfully move through the room avoiding any collision. D.3) Localization: Localization is achieved by working with the amcl package (http://wiki. ros.org/amcl). This package is a probabilistic localization system for a robot moving in 2D. It implements the adaptive Monte Carlo localization approach (MCL), which uses a particle filter to track the pose of a robot against a known map. MCL generates a cloud of particles which represent the possible states of the robot distribution. Each particle represents a possible pose and orientation of the robot on the map. 5.7.2 Task manager algorithm The SkillMaN is not implemented for a specific task, it is quite general and it accepts several tasks in indoor environments with the consideration of some changes regarding the description of the environment, as discussed in Sec. 5.9.3. All the modules mentioned in Fig. 5.3, and the provided services of each module as described in Fig. 5.10, are used by the task manager. The task manager is responsible to call these services in order to autonomously execute the tasks, as described in Algorithm 1. The sensing module is managed through the perceptual knowledge, the following service is Page 111 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning Algorithm 1: taskManager 1initialState ←runPerception (RFID) // run RFID sensor to perceive the environment using the perceptual knowledge 2while Task goal not delivered do 3D = loadP DDL // load the domain and problem files from skill knowledge 4P = F F (D)// plan at symbolic level to compute the sequence of skills 5skillName ←firstAction(P) 6while skillName do 7objects, poses ←runPerception(Camera) // perceive the actual state of the environment according to the skill 8Y = skillName, objects // store the current situation. 9F =filterSituation(skillName) // return a set of situations that use the same skill 10 S = similarityCheck(F, Y) // return the situation which is most similar in the similarity check, if any 11 if S! = emptySet then 12 Mot ←loadMotion (S) 13 expKnow ←experientialKnow // return the geometric-skills experience 14 AdaptedMot ←adaptMotion(Mot, expKnow) // adapt motion into the current situation 15 C=checkF easibility(AdaptedMot)// verify the feasibility of the adapted motion 16 if AdaptedMot =feasible then 17 execute(AdaptedMot) 18 store[AdaptedMot, skillName, Y ]// store the executed motion, skill and the current scene situation 19 if S=emptySet or C=infeasible then 20 Mot =generateMotion(skillName)// generate a collision-free motion for a new situation 21 if Mot =feasible then 22 execute(Mot) 23 store[Mot, skillName, Y ] 24 skillName ←nextAction(P) Page 112 of 151 5.7. Implementation and set-up used for this purpose: •runPerception used to select the corresponding algorithm(s) associated to the available sensors. The sensing services are used to provide the initial state of the environment to the planner or whenever required, using RFID sensor and cameras. The RFID sensor has four main services: 1. InitializeRFID used to set up the RFID system (i.e., reader, antennas and tags), 2. ReadRFIDTag used to read the RFID tags ID associated to the entities, 3. ReadRFIDMem used to read the dynamic data stored in tags’ memory, and 4. WriteRFIDMem used to update the tags’ memory. The camera has two main services: 1. Cam status used to initialize the camera, and to input (from either a human guidance process or a motion planner) the motion to be adapted, and 2. LocateCam used to estimate the objects poses and their IDs. In the planning phase, two services have been used to call the the heuristic-based task planner FF (Fast Forward) and to load the domain and problem files: 1. loadPDDL used to automatically load the PDDL domain and problem files, as described in Sec. 5.4.4 2. FF used to automatically compute symbolically the sequence of skills to be executed. In the analysis of each action the solution plan, with a guidance from knowledge modules, the following services are used: 1. filterSituation used to filter the situations that include a specific skill in the database, 2. similarityCheck used to compare the current situation with the others stored in the database, Page 113 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning 3. experientialKnow used to provide geometric skills, based on the robot experience, e.g. the type of grasp according to the current situation. The geometric services are manged through the geometric knowledge by calling following service: •generateMotion used to compute the initial and goal configurations (according to the action/skill to be performed and the reachability and spatial reasoning predicates proposed in Sec. 5.4.4), and the collision-free path between them using the motionPlanning service. The geometric services are used to combine the geometric module with a symbolic planning level to guarantee the feasibility of the planned skills. This module has four main services, provided by The Kautham Project: 1. motionPlanning used to compute a collision-free path, 2. collisionCheck used to verify whether a robot configuration is collision-free or an object at a given pose is not interfering with others, 3. inverseKinematics used to compute the robot configurations for a given desired pose of the end-effector, 4. objectPlacement used to sample/check the availability of placement locations for the objects. The adaptation services are used to imitate/adapt the motion of each skill to be executed in such situations. There are two services managed through the geometric knowledge: 1. trainMotion used to input (from either a human guidance process or a motion planner) the motion to be adapted and returns the weights used to shape the this motion, and 2. adaptMotion used to compute the initial and goal configurations (according to the action/skill to be performed and the reachability and spatial reasoning predicates proposed in Sec. 5.4.4), and to adapt the trained motion is verified with the trainMotion service. This interface is established based on ROS (Robot Operating System) service-client communication. 5.7.3 Experimental set-up The experiment has been done at IOC lab and it is composed of: Page 114 of 151 5.7. Implementation and set-up Figure 5.11: a) the robot plans how to open the first drawer; b) the robot executes the openDrawer skill; c) the robot plans how to pick the black can (based on its status, here it is empty) with the help of experiential knowledge about what is the best grasp to place it in the first drawer; d) the robot executes the place skill; e) the robot executes the close action using the rule-based skill; f) the robot checks the similarity of the current situations, it finds the same skill has been used with the same object (a drawer in the file cabinet), then executes the skill with the same motion used to open the first drawer; g) the robot plans how to pick the red can (based on its status, here it is full) with the help of experiential knowledge about what is the best grasp to place it in the second drawer; h) then, the robot executes the place skill. Video URL: https://www.youtube.com/watch?v=bTmWAkjC93c 1. The TIAGo robot. 2. A file cabinet with four drawers. 3. A storage table that contains the objects (cans). 4. Several cans that may be full or empty. 5. A serving table that contains a cup on a tray where the robot must pour the contents of a can to a customer. 6. A perception system with camera and an RFID sensor that includes: •A reader that has the capacity of reading four antennas, distributed around the lab, and let the robot determine the region where the objects are located. •The tags which have a unique ID and a memory with a space of 64 characters. Page 115 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning Figure 5.12: a) the robot checks the similarity of the current situation, it finds the same skill has been used with the same object (i.e., a drawer in the file cabinet), then adapts the skill with the same motion used in the database; b) the robot figures out the top-grasp is not feasible for pouring action, the top of the file cabinet is used as a placement room to change the grasp type; c) the robot changes the grasp type from the top-grasp to the side-grasp; d) the robot serves the contents of the can in the cup to a customer, the serve motion is adapted from the experience, according to the current pose of the robot and location of the cup. Video URL: https://www.youtube.com/watch?v=bTmWAkjC93c 5.8 Experimental scenarios Before describing the proposed scenarios, some assumptions should be taken into account. 1. All the object models are defined in the knowledge database. 2. All atomic actions/skills to be used are described with their preconditions and effects in knowledge database. 3. All the RFID’s antennas are distributed in the environment in a way that avoids the interaction between the signals received from each one. That means that each antenna only receives the information of the tags located in its coverage region. 4. All environmental entities are labeled with either RFID or vision-based tags. 5. All the features received from the multi-sensory perception system (RFID and camera) are reliable enough. 5.8.1 Scenario one: Storage task Fig. 5.11 shows a sequence of snapshots of the storage cans task. The task is to classify the cans on the table to store them in the drawers based on their status. Firstly, the robot checks the status of the selected can by reading this information from RFID memory. After computing the symbolic plan, the robot can apply the skill openDrawer (the first action of the symbolic plan) Page 116 of 151 5.8. Experimental scenarios to the corresponding drawer of the file cabinet, as shown in Fig. 5.11 a-b. Then, to generate a collision-free path, the motion planner has been called. Then, with guidance from semantic knowledge and the experiential knowledge (if geometric experience exist), the robot can reason about how to apply the pickUp skill to the can from the table and how to putDown it inside the drawer and close the drawer, as shown in Fig. 5.11 c-e. After similarity check process of the current situation i.e., comparing it with the ones that have a similar description stored in the database, the imitation process is used to imitate those skills to be applied for the other cans, as shown in Fig. 5.11-f. The second can is full and the corresponding drawer where to be stored is the second one, shown in Fig. 5.11 g-h. The perception system is used to check if the preconditions of the actions are satisfied or not. For example, to apply the openDrawer skill in the second drawer, the first one should be closed to allow the robot to putDown the selected can correctly in the drawer. 5.8.2 Scenario two: Serving task Fig. 5.12 shows a sequence of snapshots of the serving can task. The task to serve the contents of the can inside a cup on the tray. The robot can not apply the pouring skill with the grasping pose used to pick it up from the drawer (which is top-grasp). The robot needs to temporally place the can to change the grasp from the top to side grasping configuration. The top surface of the drawer is used as a free placement room for this sub-task. Then the robot is able to serve the can. Moreover, if the position of the cup is changed, the robot will be able to adapt the motion with the new position. Table 5.1: Test the skill openDrawer, pickUp and serving using adaptation method vs the planning system with and without experiential knowledge. Skill Parameter Adaptation Planning With Exp.know Without Exp.know openDrawer S – success rate 100 100 50 T – Avg. time (sec.) 4.5 11.5 39.5 pickUpFromDrawer S – success rate 100 100 50 T – Avg. time (sec.) 12.5 33.5 58.5 serving S – success rate 100 – – T – Avg. time (sec.) 12 – – Page 117 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning Figure 5.16: The conditional plan results from the planning process. a) flowchart describing the plan obtained by the contingent FF. b) flowchart added when executing the plan for monitoring and repair if necessary the action outcomes shown in red in the plan. Figure 5.17: The executable plan. •Transfer: a skill done by the robot to move an attached object between poses. •Push: a skill done by the robot to push an object from one pose to another one. •Open: a skill done by the robot to open a box-like container (articulated cap with prismatic joint is assumed with two positions corresponding to fully closed and fully opened, the state being stored in the containers objects features). •HumanTransfer: a skill done by a person to transfer/push an object to the robot workspace. •HumanOpen: a skill done by a person to open a box-like container. Sensing actions do not involve motion, and are devoted to observe object status. The observation is done at run-time. The sensing actions considered in the example are : •CheckColor: a sensing action done by a robot to determine object color. •CheckPose: a sensing action done by a robot to determine object pose. Page 124 of 151 5.9. Discussion Figure 5.18: The integration of contingency plan with recovery knowledge. •CheckContainer: a sensing action done by a person to evaluate whether a container is open or not. •CheckCan: a sensing action done by a person to evaluate whether can-like objects are filled or not. The reasoning actions are devoted to interpret execution phase failures, providing a strategy (or maybe more according to the failure type) for recovery. The reasoning process is done at run-time. In this example the following reasoning action is considered: •Transfer-to-box-FailureInterpretation: a reasoning action over failure ontology interprets failure cause while transferring an object in a container, and provides a recovery action for the robot or human depending on the interpretation process. The contingent Fast-Forward planner finds a the conditional tree of manipulation skill plans to execute the task, sensing actions will be involved in the branching nodes, and reasoning actions will be associated to the monitoring of the execution of the more sensitive manipulation skills. C) The integration of recovery module in SkillMaN with conditional plan The complete conditional tree of plans is represented in Fig. 5.16. While the planning process is taking place, there are several challenges in terms of interpretation of the actual state which Page 125 of 151 CHAPTER 5. A Skill-based Robotic Manipulation Framework based on Perception and Reasoning is captured by a sensory action and handled by the proposed state interpretation. The main challenge is to include in the plan the knowledge about who is responsible for the recovery action execution (i.e., human or robot). That means, for instance, that the robot can autonomously repair the failure of transferring the blue cylinder B inside the box if it interprets that the box is closed. The plan automatically obtains branching nodes in which sensing and/or reasoning actions are assigned to monitor manipulation skill outcomes in a semi-automatic way. Logically, the sensing/reasoning actions should be assigned after each manipulation skill, however this increases the computational cost. Instead, we assign them to some manipulation skill that are expected to have a high probability of failure. The reasoning action Transfer-to-box-FailureInterpretation is assigned to the action TransferB-box1 based on monitoring the result of the skill execution. The result has a Boolean outcome, the success sequence is automatically obtained by the contingent FF planner as shown in Fig. 5.16-a). If, while monitoring, a failure occurs, the reasoning action interprets the cause of the failure which is a failed task execution that has unmanifested postconditions (i.e., NonrealizedSituation). The recovery strategies have been provided based on the current status of the box (closed or flipped) as shown in Fig. 5.16-b). Recovery strategies could be either to ask help from humans or the robot recovers the failure by itself. The other FALSE outcome branch of the sensing action CheckContainerBox1 states the box is already closed and based on the planner, the transfer of the box cap is applied. The final executable plan generated by the planning system has a set of feasible manipulation skills as shown in Fig. 5.17. The planner may ask the ontology questions about why cylinder B has not been transferred to the box?, once it interprets the current situation (i.e., the box is closed), then the next request is which is the best recovery strategy?. These requests of the planner are handled and answers by retrieving information, updating/deleting or reasoning over it. As shown in Fig. 5.18a, the request-answer relation is done using the service-client communication of ROS (Robot Operating System, www.ros.org), and in Fig. 5.18-b, the request-answer queries of the planner are described. 5.10 Summary of the chapter This framework discusses the importance to integrate perception, planning, knowledge-based reasoning (including experience), in a skill-based manipulation framework to let the robot automatically perform the tasks that include every-day activities. Moreover, the framework also includes the procedures to determine how to manage the data required to efficiently perform the tasks. Two examples have been introduced. In both examples, a set of skills such as pickUp, putDown, openDrawer and serving are introduced. The first example with two scenarios including manipulation in indoor environment has been introduced to show the capabilities of the robot to use the proposed modules to execute the every-day tasks in semi-structured Page 126 of 151 5.11. Enhancement environments. For every-day tasks, the adaptation method is powerful in terms of time when the robot already has experience of how to execute the task. For planning, experiential knowledge is used as a geometric-skill experience to facilitate the planning process and reduce the cost that increases due to the exploration process. In the second example, the integration of the framework with a recovery module has been done, besides the manipulation skills, a sensing and reasoning actions are introduced to interact with the physical environment and monitor the results of the manipulation skills. The system shows flexibility to be adapted in several environments and robotic structures. 5.11 Enhancement Future work will be how to increase the adaptation capability to work with more complex situations which include spatial relations. Also, finding out the way to automate the process of establishing a new concept of an unknown environmental entity. Page 127 of 151 Chapter 6 Conclusions and Future Work 6.1 Conclusions The present thesis has developed several frameworks based on perception, reasoning, learning, and planning to address the increasing challenges of robotic manipulation problems. Different sorts of modeling and reasoning processes have been also proposed inside the frameworks to come up with a feasible manipulation plan. To sum up, the challenges of robotic manipulation problems considered in the thesis are summarized as follows: •Table-top manipulation problems: The need to reason on the current state of the world in order to apply one action or another (like pick or push an object to take it apart) in a table-top manipulation scenario. KTMP framework offers the way to encode the knowledge and reason upon it. •Assembly manipulation problems: The need to reason on the result of actions or the need to evaluate actions preconditions (like the feasibility of a given grasp to execute a pick action) in an assembly task. FailRecOnt framework offers a flexible reasoning tool that is perfectly adapted to knowledge-driven planning schemes. •Mobile-based every-day manipulation problems: The need to make the robot aware of situation similarities to efficiently re-use previous known ways to execute actions by adapting the robot motions to stored patterns (like opening one drawer once the robot knows how to open another). The SkillMaN framework integrates the previous tools with a similarity situation evaluator and a motion adaptation tool. Two approaches of planning are used in this thesis, heuristic-based approaches (i.e., Fast Forward (FF) and contingent-based FF), and knowledge-based planner provided by the KowRob 129 CHAPTER 6. Conclusions and Future Work group. The execution of manipulation tasks with knowledge-based planning approaches not explicitly prepared for TAMP, like (Tenorth and Beetz,2009), can be a challenge because this would require, on the one hand, from the knowledge perspective, to provide all the components in a way that they match with their planning system. On the other hand, from the planning perspective, they would require the definition of the recipes (strategies) for executing the tasks (sequence of actions), including all possible strategies of execution and the way to switch between them when required, which can be a very expensive process, especially for tasks that need long sequences of actions, such as those involving manipulation in cluttered environments. Some other planning approaches rely on PDDL and on planning strategies best fit to cope with difficult task planning challenges, like those found in the manipulation domains, although the use of PDDL implies a closed-world assumption, which precludes their use in more dynamic environments that could require perception and knowledge-based geometric reasoning. PMK allows to break the closed world assumption of classical-based manipulation planning approaches. Robots, like any other agent, sometimes fail. Knowledge-based robots can recover from failure by reasoning whether to try once more, to try something else or to move to other tasks. We argued that the choice has to be based on the conceptual (ontology), the planning (task) and the execution (feasibility) levels. This requires to integrate traditional robotics domains (the robot has to act) and AI (the robot has to plan) with unplanned situations (the robot is in an unexpected state). The integration of all these views raises a variety of research questions, and so does our work which addresses only part of this research topic. For instance, hardware related failures that may appear when e.g. the robot in a real environment requires to be recalibrated (gripper or arm). Also, software agent related failures, that may appear when e.g. the robot has software components that fail like when an algorithm is not able to extract the proper features. One of the focus of this thesis has been the justification and development of FailRecOnt, a general and reliable framework for failure and recovery management. Foundationally, the proposed ontologies are modeled under SUMO and DUL foundations. We found that SUMO concepts have some limitations in the terms descriptions and some missing vocabulary that we proposed in PMK ontology. On the contrary, DUL has a well-structure and wide range of meaningful concepts that can be flexibly used in such domains. To increase the robot autonomy, the integration of services that are necessary for everyday activity tasks, with a knowledge source, reasoning engine and skills descriptions are very important. Moreover, the use of experiential knowledge is very useful when the robot encounters the same situation. Moreover, the adaptation methodology saves time comparing to motion planning. However, huge efforts must be done in this line to enhance the capabilities of learning from demonstrations in such situations. Finally, the frameworks have been illustrated to show the main tools and the flow of information among modules used to perception, reasoning, learning and planning levels. Concerning the robotic systems, we have tested our results with the robots Yumi ABB and TIAGo PAL. All the related results are shown in videos in URL: https://www.youtube.com/ channel/UC6lZ7d7qm5wh5v3fsbEIFgg?view_as=subscriber and alternatively in https://sir.upc. Page 130 of 151 6.2. Future Work edu/projects/kautham/Videos.html. 6.2 Future Work Along with the conclusion points stated above, the current thesis, moreover, opens new research problems that require further consideration such as: From the many issues that the KTMP framework raises, in the future we aim to •increase the reasoning capabilities of the framework; •reduce the uncertainty of the low-level information by using deep learning techniques. •increase the abstract concepts to include some notions like behavior which is important to be used in constraint-based manipulation planning. •benchmark PMK with other approaches that includes more concrete metrics in relation to overall system performance, compatibility with other frameworks (e.g. planners, perception systems), extensibility, reusability, scalability, types of applications it can be applied to. From the many issues that the FailRecOnt framework raises, in the future we aim to •enrich the causal explanatory module; •improve the search for an optimal match between what is known about a detected failure and the recovery strategies; •include recovery strategies from a false belief state (e.g, caused by false detection), and •optimize the interconnections among the FailRecOnt submodules. From the many issues that the SkillMaN framework raises, in the future we aim to •increase the robot autonomy by implementing a library of actions/skills and a sophisticated reasoning mechanism to allow the robot to reason on the best action/skill that can be used in the current situation; •increase the adaptation capabilities to cover more complex manipulation problems which requires spatial reasoning; Page 131 of 151 CHAPTER 6. Conclusions and Future Work •build a failure-based experiential knowledge that allows the robot to prevent the repetition of its mistakes. Some of these works are already in preparation. Their development shall make the robots smarter and more adaptive. Page 132 of 151 Appendices 133