Planning with Natural Language for Task-Oriented Robots

1 Planning with Natural Language for Task-Oriented Robots Enrico Saccon, Kuba di Quattro, Edoardo Lamon, Matteo Saveriano, Paola Quaglia, Marco Roveri, Luigi Palopoli Abstract— This work outlines a framework that combines knowledge representation systems with planning capabilities for robotics applications. It is based on four main pillars: •Automated knowledge extraction from natural language descriptions using large language models. •Generation of temporal plans for multi-robot systems using inference rules. •Automatic translation of plans into executable formalisms. •Runtime monitoring of the plan execution, learning from exceptions, and updating the knowledge-base dynamically. I. INTRODUCTION Artificial Intelligence (AI) is becoming an increasingly integral part of people’s lives, revolutionizing the way we interact with information and with robots. One of the central challenges in robotics and AI is the management of knowledge that enables robots to autonomously devise plans, solve tasks, and execute them effectively. In particular, we want to focus on −Multi-agent collaboration: robotic interaction should be harnessed to solve more complex tasks; −Robustness: the robots’ solutions should be robust and explainable by incorporating planning theories and uncertainty. This work builds on the PLANTOR1framework [1], with a particular focus on enhancing its planning component. PLANTOR introduces several key contributions that make it a powerful and versatile tool. First, it enables the automated generation of knowledge bases (KBs) directly from natural language descriptions using Large Language Models (LLMs). It also supports explainable plan generation in multi-agent scenarios, making the decision process transparent and interpretable. Furthermore, it incorporates probabilistic reasoning by automatically constructing Markov Decision Processes (MDPs) to model uncertainty in the environment. Finally, it allows for experience-driven adaptation, meaning that the KB can be continuously refined through feedback. As it can be seen from Figure 1, the framework is structured into three main modules: −Knowledge Management System (KMS): automates and streamlines the creation of high-quality Prolog KBs from natural-language prompts, leveraging LLMs with few-shot prompting to reduce manual effort and errors; −Planner: derives feasible temporal task plans for multiple robots from the KB; −Execution Module: executes the generated plan via a Behavior Tree (BT); Co-funded by the European Union under the NextGenerationEU (FAIR - Future AI Research - PE00000013), under project INVERSE (Grant Agreement No. 101136067), under project MAGICIAN (Grant Agreement n. 101120731) and by the project MUR PRIN 2020 (RIPER - Resilient AI-Based Self-Programming and Strategic Reasoning - CUP E63C22000400001) Department of Information Engineering and Computer Science, and Department of Industrial Engineering, University of Trento, Trento, Italy. {name.surname}@unitn.it, [email protected]. 1https://www.github.com/idra-lab/plantor Fig. 1: The structure of PLANTOR. The planner module can be replaced with the process explained in Section II. As aforementioned, in this work, we will generally present the modules of PLANTOR, but we will especially focus on the planner for which we propose an improved way of generating temporal task plans from the KB. II. CURRENT FRAMEWORK Knowledge Management System. The goal of this module is to take a prompt description from an user and generate a Prolog KB. The prompt must contain information regarding the environment, the actions and the abilities of the robot, as well as the initial and the goal states. This module heavily relies on LLMs to generate the KB from NL, with the models refined only using few-shot prompting, i.e., passing examples when querying the model. For this task, we use three LLMs, which can either be the same model or different models. A first LLM checks the validity of the input query, e.g., it queries whether or not the LLM thinks the prompt is feasible with the available tools. A second LLM is used to generate a high-level KB of the prompt, i.e., predicates and actions that represent more general and abstract aspects of the task. Finally, a third LLMs, takes both the input prompt and the high-level KB and generates a more precise KB with the actions corresponding to ones directly executable by the robots. As it is possible to see in Figure 1, for both the last two LLMs, a consistency check (CC) is carried out. This allows for identifying syntactic and semantic errors. The division on two levels allows the LLM to focus on different aspects and generate a more robust KB, with direct mapping to the robots APIs at the second level. This has been shown both in internal studies and from other works [2]. Planner. Initially we had exploited Prolog’s inference mechanics to obtain a total order plan, i.e., a sequence of actions, from the KB that would represent a plan to go from the initial to the final state. We would subsequently extract all 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629742 115 2 INIT PARALLEL SEQUENTIAL a1SEQUENTIAL WAIT FOR a1,a2a4 SEQUENTIAL a3SEQUENTIAL a2 Legend: Action Control Condition Fig. 2: An example of BT. the causal relationships, i.e. creating a partial order, between the different actions to parallelize them on the different agents available and then minimize the makespan of the plan using an optimization tool such OR-Tools. While such approach allowed us to have multiple level of customization, it also proved to be generally slower than state of the art planner for PDDL [3]. For this reason, we devised a system that allows to translate the Prolog KB in a an intermediate formalization, which encodes all the essential information in the KB and can be translated into formalisms such PDDL or Unified Planning Framework (UPF) [4]. This system works on three steps: I. Structural extraction: we parse the Prolog KB and recover predicates, parameters, initial and final states, and preconditions and effects of the actions. This step is crucial as it must deal with Prolog syntax, such as wildcards and negative effects, which not all PDDL versions support. This step also infers the type signatures of the predicates through iterative analysis. II. Intermediate JSON representation: extracted knowledge is serialized into a lossless JSON, capturing types, predicates, actions and states. This step also enables to decouple parsing from code generation, allowing for using different backend planners in the future. III. UP and PDDL code generation: it compiles the JSON file into UPF code. This also allows for exporting PDDL code through UPF APIs. The process is key-agnostic, as it does not rely on specific rules to translate predicates and can operate with any Prolog KB generated by PLANTOR. It is also reproducible, ensuring that the outputs of the workflow are fully deterministic. Finally, it is designed to be independent from the final planner, allowing the integration of multiple backends. Once the UPF or PDDL files have been generated, we can use standard planners to solve for temporal plans which are then translated into BTs [5]. Execution Module. Once we have a BT as shown in Figure 2, then we can directly execute it on the robots through one of the many libraries, like ROS2 or BehaviorTree.CPP. III. RESULTS Knowledge Management System. We tested [1] the KMS on two use-cases: blocks world, inspired by the classical domain where blocks are re-allocated using one or more robotic arms, and arch, which adds complexity through pillars and architrave. For each use-case, we created multiple prompts varying the number of agents and blocks, thus changing the KB complexity. The results are shown in Table I: a ✓indicates a fully correct KB, while X denotes an incorrect one. When a fixable number of errors occurred, the values # Predicates High-level Low-level HL LL GPT4 120K GPT4o GPT4 120K GPT4o Blocks world 1 137 197 X(2,14) ✓X✓ 2 154 205 X(3,17) ✓X X(1,8) 3 145 193 X(1,2) ✓X X(2,10) 4 133 169 X(1,2) ✓X X(1,8) 5 241 286 X(2,2) ✓X✓ Arch 1 167 222 X ✓X X(1,10) 2 172 229 X X(1,1) X X(4,18) TABLE I: Results for the generation of the high-level (HL) and low-level (LL) KBs using the LLMs. in parentheses represent the number of logical errors and required corrections. While LLMs do not always generate perfect KBs, they provide a solid first base that can be easily refined. For example, the low-level KB of the second arch test (229 predicates) required only 18 corrections, most of which were straightforward to identify through the KMS. Planner. The Prolog planner [1] proves to be very slow and lead to suboptimal plans given the recursive nature of Prolog and the depth-first approach used in the original planner. In this abstract we have presented an improvement of the planner, which leads to optimal plans and better performance. While experimental results are still preliminary, we found that steps I. (structural extraction) and II. (intermediate JSON representation) are generally fast to be completed (in the order of milliseconds) scaling linearly with the number of predicates. Step III. (UP and PDDL code generation) makes up for circa 90% of the time, although from an indepth analysis it appears that the higher time is due to the initialization of UPF. IV. CONCLUSION This paper presents a comprehensive framework that integrates knowledge representation, planning and execution for multi-robot systems. Leveraging LLMs for knowledge extraction and Prolog for inference, the system generates temporal plans translated into executable formalisms. We described an improvement to the planner, which will be integrated in the workflow to improve both computational times and optimality of the solutions. Future improvements include autonomous KB management, probabilistic planning, and monitoring for adaptive execution. The framework offers promise for automation in diverse industries, addressing challenges of robustness, uncertainty handling, multi-agent collaboration, and visual input queries. REFERENCES [1] E. Saccon et al., “A Temporal Planning Framework for MultiAgent Systems via LLM-Aided Knowledge Base Management,” arXiv preprint arXiv:2502.19135, 2025. [2] E. Gestrin et al., “Nl2plan: Robust llm-driven planning from minimal text descriptions,” arXiv preprint arXiv:2405.04215, 2024. [3] C. Aeronautiques et al., “Pddl—the planning domain definition language,” Technical Report, Tech. Rep., 1998. [4] A. Micheli et al., “Unified planning: Modeling, manipulating and solving ai planning problems in python,” SoftwareX, vol. 29, 2025. [5] J. Zapf et al.,Constructing behavior trees from temporal plans for robotic applications, 2024. arXiv: 2406.17379 [cs.RO]. 116