Adaptive autonomy in multi-agent systems with modeled interactions : an empirical analysis

FACULTY OF INFORMATION TECHNOLOGY AND ELECTRICAL ENGINEERING DEGREE PROGRAMME IN ELECTRONICS AND COMMUNICATIONS ENGINEERING MASTER’S THESIS Adaptive Autonomy in Multi-Agent Systems with Modeled Interactions: An Empirical Analysis Author Athmajan Vivekananthan Supervisor Chathuranga Weeraddana Second Examiner Sumudu Samarakoon Technical Advisor Mehdi Bennis December 2025 2 Vivekananthan A. (2025) Adaptive Autonomy in Multi-Agent Systems with Modeled Interactions: An Empirical Analysis. University of Oulu, Faculty of Information Technology and Electrical Engineering, Degree Programme in Electronics and Communications Engineering, Master’s Thesis, 41 p. ABSTRACT This thesis presents a decision-making framework for distributed systems where multiple autonomous agents operate in dynamic environments. The proposed approach integrates model predictive control with concepts from information theory and simulation-based planning. A key contribution is the development of a planning strategy that allows agents to make sequential or independent decisions while adapting for changing conditions of the environment in real time. The method handles systems with continuous state and action spaces and introduces mechanisms for dynamic agent prioritization, efficient use of computational resources, and robustness to inaccuracies in system models. Experimental results demonstrate that the method shows less vulnerability and scalability compared to traditional learning-based approaches when faced with changing conditions. These findings suggest that the proposed planning and coordination strategy is well-suited for online, scalable control in decentralized environments such as those found in large-scale sensor and actuator networks. Keywords: Multi-agent systems, IoT, Model Predictive Control, Rollout Algorithms, Decentralized Planning, Adaptive Autonomy, Monte Carlo Simulation, Online Learning, Agent Coordination, Distributed Control. VivekananthanA.(2025)Diplomityönnimi.Oulunyliopisto, tieto-jasähkötekniikan tiedekunta, elektroniikan ja tietoliikennetekniikan tutkinto-ohjelma. Diplomityö, 41 s. TIIVISTELMÄ Tiivistelmä kirjoitetaan tähän. Tämä on diplomityöpohja, mutta sitä voidaan soveltaa myös kandidaatintöihin pienin muutoksin. Pohja soveltuu suomenja englanninkielisten opinnäytteiden tekemiseen. Tämä tutkielma esittää päätöksentekomenetelmän hajautetuille järjestelmille, joissa useat autonomiset osajärjestelmät/agentit toimivat dynaamisissa ympäristöissä. Ehdotettu lähestymistapa yhdistää mallipohjaisen ennakoivan ohjauksen informaatioteorian käsitteisiin ja simulaatiopohjaiseen suunnitteluun. Tutkielman keskeinen tulos on suunnittelustrategia, joka mahdollistaa osajärjestelmien/agenttien tehdä päätöksiä joko peräkkäin tai itsenäisesti samalla, kun ne mukautuvat reaaliaikaisesti muuttuviin ympäristöolosuhteisiin. Menetelmä soveltuu järjestelmiin, joilla on jatkuvat tilaja toimintatilaavaruudet. Se sisältää mekanismeja osajärjestelmien/agenttien dynaamiseen priorisointiin, laskentaresurssien tehokkaaseen hyödyntämiseen sekä järjestelmämallien epätarkkuuden sietoon. Kokeelliset tulokset osoittavat, että menetelmä on vähemmän haavoittuva ja paremmin skaalautuva kuin perinteiset oppimispohjaiset lähestymistavat muuttuviin olosuhteisiin sopeutuessa. Tulokset viittaavat siihen, että ehdotettu suunnitteluja koordinointistrategia soveltuu hyvin verkossa tapahtuvaan, skaalautuvaan ohjaukseen hajautetuissa ympäristöissä, kuten laajoissa sensorija toimilaitteiden verkoissa. Avainsanat: Hajautetut järjestelmät, Internet-of-Things, Ennakoiva Mallipohjainen ohjaus, Rollout-algoritmit, Hajautettu suunnittelu, Adaptiivinen autonomia, Monte Carlo -simulointi, Verkkopohjainen oppiminen, Osajärjestelmien/agenttien koordinointi, Hajautettu ohjaus TABLE OF CONTENTS ABSTRACT TIIVISTELMÄ TABLE OF CONTENTS FOREWORD LIST OF ABBREVIATIONS AND SYMBOLS 1 INTRODUCTION............................................................................................. 8 1.1 Related work ........................................................................................................ 8 1.2 Contributions and organization of thesis ............................................................. 10 2 TERMINOLOGY AND BACKGROUND............................................................ 11 2.1 Multistage decision problems .............................................................................. 11 2.2 Approximation in value space - rollout algorithms ............................................. 12 2.3 Extensions to multi-agent (MA) case................................................................... 13 2.4 Sequential MA rollout.......................................................................................... 14 2.5 Autonomous MA rollout...................................................................................... 14 3 METHODOLOGY............................................................................................ 16 3.1 Certainty-equivalent model predictive control (MPC) for rollouts: Caveats....... 16 3.2 rollout augmented information theoretic model predictive control (RAIT-MPC) 16 3.3 MA-sequential rollout augmented information theoretic model predictive control (S-RAIT-MPC).............................................................................................. 18 3.4 MA-autonomous rollout augmented information theoretic model predictive control (A-RAIT-MPC)........................................................................................ 21 3.5 Learning approximate agent controls................................................................... 22 3.6 Agent ordering in sequential MA rollout............................................................. 23 3.7 MA rollout for problems with finite state and control spaces.............................. 24 4 RESULTS ........................................................................................................ 26 4.1 Experimental setups............................................................................................. 26 4.2 Performance evaluation of MA-S-RAIT-MPC and MA-A-RAIT-MPC.............. 28 4.3 Evaluation under changing disturbance distributions .......................................... 29 4.4 Evaluating optimization of agents order .............................................................. 31 4.5 Performance under varying number of Monte Carlo (MC) simulations.............. 32 4.6 Prioritized sampling: adaptive number of MC simulations allocation strategy .. 33 4.7 Evaluating the balance between look ahead (LA) depth, 𝑁𝑚𝑐𝑡𝑠 and run time ..... 35 5 SUMMARY AND FUTURE DIRECTIONS ........................................................ 36 6 REFERENCES ................................................................................................. 38 FOREWORD This thesis would not have taken shape without the guidance and generosity of many. I am deeply grateful to my supervisors, Chathuranga Weeraddana and Sumudu Samarakoon, for their insight, patience, and encouragement throughout this journey. Their mentorship helped shape not only this work, but my way of thinking. I also wish to thank Professor Mehdi Bennis for the opportunity to work within the Intelligent Connectivity and Networks (ICON) research group at CWC-RT, University of Oulu. The experience enriched my perspective, and I remain thankful to the entire team for their support and collaboration. Beyond the academic, there was a steadier current beneath it all supporting me, holding up everything else while I focused on this one task. It revealed itself in quiet, daily acts of patience, generosity, and unwavering strength. While this thesis carries my name, it rests on a foundation built by someone who gave much and asked for little. Some contributions cannot be measured in citations or funding but they are measured in presence, in belief, in love. For all of this, and more than I can put into words, I thank my wife. Oulu, June 02, 2025 Athmajan Vivekananthan LIST OF ABBREVIATIONS AND SYMBOLS AI artificial intelligence BP base policy CBL case based learning CBP continual backpropagation CRN cognitive radio network CTDE centralized training with decentralized execution DNN deep neural network DP dynamic programming DQN deep Q-network EC edge computing EV electric vehicle GIS generalized importance sampling IT-MPC information theoretic model predictive control IoT internet of things LA look ahead LLM large language model MA multi-agent MADDPG multi-agent deep deterministic policy gradient MADRL multi-agent deep reinforcement learning MARL multi-agent reinforcement learning MAS multi-agent systems MC Monte Carlo MCTS Monte Carlo tree search MDP Markov decision process ML machine learning MPPI model predictive path integral MPC model predictive control MPE multi-particle environments NN neural network PI policy iteration PPO proximal policy optimization RL reinforcement learning SA single-agent SOTA state of the art SP signaling policy VI value iteration 𝒗𝑘,𝑖 perturbation added control action of trajectory 𝑖at time step 𝑘 𝑊𝑟𝑒𝑤 window size for moving average 𝝅sequence of policies ˜ 𝝅sequence of approximate policies 𝐻control sequence generated by a specified base policy 𝐻𝑗control sequence generated by a specified base policy of 𝑗𝑡ℎ agent 𝒖𝑗control decision of agent 𝑗 Δ𝒖cost-weighted average of the sequence of trajectories 𝒂𝑗 𝑘position of agent 𝑗at time 𝑘 𝒃𝑖 𝑘position of target 𝑖at time 𝑘 𝐷size of the training dataset 𝑓𝑘discrete time system dynamics at time 𝑘 𝑔time invariant stage cost 𝑔𝑁terminal cost function ℎ(𝑥𝑘, 𝑗)local heuristic q-factor of agent 𝑗at time step 𝑘 Nnormal distribution 𝐽★(𝒙0)optimal total cost of 𝝅 𝐽𝝅(𝒙𝑘)the total cost of 𝝅starting at 𝒙𝑘 𝐽𝑁(𝒙)optimal cost of the problem involving 𝑁stages 𝑘time index ¯ 𝑘another time index 𝑚number of multi agents 𝑀number of trajectories 𝑁𝑚𝑐𝑡𝑠 number of MC simulations 𝒘𝑘random disturbance at time step 𝑘 Eexpectation 𝑂(·) computational complexity order 𝑝𝒘probability distribution of noise 𝜋𝑘policy at time step 𝑘 𝝅★optimal policy 𝝁★an optimal policy 𝑞cardinality of a local control space 𝑅𝑘reward at time 𝑘 𝑠𝑖, ¯ 𝑘cost of the 𝑖𝑡ℎ from time index ¯ 𝑘onward Σcovariance matrix Σ𝑗covariance matrix of 𝑗𝑡ℎ agent 𝒙𝑘state at time step 𝑘 𝑇𝑗length of the horizon of agent 𝑗for BProll ˜ 𝐽approximate total cost ˜ 𝝁approximate policy Ufinite control space U𝑗control space of agent 𝑗 U𝑗 cont(𝒙)continuous control space of agent 𝑗at state 𝒙 U𝑗 dis(𝒙)discrete control space of agent 𝑗at state 𝒙 V𝑗 𝑘sampled control trajectories of agent 𝑗at time 𝑘 ˆ 𝒖pre(𝑗)an approximate vector of control ac of the preceding agents 𝑖∈ [ 𝑗−1] 𝒖pre(𝑗)vector of control ac of the preceding agents 𝑖∈ [ 𝑗−1] 𝒖seq 𝑘combined action after sequential decision making 𝒖aut 𝑘combined action after autonomous decision making 𝒖succ(𝑗)vector of control ac of the succeeding agents 𝑖∈ [𝑚]\[𝑗] ˆ 𝒘𝑘predicted value of random disturbance at time step 𝑘 ˆ 𝝁𝑗() autonomous policy ˜ 𝝁𝑗approximate policy of agent 𝑗 𝒖𝑘control input at time step 𝑘 𝒖★an optimal control Xstate space Xcont continuous state space Xdis discrete state space ¯ Tset of uncaptured targets BP the base policy for discrete setting BPmpc the base policy for information theoretic model predictive control (IT-MPC) BProll the base policy for rollout 𝝐𝑗noise perturbation of agent 𝑗 D𝑗training dataset of agent 𝑗 Σcovariance matrix Σ𝑗covariance matrix of 𝑗𝑡ℎ agent 𝛾sensitivity threshold for reward 𝜎an agent order 𝜙𝑗neural network function of agent 𝑗 𝜃𝑗parameters of the neural network of 𝑗𝑡ℎ agent 14 where U𝑗(𝒙)corresponds to the control space of agent 𝑗. Consequently, the control action 𝒖is composed of 𝑚individual components, 𝒖𝑗for 𝑗=1, . . . , 𝑚, where each 𝒖𝑗∈ U 𝑗(𝒙) corresponds to the control decision of a specific agent at state 𝒙, resulting in 𝒖=[𝒖1, . . . , 𝒖𝑚] ∈ U(𝒙). The aforementioned structure of the control-space facilitates the development of efficient and reliable algorithms for computing stationary policies in MASs, which would otherwise require an overwhelming computational effort. For instance, in a finite control-space problem where |U 𝑗(𝒙)| =𝑞for all agents 𝑗and states 𝒙, the computational complexity is reduced from 𝑂(𝑞𝑚)to 𝑂(𝑞𝑚)[4]. The class of such algorithms are known as sequential MA rollout (or agent-by-agent rollout) which offers strong performance guarantees under appropriate conditions [4]. 2.4 Sequential MA rollout In this section, only the one-step LA approach is presented while the extension to the multistep case follows analogously [4, § IV-A]. The key mechanism underpinning the sequential MA rollout is alternating optimization [37]. More specifically, given a fixed ordering of [𝑚]≜{1, . . . , 𝑚}, the minimization operation in (6) sequentially iterates through each index 𝑗, optimizing over the corresponding variable 𝒖𝑗at each step. For clarity, consider the case where the ordering follows [𝑚]itself, i.e., in its natural sequence1. Under this setup, each agent 𝑗∈ [𝑚]yields its policy ˜ 𝝁𝑗(𝒙𝑘)from, ˜ 𝝁𝑗(𝒙𝑘)=arg min 𝒖𝑗∈U 𝑗(𝒙𝑘) Eh𝑔𝒙𝑘,𝒖pre(𝑗) 𝒙𝑘,𝒖𝑗,𝒖succ(𝑗) 𝒙𝑘,𝒘𝑘 +˜ 𝐽𝑓𝒙𝑘,𝒖pre(𝑗) 𝒙𝑘,𝒖𝑗,𝒖succ(𝑗) 𝒙𝑘,𝒘𝑘i.(10) It is important to note that when each agent 𝑗∈ [𝑚]solves (10), two additional problem parameters, namely 𝒖pre(𝑗) 𝒙𝑘and 𝒖succ(𝑗) 𝒙𝑘, must be known beforehand. In particular, 𝒖pre(𝑗) 𝒙𝑘is the vector of policies of the preceding agents 𝑖∈ [ 𝑗−1], i.e., 𝒖pre(𝑗) 𝒙𝑘=(˜ 𝝁1(𝒙𝑘), . . . , ˜ 𝝁𝑗−1(𝒙𝑘)).(11) Accordingly, it is assumed that a coordination mechanism is in place to facilitate intercommunication from all 𝑖∈ [ 𝑗−1]to agent 𝑗. On the other hand, 𝒖succ(𝑗) 𝒙𝑘is the vector of an appropriately chosen BPs 𝝁𝑖(𝒙𝑘)of the succeeding agents 𝑖∈ [𝑚]\[𝑗], i.e., 𝒖succ(𝑗) 𝒙𝑘=(𝝁𝑗+1(𝒙𝑘), . . . , 𝝁𝑚(𝒙𝑘)).(12) Additional approximation techniques are employed in the online implementation of the minimization in (10), similar to those outlined in (6). 2.5 Autonomous MA rollout Recall that each agent 𝑗requires the control 𝒖pre(𝑗) 𝒙𝑘of preceding agents to determine its decision ˜ 𝝁𝑗(𝒙𝑘)as per (10). In contrast, autonomous MA rollout utilizes past decision experiences 1Generalizations are discussed in our algorithms presented in § 3. 15 to model ˜ 𝝁𝑖(𝒙𝑘)for preceding agents 𝑖∈ [ 𝑗−1]using data-driven methods such as NNs. Specifically, for each agent 𝑗∈ [𝑚], a NN is trained to output ˆ 𝒖pre(𝑗) 𝒙𝑘, an approximate of 𝒖pre(𝑗) 𝒙𝑘, based on training data pairs 𝒙(𝑠),𝒖pre(𝑗) 𝒙(𝑠)for 𝑠=1, . . . , 𝑆. Consequently, the output ˆ 𝒖pre(𝑗) 𝒙𝑘of the NN is used for computing the autonomous policy ˆ 𝝁𝑗(𝒙𝑘)given by, ˆ 𝝁𝑗(𝒙𝑘)=arg min 𝒖𝑗∈U 𝑗(𝒙𝑘) Eh𝑔𝒙𝑘,ˆ 𝒖pre(𝑗) 𝒙𝑘,𝒖𝑗,𝒖succ(𝑗) 𝒙𝑘,𝒘𝑘 +˜ 𝐽𝑓𝒙𝑘,ˆ 𝒖pre(𝑗) 𝒙𝑘,𝒖𝑗,𝒖succ(𝑗) 𝒙𝑘,𝒘𝑘i.(13) Autonomous MA rollout (13), in contrast to its sequential counterpart (10), enables efficient parallelized decision-making by eliminating the necessity for inter-agent communication. 16 3 METHODOLOGY A novel algorithm has been proposed in this chapter for addressing multistage MA decision problems in continuous (infinite) control/state spaces. Broadly, the algorithms are centered on integrating MA rollout framework with approximate solution methods for stochastic optimal control, formulated within the information theoretic model predictive control (IT-MPC) framework [35], which is shown to have close technical relations to the MPPI under restricted conditions [34]. Subsequently, it is demonstrated how agent ordering can be seamlessly incorporated into MA rollout algorithms to enhance performance, followed by the development of computationally efficient variants that are well-suited for online decision-making with integrated decision-time planning. 3.1 Certainty-equivalent MPC for rollouts: Caveats Let us start by considering a SA case with classic certainty equivalent MPC. In this case, the resulting MPC formulation that is viewed as an approximation in value space with ℓ-step LA minimization (7), is given by minimize Í𝑘+ℓ−1 𝑗=𝑘𝑔(𝒙𝑗,𝒖𝑗,ˆ 𝒘𝑗) + ˜ 𝐽(𝒙𝑘+ℓ) subject to 𝒙𝑗+1=𝑓𝒙𝑗,𝒖𝑗,ˆ 𝒘𝑗, 𝑗 =𝑘, . . . , 𝑘 +ℓ−1 𝒖𝑗∈ U(𝒙𝑗), 𝑗 =𝑘, . . . , 𝑘 +ℓ−1, (14) where the variables are controls (𝒖𝑘, . . . , 𝒖𝑘+ℓ−1)and states (𝒙𝑘+1, . . . , 𝒙𝑘+ℓ). It is worth noting that the 𝒘𝑘is replaced with ˆ 𝒘𝑘, which is a predicted value of the former. However, directly addressing MPC formulation (14) typically requires solving nonlinear programming problems using algorithms such as sequential quadratic programming [38]. In addition to control constraints, multistage decision problems involving continuous state and control spaces often incorporate state constraints. While such constraints can be encoded into the cost functions 𝑔and ˜ 𝐽using non-smooth functions such as the indicator function, this approach compromises the smoothness of the overall cost function, thereby limiting the applicabilityofconventionalsolvers relyingonsequentialquadraticprogramming. Alternatively, state constraints can be handled separately using more general methods based on sequential convex programming [37]. Nevertheless, MPC formulations of the form (14) may still result in infeasible optimization problems, especially in real-time applications, thus requiring auxiliary mechanisms to address feasibility issues online. Last but not least, the functions 𝑔,𝑓, and ˜ 𝐽are not available in closed form in nearly all multistage decision problems emerging across diverse application domains [3, 4, 5]. Therefore, standard mathematical optimization solvers are generally inapplicable to solving certainty-equivalent MPC, making such formulations less promising for integration with rollout algorithms. Accordingly, the proposed rollout algorithms are developed based on IT-MPC [35], which circumvents the limitations inherent to certainty-equivalent MPC formulation (14). The characteristics of IT-MPC provide a sampling-based control framework that is conveniently amenable to integration with the rollout algorithms, as will be discussed in the sequel. 3.2 RAIT-MPC To formally present the the IT-MPC formulation to build the proposed subsequent rollout algorithms, it is useful to introduced some notations which are not originally present in the 17 ℓ-step LA minimization (7). It is assumed that there is no direct control available over the input of the dynamical system (1). Instead, over each finite horizon of length ℓ, the nominal input 𝒖𝑘, . . . , 𝒖𝑘+ℓ−1, for which the control defined above, is subject to additive Gaussian noise with known characteristics. More specifically, each control 𝒖𝑗is perturbed by a noise term 𝝐𝑗∼N(0,𝚺), resulting in the actual control applied to the system being (𝒖𝑗+𝝐𝑗). Under this assumption, IT-MPC formulated by, minimize (𝒖𝑘,...,𝒖𝑘+ℓ−1) E𝝐E𝒘Í𝑘+ℓ−1 𝑗=𝑘𝑔𝒙𝑗,𝒖𝑗+𝝐𝑗,𝒘𝑗+˜ 𝐽(𝒙𝑘+ℓ).(15) This yields that, for a given current state 𝒙𝑘, the solution to the IT-MPC (15) characterizes a control trajectory of length ℓthat minimizes the expected cost across sampled realizations of the system dynamics. Under more specific conditions, it has been shown in the IT-MPC literature that the notion of optimality of (15) coincides with that of the stochastic optimal control [35, § IV]. More importantly, the solution is approximated by a sampling based schemes precluding the need for optimization solvers [35, Algorithm 1]. The principle of the sampling based solution within the context of (15) involves the following key steps. First the algorithm generates a sequences of sampled control trajectories, V𝑘={(𝒗𝑘,𝑖, . . . , 𝒗𝑘+ℓ−1,𝑖)}𝑖∈[𝑀],(16) where 𝒗𝑡,𝑖 =𝒖𝑡+𝝐𝑡,𝑖 for all 𝑡and 𝑖with 𝝐𝑡,𝑖 drawn from the distribution N(0,Σ)and (𝒖𝑘, . . . , 𝒖𝑘+ℓ−1)being an ℓstep trajectory from an adequate BP. Second, the cost of the 𝑖th trajectory from time index ¯ 𝑘onward, denoted 𝑠𝑖, ¯ 𝑘, is computed. In particular, associated with each control trajectory 𝑖∈ [𝑀]and its time index ¯ 𝑘∈ {𝑘, . . . , 𝑘 +ℓ−1},𝑠𝑖, ¯ 𝑘is computed by 𝑠𝑖, ¯ 𝑘=Í𝑘+ℓ−1 𝑡=¯ 𝑘𝑔𝒙𝑡,𝑖,𝒗𝑡,𝑖,𝒘𝑡,𝑖+˜ 𝐽(𝒙𝑘+ℓ,𝑖),(17) where for all 𝑖∈ [𝑀],𝒙𝑘,𝑖 =𝒙𝑘,𝒘𝑡,𝑖 is drawn according to the distribution 𝑝𝒘, and for all control trajectory 𝑖∈ [𝑀], time index 𝑡∈ {𝑘+1, . . . , 𝑘 +ℓ},𝒙𝑡,𝑖 is the state landed according to (1). Finally, an approximate solution for (15), denoted (˜ 𝒖𝑘, . . . , ˜ 𝒖𝑘+ℓ−1)is computed as ˜ 𝒖𝑡=𝒖𝑡+Δ𝒖𝑡, 𝑡 =𝑘, . . . , 𝑘 +ℓ−1,(18) where Δ𝒖𝑡is a cost-weighted average of the sequence of trajectories given by Δ𝒖𝑡=Í𝑀 𝑖=1exp −(1/𝜆)𝑠𝑖,𝑡 𝝐𝑡,𝑖 Í𝑀 𝑖=1exp −(1/𝜆)𝑠𝑖,𝑡 ,(19) and 𝜆is a positive scalar. The first control input ˜ 𝒖𝑘is applied to the system, while the remaining controls define the mean of the distribution used for sampling in the subsequent time index 𝑘+1. Building upon the preceding discussion, the proposed RAIT-MPC algorithm is outlined here. For brevity, introduction of the notation 𝐻(𝑝𝒘,BP, 𝑓 , ℓ;𝒙𝑗)is presented to denote the control sequence generated by a specified base policy BP over ℓstages, starting from state 𝒙𝑗and evolving according to the system dynamics 𝑓, with disturbances at each stage sampled from the distribution 𝑝𝒘. Algorithm 1 outlines the core steps of the RAIT-MPC framework of a single agent, (extended to MAS subsequently in c.f. § (3.3) ) with infinite state and control spaces. At each time step 18 Algorithm 1 RAIT-MPC 1: Given: 𝒙0, initial state; 𝑀, number of sampled trajectories; ℓ, number of multistages; 𝑝𝒘, disturbance distribution, 𝑓, transition model, Σ, noise covariance; BPmpc, the base policy for IT-MPC; BProll, the base policy for rollout; 𝑇, length of the horizon for BProll. 2: 𝑘=0 3: while Task is not completed do 4: current state =𝒙𝑘 5: (𝒖𝑘, . . . , 𝒖𝑘+ℓ−1)=𝐻(𝑝𝒘,BPmpc, 𝑓 , ℓ;𝒙𝑘) 6: Compute V𝑘from (16) 7: For each trajectory 𝑖∈ [𝑀], compute 𝒙𝑘+ℓ,𝑖 and the rollout base policy controls for 𝑇 stages, i.e., (𝒖𝑘+ℓ,𝑖, . . . , 𝒖𝑘+ℓ+𝑇−1,𝑖)=𝐻(𝑝𝒘,BProll, 𝑓 , 𝑇;𝒙𝑘+ℓ,𝑖) 8: For each trajectory 𝑖∈ [𝑀], compute rollout base policy cost denoted by 𝑐𝑖, where 𝑐𝑖=Í𝑘+ℓ+𝑇−1 𝑡=𝑘+ℓ𝑔𝒙𝑡,𝑖,𝒖𝑡,𝑖,𝒘𝑡,𝑖(20) 9: For each 𝑖∈ [𝑀], let ˜ 𝐽(𝒙𝑘+ℓ,𝑖)=𝑐𝑖+𝑑𝑖where 𝑑𝑖is the terminal cost at 𝒙𝑘+ℓ+𝑇,𝑖 10: Compute 𝑠𝑖, ¯ 𝑘from (17) 11: Compute (˜ 𝒖𝑘, . . . , ˜ 𝒖𝑘+ℓ−1)from (18) 12: Observe 𝒘𝑘from the environment and pick ˜ 𝒖𝑘to move to 𝒙𝑘+1according to (1) 13: 𝑘=𝑘+1 14: end while 𝑘, a nominal control sequence is generated using the base policy BPmpc and perturbed with Gaussian noise to create 𝑀sampled trajectories V𝑘. The resulting post-horizon states 𝑥𝑘+ℓ,𝑖 are then evaluated using a 𝑇-step rollout with base policy BProll. The associated rollout and terminal costs define the surrogate cost-to-go e 𝐽(𝑥𝑘+ℓ,𝑖), used to update trajectory costs via (17). These trajectory costs are then used in an importance-weighted update (18) to shift the control sequence (˜ 𝒖𝑘, . . . , ˜ 𝒖𝑘+ℓ−1)toward lower-cost actions. The updated sequence also defines the sampling mean for the next iteration. This sampling-based approach bypasses direct nonlinear optimization, enabling iterative refinement of control decisions through statistical inference in complex, uncertain environments. 3.3 MA-sequential rollout augmented information theoretic model predictive control (S-RAIT-MPC) It is important to note that Algorithm 1 represents the integration of IT-MPC with an ℓ-step LA with rollout techniques [c.f. § 2.2]. In a similar manner, IT-MPC can also be combined with sequential MA ℓ-step LA with rollouts [c.f. § 2.4], which is discussed in the sequel. Recall the minimization operation in the sequential MArollout is performed by fixing an order [𝑚]and optimizing through each agent’s control variable 𝒖𝑗according to the order, c.f. § 2.4. Here, the goal is to formulate an agent-centric IT-MPC problem. To this end, a nominal input is considered, 𝒖𝑘, . . . , 𝒖𝑘+ℓ−1of length ℓ, where for each 𝑗∈ [𝑚],𝒖𝑡is partitioned as 𝒖𝑡=(𝒖pre(𝑗) 𝑡,𝒖𝑗 𝑡,𝒖succ(𝑗) 𝑡), 𝑡 =𝑘, . . . , 𝑘 +ℓ−1.(21) From the point of view of agent 𝑗,𝒖pre(𝑗) 𝑡and 𝒖succ(𝑗) 𝑡are known problem parameters and represent the preceding and succeeding agents’ controls, respectively. Only the decision 𝒖𝑗 𝑡is 19 Figure 2. MA-S-RAIT-MPC with 𝑚=3. Each proceeding agents’ computed control action is communicated to the successive agents as marked with orange lines. at the disposal of agent 𝑗and is perturbed by a noise term 𝝐𝑗 𝑡∼N(0,Σ𝑗). As a result, for 𝑡=𝑘, . . . , 𝑘 +ℓ−1, the actual control, denoted 𝒖𝑡(𝒖𝑗 𝑡,𝝐𝑗 𝑡), applied to the system is 𝒖𝑡(𝒖𝑗 𝑡,𝝐𝑗 𝑡)=(𝒖pre(𝑗) 𝑡,𝒖𝑗 𝑡+𝝐𝑗 𝑡,𝒖succ(𝑗) 𝑡)(22) and consequently, the 𝑗th agent-centric IT-MPC problem formulation is given by minimize (𝒖𝑗 𝑘,...,𝒖𝑗 𝑘+ℓ−1) E𝝐𝑗E𝒘Í𝑘+ℓ−1 𝑡=𝑘𝑔𝒙𝑡,𝒖𝑡(𝒖𝑗 𝑡,𝝐𝑗 𝑡),𝒘𝑡+˜ 𝐽𝑗(𝒙𝑘+ℓ).(23) The principle of the sampling based solution [35, Algorithm 1] can be incorporated to yield a solution method to problem (23) by suitably modifying (16)-(19). In particular, instead of V𝑗 𝑘 in (16), each agent 𝑗∈ [𝑚]now maintains an own sequence of 𝑀𝑗sampled control trajectories, denoted by V𝑗 𝑘, and defined as follows: V𝑗 𝑘=(𝒗𝑗 𝑘,𝑖, . . . , 𝒗𝑗 𝑘+ℓ−1,𝑖)𝑖∈[𝑀𝑗],(24) where 𝒗𝑗 𝑡,𝑖 =𝒖𝑡+¯ 𝝐𝑗 𝑡,𝑖 for all 𝑡and 𝑖and (𝒖𝑘, . . . , 𝒖𝑘+ℓ−1)is a given nominal vector. More importantly, ¯ 𝝐𝑗 𝑡,𝑖 is a vector composed of 𝑚blocks, where all blocks except the 𝑗th are identically zero, and the 𝑗th block is given by 𝝐𝑗 𝑡,𝑖, sampled from the distribution N(0,Σ𝑗), i.e., ¯ 𝝐𝑗 𝑡,𝑖 =(0, . . . , 0 | {z } 𝑗−1blocks ,𝝐𝑗 𝑡,𝑖,0, . . . , 0 | {z } 𝑚−𝑗blocks ).(25) Equations(24)and(25)indicate that, incontrasttotheformulation in(16), thenoiseperturbations to the sampled control trajectories are introduced solely by agent 𝑗. Next, in place of (17), an 20 Algorithm 2 MA-S-RAIT-MPC 1: Given: 𝒙0, initial state; 𝑀𝑗, number of sampled trajectories of agent 𝑗;ℓ, number of multistages; 𝑝𝒘, disturbance distribution; 𝑓, transition model; Σ𝑗, noise covariance for agent 𝑗;BPmpc, the base policy for IT-MPC; BProll, the base policy for rollout; 𝑇𝑗, length of the horizon for BProll at agent 𝑗. 2: 𝑘=0 3: while Task is not completed do 4: current state =𝒙𝑘;¯ 𝒖pre(1) 𝑘=∅;𝒖seq 𝑘=∅ 5: for each agent 𝑗∈ [𝑀]do 6: Receive preceding agents’ controls ¯ 𝒖pre(𝑗) 𝑘 7: (𝒖𝑘, . . . , 𝒖𝑘+ℓ−1)=𝐻𝑗(𝑝𝒘,BPmpc, 𝑓 , ℓ;𝒙𝑘,𝒖pre(𝑗) 𝑘=¯ 𝒖pre(𝑗) 𝑘) 8: Compute V𝑗 𝑘from (24) 9: For each trajectory 𝑖∈ [𝑀𝑗], compute 𝒙𝑘+ℓ,𝑖 and the rollout base policy controls for 𝑇𝑗stages, i.e., (𝒖𝑘+ℓ,𝑖, . . . , 𝒖𝑘+ℓ+𝑇−1,𝑖)=𝐻(𝑝𝒘,BProll, 𝑓 , 𝑇 𝑗;𝒙𝑘+ℓ,𝑖) 10: For each trajectory 𝑖∈ [𝑀𝑗]compute rollout base policy cost denoted by 𝑐𝑖, where 𝑐𝑖=Í𝑘+ℓ+𝑇−1 𝑡=𝑘+ℓ𝑔𝒙𝑡,𝑖,𝒖𝑡,𝑖,𝒘𝑡,𝑖(27) 11: For each 𝑖∈ [𝑀𝑗], let ˜ 𝐽𝑗(𝒙𝑘+ℓ,𝑖)=𝑐𝑖+𝑑𝑖where 𝑑𝑖is the terminal cost at 𝒙𝑘+ℓ+𝑇,𝑖 12: Compute 𝑠𝑗 𝑖, ¯ 𝑘from (26) 13: Compute (˜ 𝒖𝑘, . . . , ˜ 𝒖𝑘+ℓ−1)from (18) by substituting 𝑠𝑗 𝑖,𝑡,¯ 𝝐𝑗 𝑡,𝑖 in (19) 14: Perform block extraction to yield (˜ 𝒖𝑗 𝑘, . . . , ˜ 𝒖𝑗 𝑘+ℓ−1) 15: ¯ 𝒖pre(𝑗+1) 𝑘← ( ¯ 𝒖pre(𝑗) 𝑘,˜ 𝒖𝑗 𝑘);𝒖seq 𝑘← (𝒖seq 𝑘,˜ 𝒖𝑗 𝑘) 16: end for 17: return 18: Observe 𝒘𝑘from the environment and pick 𝒖seq 𝑘to move to 𝒙𝑘+1according to (1) 19: 𝑘=𝑘+1 20: end while agent-specific trajectory cost 𝑠𝑗 𝑖, ¯ 𝑘is computed in an analogous manner. Specifically, for each agent 𝑗∈ [𝑚], control trajectory 𝑖∈ [𝑀𝑗]and its time index ¯ 𝑘∈ {𝑘, . . . , 𝑘 +ℓ−1},𝑠𝑗 𝑖, ¯ 𝑘is computed by 𝑠𝑗 𝑖, ¯ 𝑘=Í𝑘+ℓ−1 𝑡=¯ 𝑘𝑔𝒙𝑡,𝑖,𝒗𝑗 𝑡,𝑖,𝒘𝑡,𝑖+˜ 𝐽𝑗(𝒙𝑘+ℓ,𝑖).(26) Consequently, (˜ 𝒖𝑘, . . . , ˜ 𝒖𝑘+ℓ−1)is computed according to (18) by replacing 𝑠𝑖,𝑡 and 𝝐𝑡,𝑖 in (19) by 𝑠𝑗 𝑖,𝑡 and ¯ 𝝐𝑗 𝑡,𝑖, respectively. Finally, the approximate solution of (23), denoted (˜ 𝒖𝑗 𝑘, . . . , ˜ 𝒖𝑗 𝑘+ℓ−1), is computed from (˜ 𝒖𝑘, . . . , ˜ 𝒖𝑘+ℓ−1), where ˜ 𝒖𝑗 𝑡is the 𝑗th block of ˜ 𝒖𝑡. The extraction of 𝑗th block from ˜ 𝒖𝑡is referred to as block extraction and is denoted by the operator Block𝑗(·) where ˜ 𝒖𝑗 𝑡=Block𝑗(˜ 𝒖𝑡). The first control input ˜ 𝒖𝑗 𝑘is scheduled to be applied to the system, while the remaining controls define the mean of the distribution used for sampling in the subsequent time index 𝑘+1. The steps of the preceding discussion is integrated in an iterative manner with an agent order specified by the identity permutation 𝜎id of [𝑚]. As such, steps for MA-S-RAIT-MPC are given in Algorithm 2. The central idea is to solve each agent’s control problem one after another in 21 Figure 3. MA-A-RAIT-MPC with 𝑚=3. Successive agents do not have to wait until proceeding agents communicate their actions. Instead they predict them with a learned model of predecessors. a fixed order, enabling a structured coordination mechanism. At each time step 𝑘, each agent 𝑗computes its own nominal control sequence (𝒖𝑘, . . . , 𝒖𝑘+ℓ−1)by using BPmpc, see step 7 of Algorithm 2. When doing so, it is assumed that the exact control of the preceding agents 𝒖pre(𝑗) 𝑡 for 𝑡=𝑘is available at agent 𝑗and 𝒖𝑘is updated accordingly. The corresponding computation is compactly represented by the operator 𝐻𝑗. For each agent 𝑗∈ [𝑚], the control sequence is partitioned as per (21) into three parts: the controls of preceding agents, the current agent’s control, and the controls of succeeding agents. Only the current agent’s control is perturbed with Gaussian noise as per (22), while the other parts remain fixed. This results in a set of perturbed trajectories for that agent. For each trajectory, the values of states that come after the planning horizon (after time 𝑘+ℓ) is evaluated using a separate rollout procedure of length 𝑇𝑗 governed under BProll. The cumulative cost of the rollout computed using (26), plus any terminal cost forms the surrogate cost-to-go, which is used to compute a cost-weighted update of the agent’s control using importance sampling similar to explanation in Algorithm 1. After each agent optimizes the joint control sequence, its own control trajectory is extracted and the very fist component of the trajectory is passed along to the next agent in the sequence and also stored in 𝒖seq 𝑘, generated by the joint action merger (JAM), to be applied to the system to transition to the next state 𝒙𝑘+1,c.f. Fig. 2. 3.4 MA-autonomous rollout augmented information theoretic model predictive control (A-RAIT-MPC) The autonomous variant eliminates the need for sequential agent coordination by replacing the explicit dependence on preceding agents’ controls with learned approximations. Specifically, instead of receiving ¯ 𝒖pre(𝑗) 𝑘from agents 𝑖∈ [ 𝑗−1]during sequential updates (as required in (21)–(23)), each agent 𝑗∈ [𝑚]uses a NN to approximate it with ˆ 𝒖pre(𝑗) 𝑘, as introduced in § 2.5. Thus, the only difference being that preceding agents’ controls are estimated rather than com- 22 Figure 4. Training the 𝑗𝑡ℎ agent to approximate ¯ 𝒖pre(𝑗) 𝑘using a neural network equipped with a reward comparator to trigger adaptation under changing disturbance distribution. municated, c.f. Fig. 3. This shift from sequential to autonomous execution removes inter-agent communication bottlenecks and enables fully decentralized and parallel action computation effectively applying 𝒖aut 𝑘on the environment similar to the sequential counterpart 𝒖seq 𝑘, significantly improving scalability and runtime efficiency in MASs. 3.5 Learning approximate agent controls In the MA-A-RAIT-MPC framework, each agent 𝑗∈ [𝑚]requires an estimate of the control inputs of preceding agents 𝑖∈ [ 𝑗−1]to solve its agent-centric control problem independently. To this end, a NN function 𝜙𝑗:X → Uprec(𝑗), parameterized by 𝜽𝑗, is trained to approximate the mapping ˆ 𝒖prec(𝑗) 𝑘=𝜙𝑗(𝒙𝑘;¯ 𝜽𝑗),(28) where ¯ 𝜽𝑗is the best parameter vector obtained by optimizing some cost function over 𝜽𝑗. The cost function for training the NN model is based on a dataset D𝑗of size 𝐷, consisting of state-control input pairs. In particular, we have D𝑗=n𝒙(𝑑) 𝑘,¯ 𝒖prec(𝑗) 𝑘 (𝑑)o𝐷 𝑑=1,(29) and is generated from past sequential MA rollout executions. The dataset may also be stored in the edge cloud so that it can be downloaded by agents whenever needed for training their NNs. It is worth noting that the dataset is collected under the distribution 𝑝𝒘of the disturbance. A commonly used cost function is the mean-squared error, where the parameter vector ¯ 𝜽𝑗is given 23 by ¯ 𝜽𝑗∈arg min 𝜽𝑗 1 𝐷 𝐷 ∑︁ 𝑑=1𝜙𝑗(𝒙(𝑑) 𝑘;𝜃𝑗) − ¯ 𝒖prec(𝑗) 𝑘 (𝑑) 2 .(30) This trained function 𝜙𝑗replaces real-time communication in (23), allowing each agent to independently approximate the influence of others during decision-making c.f. Fig.4. Nonetheless, the previously collected dataset D𝑗becomes outdated when the distribution 𝑝𝒘evolves rendering the model of the other agent outdated. In this respect, to yield an adaptive autonomy, first one needs to identify any changes of the distributions of the disturbances. To identify such changes, a reward comparator is deployed (c.f. Fig.4) to monitor the system’s performance using a moving average of collected rewards. Let 𝑅𝑘be the reward at time 𝑘. The moving average over a window of size 𝑊rew is computed as, ¯ 𝑅𝑘=1 𝑊𝑟𝑒𝑤 𝑘 ∑︁ 𝑡=𝑘−𝑊𝑟𝑒𝑤+1 𝑅𝑡.(31) When 𝑅𝑘< 𝛾 ¯ 𝑅𝑘for a fixed sensitivity threshold 𝛾∈ (0,1), we say that a performance degradation is detected and the current neural network 𝜙𝑗is flagged as misaligned with the current environment (assuming there are no other factors contributing to this). This entails collection of new data D𝑗under the new disturbance distribution so that 𝜙𝑗is updated by retraining, to yield a new parameter vector according to (30). 3.6 Agent ordering in sequential MA rollout The performance of sequential MA rollout depends heavily on the order in which agents are optimized. While a fixed ordering [𝑚]≜{1, . . . , 𝑚}is commonly assumed, different strategies for determining the agent order can significantly affect the quality and efficiency of control. Three mechanisms for agent ordering are investigated in this work, each of which generates an ordering 𝜎that is passed into the sequential rollout planner (Algorithm 2): 1. State-Dependent Exhaustive Ordering (Algorithm 3): At each timestep 𝑘, all 𝑚! permutations of agent orderings are evaluated by invoking Algorithm 2 for each, and the ordering that yields the lowest rollout cost is selected as the best ordering best_order𝑘. This provides optimal ordering per state but is computationally expensive and typically requires a centralized controller. 2. Fixed Ordering from Initial State (Algorithm 4): A one-time exhaustive search is performed at the initial state 𝑥0to determine the best ordering best_order0, which is then reused for all subsequent time steps. Our argument here is that the initial distribution of the state has an impact on the agent ordering. This amortizes the search cost over time, but assumes that the best initial ordering remains valid across state evolution. Like the previous method, this may also assumes centralized coordination. 3. Heuristic Q-Factor-Based Ordering (Algorithm 5): At each state 𝑥𝑘, each agent independently computes a local heuristic score ℎ(𝑥𝑘, 𝑗)(e.g., based on expected reward). The agents are then ordered in ascending order of their scores to form 𝜎𝑘. This strategy is decentralized such that agents compute scores locally, and a one-time consensus protocol can be used to agree on the final ordering across the team. 30 Figure 6. Mean reward comparison with increasing length of the horizon 𝑇of BProll with ℓ=1 Table 3. Worst case MA-A-RAIT-MPC performance degradation and recovery comparison. Method Mean Reward Drop (%) Recovery Steps (×100) MADDPG 57.14% 810 MA-A-RAIT-MPC (𝑇=0) 22.22% 420 of this degradation depends on the configuration of the 𝑇. Recall that the proposed method integrates three components: multi-step LA (ℓ), truncated rollout of length 𝑇and a terminal cost approximation 𝑑. When 𝑇=0, the terminal cost relies entirely on a heuristic approximation, i.e., the average distance between each predator and the remaining prey. In this setting, model adaptation becomes essential for performance recovery, and retraining the NNs eventually restores effectiveness. With a short rollout horizon (𝑇=5), some tasks are already completed during the rollout, reducing the average reliance on the terminal cost estimate. Consequently, the performance drop is smaller, and recovery via model adaptation is faster. In contrast, for a longer rollout horizon (𝑇=30), most tasks are completed within the truncated rollout itself, rendering the terminal cost estimate nearly irrelevant. Notably, in this case, performance remains robust without any model updates, illustrating the inherent resilience of the system to environmental changes when deep planning is employed with the trade-off of higher computational load as presented in Fig. 7. Even though MADDPG showed far superior performance as shown in Fig. 6 and 7, it shows far worse percentage drop of performance when met with changing disturbances of the system. As presented in Table 3 it takes 92% longer to get back to normality compared to that of the worst case performance of MA-A-RAIT-MPC. It is also noteworthy that for MADDPG even to reach the normality performance of MA-A-RAIT-MPC it takes approximately 23% longer. This reiterates the simplicity and the resiliency of the proposed MA-A-RAIT-MPC as an online algorithm. 31 Figure 7. Mean run time comparison with increasing length of the horizon 𝑇of BProll with ℓ=1 4.4 Evaluating optimization of agents order This section presents an evaluation of agent ordering strategies, namely the default ordering [𝑚], the optimized ordering strategy described in Algorithm 4, and the heuristic-based ordering strategy from Algorithm 5 in the context of the discrete-action counterpart of MA-S-RAIT-MPC (Algorithm 6). The goal is to assess how different ordering strategies affect task performance in terms of mean reward and task completion time, which includes inter-agent communication latency. Fig. 9a plots the collected mean reward for each agent ordering mechanism as the number of agents 𝑚increases, while holding the task difficulty constant (i.e., number of targets fixed at 𝑛=2). Corresponding task completion times are shown in Fig. 9b on a logarithmic scale. The results demonstrate that the optimized ordering consistently leads to higher rewards compared to the default ordering, with the performance gap widening as the number of agents increases. This supports the hypothesis that initial agent-target placement assignments significantly influence overall coordination efficiency in the MAS evaluated in this work. Although the optimized ordering (Algorithm 4) achieves the best reward outcomes, it incurs the highest runtime due to the factorial search space of agent permutations (𝑚!), even when evaluated in parallel. In contrast, the heuristic-based ordering (Algorithm 5) achieves nearly comparable rewards with substantially less computational cost. Interestingly, its runtime decreases with increasing 𝑚, as the larger agent pool facilitates quicker task resolution for a fixed number of targets. Conversely, the default ordering not only underperforms in reward collection but also exhibits increasing task completion time as the number of agents grows. This trend further emphasizes the benefits of incorporating intelligent ordering strategies, even heuristic ones, to enhance coordination efficiency in MASs. 32 Figure 8. Effect of truncated rollout length 𝑇on adaptation performance under changing disturbance distribution. Longer rollouts (higher 𝑇) reduce reliance on learned models and improve resilience to distributional shifts in 𝑝𝒘, while shorter rollouts (lower 𝑇) require model adaptation to recover performance. 4.5 Performance under varying number of MC simulations This section evaluates the effect of varying the number of MC simulations (𝑁𝑚𝑐𝑡𝑠) and the number of agents (𝑚=2,3,4,5) in the system on overall performance and computational efficiency, while maintaining a fixed task difficulty (i.e., number of targets 𝑛=2). Specifically, the system’s performance is assessed in terms of the mean reward collected and the time required to complete the task. Comparisons are made between a uniform allocation of MC simulations across agents and configurations where the total number of simulations varies with the number of agents. Fig. 10 can be analyzed by dividing it to low MC budget regime (LMCBR) where 𝑁𝑚𝑐𝑡𝑠 <20 and high MC budget regime (HMCBR) where 𝑁𝑚𝑐𝑡𝑠 >20. In the LMCBR, reward collected by the MAS is sensitive to increasing 𝑁𝑚𝑐𝑡𝑠 compared to that of HMCBR. Additionally, adding more agents enhances performance in LMCBR, but this benefit diminishes as the system configuration moves toward HMCBR where 𝑚does not have much effect on the rewards collected. Importantly, in the HMCBR, runtime begins to increase steeply. This sharp rise in computational cost highlights a critical trade-off between performance gains and real-time feasibility. The identification of such a threshold (e.g. 𝑁𝑚𝑐𝑡𝑠 =20) is particularly valuable for cooperative MAS design, where balancing task efficiency with runtime constraints is essential. A key observation from Fig. 11 is the strong inverse relationship between 𝑁𝑚𝑐𝑡𝑠 and variance of reward collected. As 𝑁𝑚𝑐𝑡𝑠 increases, the variance in reward across experiment runs drops significantly for all 𝑚configurations, indicating increased stability. However, this reduction in variance is more pronounced in smaller teams. For larger numbers of agents, the rate at which variance decreases with additional simulations is lower. These findings establish a clear baseline for how simulation budget and 𝑁𝑚𝑐𝑡𝑠 affects performance, stability, and computational cost, providing context for the adaptive allocation strategies explored in the next section. 33 (a) (b) Figure 9. Performance variation observed with reordering agents in Algorithm 6 a) Mean reward and b) Run time. Figure 10. Reward and run time comparison with 𝑁𝑚𝑐𝑡𝑠 for team configurations 4.6 Prioritized sampling: adaptive number of MC simulations allocation strategy Building on the findings from the previous section, which evaluated the system’s performance under varying but uniform MC simulation budgets across all agents, now a more dynamic strategy is considered where allocating a different number of simulations to each agent based on its individual decision context. The motivation stems from the observation that while increasing the total number of MC simulations generally improves performance, uniform allocation may not be the most efficient use of computational resources. In many MA scenarios, agents are not equally critical in every state. Some may have more influence over the system’s near-term trajectory than others. Exploiting this asymmetry, an adaptive MC allocation mechanism is proposed where the number of simulations per agent is modulated according to a state-dependent quality metric. 34 Figure 11. Variance of rewards collected vs. 𝑁𝑚𝑐𝑡𝑠 for team configurations Figure 12. Performance comparison with different adaptation schemes for 𝑚=4and 𝑛=2 To this end, a heuristic Q-factor (𝑄𝑖) for each agent is computed at each time step 𝑘as, 𝑄𝑖=1 |¯ T | ∑︁ 𝑗∈¯ T𝒂𝑗 𝑘−𝒃𝑗 𝑘1,(40) where ¯ T ⊆ {1, . . . , 𝑛}denotes the set of uncaptured targets at time step 𝑘capturing the expected value or contribution of that agent’s decision in its current state. Using this Q-factor, per-agentsimulation-numbers (𝑁𝑖 𝑚𝑐𝑡𝑠) are allocated via several functions that adaptively decide the value for 𝑁𝑖 𝑚𝑐𝑡𝑠 based on 𝑄𝑖within a given budget. It is found that, as shown in Fig. 12, monotonically increasing functions with respect to the Q-factor (e.g., 𝑥,𝑥2,𝑒𝑥) consistently yield better task performance and lower run times compared to both uniform allocation and decreasing functions (e.g., 𝑥−1,−𝑥), which tend to under-allocate resources to high-impact decisions. Among the strategies tested, exponential growth functions strike the best trade-off, due to the enabling of deeper planning by allocating higher 𝑁𝑖 𝑚𝑐𝑡𝑠 35 (a) (b) Figure 13. N-Step LA comparison of reward and time with 𝑁𝑚𝑐𝑡𝑠 for a MAS with 𝑚=4and 𝑛=2a) Mean reward and b) Run time. for agents producing high Q-factor, in other words where it matters most without unnecessary overhead elsewhere. These results demonstrate that adaptive allocation offers a principled and effective mechanism for prioritizing computational effort in rollout-based MA planning, especially when operating under resource constraints or in scenarios with heterogeneous agent influence. 4.7 Evaluating the balance between LA depth, 𝑁𝑚𝑐𝑡𝑠 and run time Analysis on the effect of increasing ℓin Algorithm 6 on collected reward and task execution run time is presented in this section. Upon initially hypothesizing that for a fixed number of agents and a constant task difficulty (fixed 𝑛), enforcing deeper LA by incorporating higher ℓvalues improves performance by enabling more informed decisions, albeit at the cost of increased computational overhead, the following observation has been made. Experimental results shown in Fig. 13 validate this hypothesis. As the LA depth increases, cumulative reward improves, indicating more effective planning. However, when pairing higher number of 𝑁𝑚𝑐𝑡𝑠 with higher ℓ, the marginal gains diminish due to immediate next ℓsteps are chosen in a cost effective manner. These findings highlight a fundamental trade-off between planning depth and computational feasibility. When run time or resource constraints are a concern, allocating budget to increase 𝑁𝑚𝑐𝑡𝑠, rather than expanding the LA depth may yield better overall task efficiency. This insight is especially relevant for online MAS operating in dynamic environments, where timely decisions are critical. 36 5 SUMMARY AND FUTURE DIRECTIONS This thesis presented a novel framework that integrates MA rollout algorithms with IT-MPC to address complex multistage decision problems. The proposed approach supports both sequential and autonomous coordination strategies, enabling scalable and efficient decentralized decision-making across agents operating in continuous state and action spaces. Systematic evaluations were presented on the impact of various algorithmic design choices, including agent ordering, simulation budgets, rollout depth, and adaptive simulation allocation. Presented results demonstrated significant gains in task efficiency, robustness to dynamic environments, and reduced computational overhead compared to baseline and SOTA RL methods. Notably, adaptive mechanisms such as prioritized simulation budgeting and heuristic-based ordering that provide practical pathways for real-time planning in resource-constrained scenarios. In real-world applications, continuous or infinite control and state spaces are prevalent, prompting the key question, can rollout algorithms be practically deployed for such problems?. This thesis addresses it by integrating rollout planning with the IT-MPC framework. Each agent independently performs online LA simulations, enabling local rollout-based policy improvements resembling distributed RL. This RAIT-MPC framework combines decentralized planning with MPC, allowing agents to anticipate future interactions while maintaining coordinated group behavior. For the development of A-RAIT-MPC more sophisticated methods of modeling of other agents could be explored as potential future avenues. Traditional computation offloading and mobility management strategies fail under connectivity loss or contention spikes. In the RAIT-MPC presented in this thesis, each agent conducts MC simulations over its control horizon, using lightweight base policies (for continuous actions) or MCTS (for discrete actions) to estimate rollout costs. Agents anticipate downstream consequences by simulating interactions under assumed behavior models, reducing the risk of cascading failures. Further work can be extended using partial observations by each agent and exchange concise statistics among the team rather than full trajectories, blending autonomy with occasional global coordination. Communication optimization in MAS has been an active area of research. Early efforts focused on predefined task-specific protocols, such as directing swarm robots by transmitting nearest-target information [40]. These methods handled challenges of non-homogeneity and partial observability. When tasks are dynamic or partially known, protocols communicating meta-level task information (e.g., actuator activations in vehicles [41]) proved more robust. The proposed framework aligns with theoretical developments in [36] and ensures that edge devices such as drones, sensors, or vehicles can make near-optimal decisions independently, even without continuous cloud connectivity. Beyond application value, the framework is motivated by two critical issues: (i) intermittent connectivity forcing high energy expenditures for communication, and (ii) systemic risks from uncoordinated failures in large-scale IoT deployments. Historical engineering failures highlight the importance of proactively addressing adaptive behaviors in interconnected systems. Vaughan [42] links the Challenger disaster to the normalization of deviance. Other cases such as Boeing 737 MAX crashes [43], software race conditions in radiation therapy machines [44], decentralized failures in the Northeast blackout [45], and algorithmic stock market losses [46] underline the need for systems that can adapt to unseen disturbances. When the distribution of the disturbance changes, agents can leverage past knowledge and incorporate continuous-learning [47, 48] strategies rather than retraining from the beginning that could further improve the resiliency of the MAS. Recent approaches shift towards agents learning their own communication protocols. Kasai et al.[49] show agents developing their own codewords, trading off communication efficiency and learning cost. Other works demonstrated emergent languages in cooperative guessing games 37 using differentiable inter-agent learning[50, 51, 52], relying on centralized training with discrete messaging during decentralized execution. Further research work can be continued following the work produced in this thesis on how the communication can be scaled effectively. There may be cases when some agents may not have an impact on the others which raises the question: is it worth maintaining a communication channel with that agent?. Beyond communication, adaptation and system resilience have been key research focuses. Although intelligence and adaptation are often used interchangeably in AI literature, adaptation fundamentally involves optimizing actions in response to environmental changes [53]. Decades of research have explored adaptive agent collaboration, including Stanford’s computer-animated theater demonstrating real-time emotional adaptation [54]. Modern large language models (LLMs) exhibit similar improvisational capabilities [55]. Adaptation is crucial both for internal adjustments such as managing vehicle states during dynamic highway scenarios [56] and for updating models of other agents. [57] shows that in predator-prey simulations, agents benefit from adaptive modeling of others via case based learning (CBL), although CBL’s static library limits future-proofing. This presented work has shown promising avenues in this domain where similar results seen in TD-Backgammon producing better results with truncated rollout compared to no rollout, the MAS driven by IT-MPC augmented rollout showed higher resiliency for system disturbances with truncated rollout. This is an exciting observation worth exploring further with partial observability and with competing agents driven by self-play. Extending this further to understand the features of the disturbance distributions that create such an inherent resiliency is an interesting research avenue to explore. 38 6 REFERENCES [1] R. S. Sutton, A. G. Barto, et al.,Reinforcement learning: An introduction. MIT press Cambridge, 2nd ed., 2018. [2] D. Bertsekas and S. E. Shreve, Stochastic optimal control: the discrete-time case, vol. 5. Athena Scientific, 1996. [3] D. Bertsekas, Rollout, policy iteration, and distributed reinforcement learning. Athena Scientific, 2021. [4] D. Bertsekas, “Multiagent reinforcement learning: Rollout and policy iteration,” IEEE/CAA Journal of Automatica Sinica, vol. 8, no. 2, pp. 249–272, 2021. [5] D. Bertsekas, A course in reinforcement learning. Athena Scientific, 2nd ed., 2024. [6] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al., “Mastering chess and shogi by self-play with a general reinforcement learning algorithm,” arXiv preprint arXiv:1712.01815, 2017. [7] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al., “Mastering the game of go without human knowledge,” nature, vol. 550, no. 7676, pp. 354–359, 2017. [8] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al., “Mastering the game of go with deep neural networks and tree search,” nature, vol. 529, no. 7587, pp. 484–489, 2016. [9] X. Yan, P. Diaconis, P. Rusmevichientong, and B. Roy, “Solitaire: Man versus machine,” Advances in Neural Information Processing Systems, vol. 17, 2004. [10] G. Tesauro, D. C. Gondek, J. Lenchner, J. Fan, and J. M. Prager, “Analysis of watson’s strategies for playing jeopardy!,” Journal of Artificial Intelligence Research, vol. 47, pp. 205–251, 2013. [11] C. Meloni, D. Pacciarelli, and M. Pranzo, “A rollout metaheuristic for job shop scheduling problems,” Annals of Operations Research, vol. 131, pp. 215–235, 2004. [12] N. Secomandi, “A rollout policy for the vehicle routing problem with stochastic demands,” Operations Research, vol. 49, no. 5, pp. 796–802, 2001. [13] Q. Huang, Q.-S. Jia, and X. Guan, “Robust scheduling of ev charging load with uncertain wind power integration,” IEEE Transactions on Smart Grid, vol. 9, no. 2, pp. 1043–1054, 2016. [14] M. C. Ferris and M. M. Voelker, “Fractionation in radiation treatment planning,” Mathematical programming, vol. 101, pp. 387–413, 2004. [15] M. C. Ferris, R. R. Meyer, and W. D’Souza, “Radiation treatment planning: Mixed integer programming formulations and approaches,” in Handbook on modelling for discrete optimization, pp. 317–340, Springer, 2006. [16] D. Bertsimas and I. Popescu, “Revenue management in a dynamic network environment,” Transportation science, vol. 37, no. 3, pp. 257–277, 2003. [17] S. Nozhati, Y. Sarkale, B. Ellingwood, E. K. Chong, and H. Mahmoud, “Near-optimal planning using approximate dynamic programming to enhance post-hazard community resilience management,” Reliability Engineering & System Safety, vol. 181, pp. 116–126, 2019. [18] D. Garces, S. Bhattacharya, D. Bertsekas, and S. Gil, “Approximate multiagent reinforcement learning for on-demand urban mobility problem on a large map (extended version),” arXiv preprint arXiv:2311.01534, 2023. 39 [19] D. Garces, S. Bhattacharya, S. Gil, and D. Bertsekas, “Multiagent reinforcement learning for autonomous routing and pickup problem with adaptation to variable demand,” in 2023 IEEE International Conference on Robotics and Automation (ICRA),pp. 3524–3531, IEEE, 2023. [20] S. Bhattacharya, S. Badyal, T. Wheeler, S. Gil, and D. Bertsekas, “Reinforcement learning for pomdp: Partitioned rollout and policy iteration with application to autonomous sequential repair problems,” IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 3967–3974, 2020. [21] S. Bhattacharya, S. Kailas, S. Badyal, S. Gil, and D. Bertsekas, “Multiagent reinforcement learning: Rollout and policy iteration for pomdp with application to multirobot problems,” IEEE Transactions on Robotics, vol. 40, pp. 2003–2023, 2023. [22] G.-Z. Yang, J. Bellingham, P. E. Dupont, P. Fischer, L. Floridi, R. Full, N. Jacobstein, V. Kumar, M. McNutt, R. Merrifield, et al., “The grand challenges of science robotics,” Science robotics, vol. 3, no. 14, p. eaar7650, 2018. [23] M. Kouzehgar, M. Meghjani, and R. Bouffanais, “Multi-agent reinforcement learning for dynamic ocean monitoring by a swarm of buoys,” in Global Oceans 2020: Singapore–US Gulf Coast, pp. 1–8, IEEE, 2020. [24] S. Nikookar, “Human-ai complex task planning,” in 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 3923–3927, IEEE, 2023. [25] A. Oroojlooy and D. Hajinezhad, “A review of cooperative multi-agent deep reinforcement learning,” Applied Intelligence, vol. 53, no. 11, pp. 13677–13722, 2023. [26] P. Dai, H. Liu, W. Yu, and H. Wang, “Distributed neural learning algorithms for multiagent reinforcement learning,” IEEE Internet of Things Journal, vol. 10, no. 23, pp. 21039– 21060, 2023. [27] H.-H. Chang, H. Song, Y. Yi, J. Zhang, H. He, and L. Liu, “Distributive dynamic spectrum access through deep reinforcement learning: A reservoir computing-based approach,” IEEE Internet of Things Journal, vol. 6, no. 2, pp. 1938–1948, 2018. [28] S. Wang, H. Liu, P. H. Gomes, and B. Krishnamachari, “Deep reinforcement learning for dynamic multichannel access in wireless networks,” IEEE transactions on cognitive communications and networking, vol. 4, no. 2, pp. 257–265, 2018. [29] X. Tan, L. Zhou, H. Wang, Y. Sun, H. Zhao, B.-C. Seet, J. Wei, and V. C. Leung, “Cooperative multi-agent reinforcement-learning-based distributed dynamic spectrum access in cognitive radio networks,” IEEE Internet of Things Journal, vol. 9, no. 19, pp. 19477– 19488, 2022. [30] J. Ribeiro, F. S. Melo, and J. Dias, “Multi-task learning and catastrophic forgetting in continual reinforcement learning,” arXiv preprint arXiv:1909.10008, 2019. [31] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actorcritic for mixed cooperative-competitive environments,” Advances in neural information processing systems, vol. 30, 2017. [32] P. Dai, W. Yu, H. Wang, and S. Baldi, “Distributed actor–critic algorithms for multiagent reinforcement learning over directed graphs,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 10, pp. 7210–7221, 2022. [33] J. W. Weber, D. R. Giriyan, D. R. Parkar, D. P. Bertsekas, and A. W. Richa, “Distributed online rollout for multivehicle routing in unmapped environments,” arXiv preprint arXiv:2305.15596, 2023. [34] G. Williams, A. Aldrich, and E. Theodorou, “Model predictive path integral control using covariance variable importance sampling,” arXiv preprint arXiv:1509.01149, 2015.