Contributions to distributed MPC: coalitional and learning approaches

Universidad de Sevilla Escuela T´ ecnica Superior de Ingenier´ ıa Tesis Doctoral Contributions to distributed MPC: coalitional and learning approaches Autora: Paula Chanfreut Palacio Directores: Jos´ e Mar´ ıa Maestre Torreblanca Eduardo Fern´ andez Camacho Septiembre 2022 Tesis Doctoral: Contributions to distributed MPC: coalitional and learning approaches Autora: Paula Chanfreut Palacio Directores: Jos´ e Mar´ ıa Maestre Torreblanca Eduardo Fern´ andez Camacho El tribunal nombrado para juzgar la Tesis arriba indicada, compuesto por los siguientes doctores: Presidente: Vocales: Secretario: acuerdan otorgarle la calificaci´ on de: El Secretario del Tribunal Fecha: This thesis has received funding from the Ministry of Science, Innovation and Universities of Spain under the Training Program for Academic Staff (FPU, by its Spanish acronym) [grant FPU17/02653], the European Research Council (ERC) under the European Union’s Horizon 2020 Research and Innovation Program [OCONTSOLAR, grant agreement No. 789051], the Spanish Ministry of Economy under Projects C3PO [DPI2017-86918-R] and C3PO-R2D2 [PID2020- 119476RB-I00], and the Manuel Gay´ an Buiza Award. Esta tesis ha sido financiada por el el Ministerio de Ciencia, Innovaci ´ on y Universidades en el marco del programa para la Formaci ´ on de Profesorado Universitario (FPU) [FPU17/02653], por el European Research Council (ERC) en el marco del programa de investigaci ´ on e innovaci ´ on Horizonte 2020 de la Uni ´ on Europea [OCONTSOLAR, ref. 789051], por el Ministerio de Econom ´ ıa en el marco de los proyectos C3PO [DPI2017-86918-R] y C3PO-R2D2 [PID2020-119476RB-I00], y por el Premio Manuel Gay´ an Buiza. Agradecimientos En primer lugar, quiero agradecer a mis supervisores, los profesores Jos ´ e Mar ´ ıa Maestre y Eduardo F. Camacho, por la confianza y apoyo que siempre han mostrado hacia m ´ ı. Muchas gracias por toda vuestra ayuda y dedicaci ´ on, y por haberme ofrecido la oportunidad de trabajar estos a ˜ nos junto a vosotros. Me gustar ´ ıa tambi ´ en agradecer al profesor Francesco Borrelli, por la acogida y ayuda durante mi estancia en la Universidad de Berkeley. Igualmente, me gustar ´ ıa dar las gracias a mis padres y hermano, por el apoyo y el cari ˜ no incondicional que siempre me han dado. No habr ´ ıa llegado hasta aqu ´ ı sin vosotros. Quiero agradecer tambi ´ en a mis amigas de ´ Ecija, por acompa ˜ narme desde siempre, y a los amigos que me ha regalado la Escuela, por todos los buenos momentos vividos, y los que nos quedan. Por ´ ultimo, quiero mencionar a Alfonso, que ha seguido conmigo la evoluci ´ on de este trabajo desde el inicio. Agradecerle la paciencia y el cari˜ no diarios. Paula Chanfreut Palacio Sevilla, 2022 I Contents 1 Introduction 1 1.1 Coalitional Control 3 1.1.1 Underlying objective 5 1.1.2 Previous works 5 1.1.3 Clustering problem 7 1.2 Data-based methods in MPC 8 1.3 List of publications 10 1.4 Outline of the rest of this thesis 12 2 Objectives 15 3 Summary of results and discussion 19 3.1 Survey on clustering methods for control systems 20 3.2 Linear Feedback Coalitional Controllers 21 3.2.1 Switching regions 21 3.2.2 Combination with an observer 23 3.3 Supervised coalitional MPC schemes 24 3.3.1 Coalitional MPC in traffic freeways 24 3.3.2 Coalitional MPC for tracking 26 3.4 Methods for the coalitions selection 28 3.4.1 Binary quadratic program 28 3.4.2 Multi-armed bandit problem 30 3.5 Data-based methods in non-centralized MPC 32 3.5.1 Dual decomposition-based DMPC with NNs 32 3.5.2 Multi-agent LMPC 34 3.6 Applications of MPC in cyber-physical systems 36 4 Publications 39 III IV Contents 4.1 Journal articles 39 4.2 Conference articles 40 5 Conclusions and future research directions 43 List of Figures 47 Bibliography 49 Glossary 63 1 Introduction In the last decades, the advances in information and communication technologies, together with the sheer size and complexity of control applications, have boosted the development of non-centralized control methods [1 – 6]. In contrast to the classic centralized approach, where a single controller has full decision-making power and complete knowledge of the system, non-centralized strategies rely on a decentralization of the overall control problem. Generally, the concentration of information and actuation capacity in a global control unit allows for a perfect coordination of all the system’s inputs, thus leading to optimal overall performance. However, in many large-scale systems (LSS), the centralized approach reaches a point of diminishing returns due to the increasing computation and communication demands, which can make it result impractical and/or inefficient for real-time control. In addition, the lack of redundancy and the difficulty to adapt the controller to the structure of the system and its changes are further drawbacks that foster the study and development of non-centralized architectures. Moreover, as stated in [7], while LSS may involve a great number of variables, they can also be modeled as an interconnection of geographically-dispersed subsystems with coupled dynamics, which make them a natural field of application for distributed approaches. Also, the LSS framework includes systems of special relevance within the engineering community such as smart grids [3,4,8], water networks [9], traffic systems [6], large manufacturing processes [10], and intelligent buildings [11]. The underlying idea of non-centralized strategies is to divide the global control problem into a set of smaller subproblems that are managed by different control units, the so-called agents [12,13]. This approach has been reported to provide increased scalability, flexibility, and redundancy of the control architecture [14 – 16], at the expense of a more sophisticated design. As it is illustrated in Fig. 1.1, each agent can access the local output variables and manipulates the inputs of its corresponding subsystem. The extent to which the agents exchange data and coordinate their local actions gives rise to various non-centralized control schemes. As described in [1] within the well-known framework of model predictive control (MPC) [17,18], we can distinguish between distributed approaches, where the set of agents uses a communication network to share local data [19,20]; and decentralized approaches, when there is no data exchange between local controllers [21,22]. 1 8Chapter 1. Introduction A common approach to deal with this issue is to separate the decisions on the communication topology from the optimization of the control inputs. For example, in [77,80], the topology can only be periodically switched, whereas the system inputs are computed at each time step according to the chosen coordination structures. Nevertheless, at the switching instants, a method able to optimize the coalitions is still needed. In [28], by using an LMI procedure, the authors compute offline a state-dependent function that provides an upper bound on the cost-to-go for each possible partition. Considering this, the coalitions are changed online through a comparative assessment that, besides weighting the latter, considers the associated communication costs. A similar procedure is used in [75,77,91,92], and [80] considers heuristic criteria based on the application at issue. Furthermore, recent advances in this regard are given in [78,79,85], where, as mentioned above, the clustering problem is modeled as a game. Nevertheless, the difficulty of this problem increases with the size of the system, since the number of possible partitions experiences a combinatorial explosion with the number of links. For example, for a network with 5 links, there are 32 different network topologies, while for a network with 20 links, there are 1048576 alternatives. For these reasons, new results in this line are stepping stones toward an increased practical impact of coalitional control. 1.2 Data-based methods in MPC The increased capability of sensing, computing, and storing data, along with the powerful advances in machine learning techniques, have promoted the application of data-driven methods within the field of MPC [93 – 96]. These methods are reported to be successful solutions to aid the design of MPC controllers, which in turn allows preventing performance losses. In particular, the review in [93] distinguishes three main directions for the integration of learning-based techniques in the MPC context. First, they have been widely employed to identify accurate prediction models and adjust them to the possible changes in time [97 – 100]. The latter intends to minimize the model mismatch and increases the controller adaptability to changes in the system dynamics, thus allowing for more reliable predictions. The second research direction focuses on the rest of components that, similarly to the prediction model, influence the MPC problem, such as the terminal set, terminal cost, and the constraints. These components can also be designed from data to attain a desired goal, e.g., to counteract the inherent finite-horizon nature of MPC controllers [101], and to robustify the system against uncertainties [102]. Finally, the third research direction focuses on the use of MPC to guarantee constraint satisfaction [103 – 105]. In this context, an MPC- based filter receives an input obtained by a learning-based method, e.g., reinceforment learning [106,107], and checks if its implementation would keep the system within the constraints bounds, or if it has to be modified otherwise. As stated in [105], this line of research is motivated by the fact that many of the available learning techniques, such as reinforcement learning, may provide non-admissible inputs during the learning process, which limits their application in safety-critical systems. This thesis makes use of the learning model predictive control (LMPC) formulation presented in [101], which is framed within the second research direction reviewed in [93]. The 1.2 Data-based methods in MPC 9 LMPC strategy focuses on systems with a strong repetitive behavior, such as autonomous racing vehicles [108], and is based on the systematic design of the terminal components of the MPC problem. In particular, the data collected each time the system repeats the same task allows the controller to recursively construct a safe set, which has control invariant properties, and to accurately define the cost-to-go in the MPC objective function. Under the assumption that the model is perfectly known, the LMPC approach [101] is proved to converge to the optimal solution of the associated infinite-horizon control problem, i.e., it attains optimal performance for the given task. Recent efforts have been made to extend the results in [101] to distributed and decentralized systems [109,110] and to a more flexible paradigm where the repeated tasks are more varied [111,112]. For example, [112] presents a task decomposition method where they are defined as aggregations of subtasks in different orders. In particular, this work leverages data to define the safe set and to lower the costs when executing tasks composed of the same sets of subtasks. Additionally, [111] considers the case in which the dynamics, constraints, or stage cost are periodically time-varying. Finally, see also [110], where a decentralized LMPC for nonlinear multi-agent systems with coupled state constraints is introduced. In this case, the agents recursively improve their performance by synthesizing their terminal components using only locally collected data. Furthermore, the opportunities offered by learning techniques within the MPC framework go beyond the three directions of research mentioned above. For example, [113 – 115] directly approximate the MPC control law by using neural networks (NN) [116], which allows a fast computation of the inputs without resorting to the explicit MPC approach [117]. As stated in [115], the range of application of explicit MPC is limited to low-order systems and small prediction horizons due to the inherent demand of memory and the complexity of the point location problem. Conversely, NN-based models allow a direct map of systems’ variables into control actions, while also offering a potential ability to accommodate nonlinear dynamics as in [113]. Moreover, recent research results on NN-based controllers also attain theoretical properties of special relevance within the control community [114,115]. See for example [114], where a NN is trained offline using samples of a robust MPC controller, and statistical guarantees on recursive feasibility, closed-loop stability, and constraint satisfaction are given based on the error between the learned control law and that provided by the robust MPC controller. Sharing similar goals, the authors of [118] also introduce a support vector machine-based method to build explicit MPC controllers for nonlinear systems with feasibility and stability guarantees. Further uses of learning techniques in the MPC context can be found in [119 – 121], where the goal is to aid the resolution of MPC problems of high complexity. As an illustrative application, see the mixed-integer vehicles’ platoon problem in [119], where the goal is to optimize lane-change decisions, and integer variables are introduced to select the best timing and spacing conditions. Although the last years have witnessed a huge growth of data-based and learningbased control methods, to the best of our knowledge, the contributions to distributed and decentralized control are still scarce. Nevertheless, many of the methods developed for centralized systems can be similarly extended to the distributed framework. Some examples are given in [122], which uses recurrent NN to identify the prediction model 10 Chapter 1. Introduction for DMPC controllers; in the previously mentioned [109,110], which apply the LMPC formulation in a decentralized and distributed context; and in [123], which introduces a distributed predictive safety certification based on [103] and on the robust DMPC in [124]. Also, distributed architectures open up new possibilities for the application of data-based methods, e.g., to aid the inputs coordination [125], to detect and isolate local faults [126], and to learn neighboring behaviors [127]. Additionally, following the approaches of [128,129], data could also be exploited to determine when the agents should or not communicate, aiming to avoid unnecessary information transmission and achieve better resources allocation. In this regard, within the particular framework of coalitional control, artificial intelligencebased solutions could provide benefits and new prospects to address the clustering problem. The suitability of different partitions may be characterized as a function of relevant system variables and evaluated by reward or loss functions. The latter paves the way for the application of reinforcement learning techniques to progressively improve the selection of coalitions by using previous observations. Finally, supervised approaches may be also of interest when it is possible to generate a training data set relating system variables to the corresponding optimal topology. 1.3 List of publications The works carried out for this thesis have originated the following publications: •Journal articles: 1. P. Chanfreut, J. M. Maestre and E. F. Camacho, “A Survey on Clustering Methods for Distributed and Networked Control Systems,” in Annual Reviews in Control, vol. 52, pp. 75-90, 2021. https://doi.org/10.1016/j.arcontrol.2021.08.002 2. P. Chanfreut, J. M. Maestre, F. J. Muros and E. F. Camacho, “Clustering Switching Regions for Feedback Controllers: A Convex Approach,” in IEEE Transactions on Control of Network Systems, vol. 8, no. 4, pp. 1730-1742, 2021. https://doi.org/10.1109/TCNS.2021.3084049 3. P. Chanfreut, J. M. Maestre and E. F. Camacho, “Coalitional Model Predictive Control on Freeways Traffic Networks,” in IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 11, pp. 6772-6783, 2021. https://doi.org/10.1109/TITS.2020.2994772 4. P. Chanfreut, J. M. Maestre, A. Ferramosca, F. J. Muros and E. F. Camacho, “Distributed Model Predictive Control for Tracking: A Coalitional Clustering Approach,” in IEEE Transactions on Automatic Control, Early Access, 2022. https://doi.org/10.1109/TAC.2021.3133486 1.3 List of publications 11 5. P. Chanfreut, J. M. Maestre, T. Hatanaka and E. F. Camacho, “Fast Clustering for Multi-agent Model Predictive Control,” in IEEE Transactions on Control of Network Systems, Early Access, 2022. https://doi.org/10.1109/TCNS.2022.3158745 6. J. M. Maestre, P. Chanfreut, J. Garc ´ ıa-Mart ´ ın, E. Masero-Rubio, M. Inoue and E. F. Camacho, “Control Predictivo de Sistemas Ciberf ´ ısicos,” in Revista Iberoamericana de Autom ´ atica e Inform ´ atica Industrial, vol. 19, no. 1., pp. 1-12, 2021. https://doi.org/10.4995/riai.2021.15771 •Conference papers: 1. P. Chanfreut, J. M. Maestre, F. J. Muros and E. F. Camacho, “A Coalitional Control Scheme with Topology-Switchings Convexity Guarantees,” 58th IEEE Conference on Decision and Control (CDC), Nice, France, 2019. https://doi.org/10.1109/CDC40024.2019.9030238 2. P. Chanfreut, J. M. Maestre, Q. Zhu and E. F. Camacho, “No-Regret Learning for Coalitional Model Predictive Control,” 21st IFAC World Congress, Berlin, Germany, 2020. https://doi.org/10.1016/j.ifacol.2020.12.1674 3. P. Chanfreut, T. Keijzer, R. Ferrari and J. M. Maestre, “A Topology-Switching Coalitional Control and Observation Scheme with Stability Guarantees,” 21st IFAC World Congress, Berlin, Germany, 2020. https://doi.org/10.1016/j.ifacol.2020.12.1792 4. P. Chanfreut, A. S ´ anchez-Amores, J. M. Maestre and E. F. Camacho, “Distributed Model Predictive Control based on Dual Decomposition with Neural- Network-based Warm Start,” 19th IEEE European Control Conference (ECC), Rotterdam, The Netherlands, 2021. https://doi.org/10.23919/ECC54610.2021.9655150 5. P. Chanfreut, J. M. Maestre, E. F. Camacho and F. Borrelli, “Collaborative Learning Model Predictive Control for Repetitive Tasks,” submitted to the 61th IEEE Conference on Decision and Control (CDC), Canc´ un, Mexico, 2022. Additionally, we indicate below other publications not included within this thesis but in which the author has also collaborated during her PhD period: 1. M. I. Fern ´ andez, P. Chanfreut, I. Jurado and J. M. Maestre, “A Data-Based Model Predictive Decision Support System for Inventory Management in Hospitals,” in IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 6, pp. 2227- 2236, 2021. https://doi.org/10.1109/JBHI.2020.3039692 2. T. Arauz, P. Chanfreut and J.M. Maestre, “Cyber-security in networked and distributed model predictive control,” in Annual Reviews in Control, In Press, 2021. https://doi.org/10.1016/j.arcontrol.2021.10.005 12 Chapter 1. Introduction 3. H. Degachi, P. Chanfreut and J. M. Maestre, “A nonlinear distributed model predictive scheme for systems based on Hammerstein model,” 21st IFAC World Congress, Berlin, Germany, 2020. https://doi.org/10.1016/j.ifacol.2020.12.1500 4. A. S ´ anchez-Amores, P. Chanfreut, J. M. Maestre and E. F. Camacho, “Coalitional Model Predictive Control with Different Inter-Agent Interaction Modes,” accepted in the 20th IEEE European Control Conference (ECC) and under review in European Journal of Control (2022 ECC Special Issue), 2022. 1.4 Outline of the rest of this thesis This dissertation is formatted as a collection of articles based on the publications listed in Section 1.3, i.e., [30 – 40]. In particular, the content of the following chapters is outlined below. • Chapter 2 introduces the main goals pursued in this thesis. The latter has been structured in six different blocks, which will be defined as Objectives 1 to 6, and which can be briefly described as follows: – Objectives 1 to 4 are framed within the field of coalitional control and include the development of supervised coalitional MPC algorithms and methods to address the associated clustering problem (see Section 1.1.3). Likewise, a further goal has been to provide a review of the state-of-the-art on distributed control schemes sharing, at least partially, the approach and fundamentals of coalitional control, such as the use of partial communication topologies. – Objective 5 includes the development of a cloud-based LMPC for multi-agent repetitive systems, and the use of NNs to aid the control inputs coordination in DMPC schemes. – Objective 6 considers the presentation of a tutorial with applications of MPC in cyber-physical systems. • Chapter 3 summarizes the contributions of this thesis and discusses the results presented in the corresponding articles. This chapter is divided into six sections as described below. – Section 3.1 focuses on the work of [30], which presents a literature review of existing control approaches based on sparse, and possibly varying, communication topologies. Special emphasis is placed on those methods providing increased scalability, flexibility, and/or modularity of distributed control architectures. We present taxonomies and discuss relevant commonalities and differences between the surveyed articles, as well as providing a review of applications in LSSs. – Section 3.2.1 describes the works of [31,32], which present LMI-based design procedures for coalitional linear feedback controllers based on [28]. In particular, [31,32] focus on the index used to dynamically select the coalitional structure and define the so-called topologies boundaries, which characterize 1.4 Outline of the rest of this thesis 13 the regions of the state space where some communication topology is more suitable than others. Additional LMI constraints to make these boundaries satisfy convexity, invariance, and submodularity properties are provided. – Section 3.3 presents the supervised coalitional MPC algorithms introduced in [34] (Subsection 3.3.1) and [35] (Subsection 3.3.2). In this regard, [34] presents a hierarchical coalitional MPC for freeways traffic systems, where a set of local MPC controllers dynamically adjust the variable speed limits (VSL) and on-ramp metering rates to reduce the time spent by the drivers traveling a given road. On the other hand, [35] introduces a coalitional robust MPC for tracking target sets based on the methodologies in [130 – 132]. The subsystems are robustified against neighboring and external uncertainties by using a tubebased control law, whose computation depends on the agents’ configuration. – Section 3.4 focuses on [36] (Subsection 3.4.1) and [37] (Subsection 3.4.2), which address the problem of dynamically optimizing the partition into coalitions. In particular, [36] formulates the clustering problem for a class of linear input-coupled systems as a binary quadratic program (BQP) where each variable represents the state (enabled/disabled) of a communication link, thus avoiding combinatorial explosion issues. Additionally, [37] uses a different approach based on multi-armed bandit (MAB) games [133,134], where the goal is to leverage real-time observations to progressively optimize the selection of the coalitional structures. – Section 3.5 discusses the works of [38] (Subsection 3.5.1) and [39] (Subsection 3.5.2), which apply learning techniques in non-centralized schemes. In this regard, [38] proposes the use of a NN-based warm start in cooperative DMPC schemes based on dual-decomposition. Particularly, the goal is to provide at each time step the Lagrange multipliers that directly leads to fully coordinated control inputs. Additionally, [39] presents a cloud-based LMPC based on [101], where a set of agents repeatedly perform a finite set of tasks subject to varying state constraints. The data collected by each agent is uploaded to a common cloud to help minimize its own and its peers’ incurred costs. – Section 3.6 focuses on the work of [40], where the authors review a series of examples of applications of MPC to cyber-physical systems. In particular, this paper describes applications in solar plants, irrigation canals, inventory systems, and energy management systems. • Chapter 4 provides a copy of the articles published within this thesis, which are in turn listed in Section 1.3. This chapter is further divided into two sections: the first one focusing on journal articles (Section 4.1), and the second one on conferences (Section 4.2). •Chapter 5 presents our conclusions and future research directions. 2 Objectives This chapter describes the objectives pursued in this thesis. In particular, the first four objectives introduced below are framed within the field of coalitional control, and are associated with most of the journal publications of this thesis. In addition, the fifth objective focuses on the use of data-based methods in MPC-based schemes; and, finally, the sixth considers applications of MPC in cyber-physical systems. • Objective 1: Review the state-of-the-art on distributed control schemes based on sparse, and possibly varying, communication topologies. Given the strong focus of this thesis on the coalitional approach [28,29], one of our goals has been to thoroughly review the literature on control methods that share, at least partially, the objectives and fundamentals of this type of controllers. This review should include methods that allow boosting the scalability of distributed architectures through an efficient management of the communication and computation resources. Special emphasis has been placed on those strategies based on the formation of clusters or operational areas where the control actions are set with minimal communication and coordination with the rest of the system. A relevant issue here is the blurry difference from the controller design standpoint between those methods where communication constraints are deliberately imposed, and those where they result from limitations or failures of the data network. For this reason, this objective also includes the review of control approaches dealing with eventbased and unpredictable communication structures. The main goal is to provide a unifying framework gathering the main research directions in this regard while drawing out their main commonalities and differences. • Objective 2: Development of methods for characterizing the switching between topologies in coalitional linear feedback controllers. The first works of this thesis within the coalitional control framework followed the formulation proposed in [28], which is also used in [74 – 76]. In this context, we aim at characterizing the transitions between partitions into coalitions while exploring the concept of boundaries and coordination effort zones. In particular, the goal is to 15 16 Chapter 2. Objectives propose LMI-based design procedures that make the criterion for switching satisfy certain properties of interest. First, the latter is designed to obtain ellipsoidal regions in the state space where certain topologies are preferred over others. By exploiting the convex property of these ellipsoidal regions, a second goal has been to derive the mentioned coordination effort zones to characterize the need for coordination at different system states. Additionally, by considering invariance, we aim at providing new ellipsoids that confine the system state between topologies transitions. Finally, an additional goal has been to attain submodularity in the pursuit of facilitating the selection of the coalitions. •Objective 3: Development of supervised coalitional MPC algorithms. This objective is divided into two subgoals. Indeed, we have considered two distinct research directions to contribute to the framework of coalitional MPC algorithms: on the one hand, we have focused on simulating and analyzing the performance of this strategy on a large-scale application; and, on the other hand, we have adopted a more theoretical perspective, aiming at developing novel coalitional MPC methods with robust and stability guarantees. These subgoals are further described below: – Objective 3.1: Development of a coalitional MPC algorithm for a large-scale application. This subgoal is in line with the works of [29] and [77], which propose a coalitional MPC for smart grids and irrigation canals, respectively. By contrast, and considering the positive results of MPC in traffic systems reported in previous works [135], [136], [137], this thesis takes freeways as a case study. We assume that there are variable speed limits (VSL) and controllable on-ramp entries along the road under study, which are managed by a set of local MPC agents. Our aim has been to implement a hierarchical structure similar to that in [77], where a supervisory layer dynamically arranges these agents into cooperative coalitions, and hence fosters the coordination of the traffic signals between different sections of the freeway. The control goal is to reduce the time spent by the drives on the road, and thus to reduce traffic jams, while efficiently managing the computational and communication resources. – Objective 3.2: Development of robust coalitional MPC controllers for tracking . The lack of previous studies on tracking approaches within the field of coalitional MPC has motivated this subgoal. In particular, we have focused on extending the MPC for tracking formulation presented in [131,132], which in turn follow [130], to the coalitional framework. In this context, one of the main goals is retaining their robust and convergence guarantees for tracking changing targets regardless of the switching between topologies that characterize the coalitional approach. •Objective 4: Development of fast clustering methods for multi-agent MPC. This objective focuses on the clustering problem described in Section 1.1.3, which is inherent to the implementation of the coalitional strategy. In particular, the aim is to introduce an efficient method for updating the number and composition of the coalitions in real time. We consider coalitional schemes where the network topology 17 is selected as a function of the system state and updated at a lower frequency than the system inputs. That is, once certain agents join in a coalition, they remain together coordinating their inputs during a certain number of time steps, which in turn drives the system to new states. The underlying goal is therefore to find a fast approach to estimate which clusters provide the best trade-off between performance and coordination costs at a given system state, while dealing with the combinatorial explosion of possibilities. We adopt two different perspectives for tackling this objective: first, the clustering problem is formulated as a MAB game [133,134], where the most promising partition according to previous observations is selected; and, secondly, we opt for a more analytical perspective that seeks to assess how the breaking up into coalitions affects the MPC optimization problem in comparison with the centralized formulation. • Objective 5: Application of learning techniques in non-centralized MPC-based control schemes. Within this objective, which may include a wide range of possibilities, we have particularly focused on two ideas. The underlying goal is to aid the MPC agents coordination and/or boost their performance by using data. Nevertheless, the mentioned ideas follow very distinct perspectives as it is described below. – Objective 5.1: Use of NNs to aid the inputs coordination in DMPC schemes based on dual decomposition. The goal here is to speed up the iterative negotiations that local controllers perform in dual-decomposition-based DMPC algorithms. To this end, our proposal aims at incorporating pre-trained NN models to provide a close to optimal warm-start of the Lagrange prices that guide the iterative negotiations at each time instant. In this way, we aim at reducing or, if possible, avoiding, the need to iterate to obtain coordinated inputs, and thus reducing the computation and communication demands in the online operation of the agents. This objective is also related to the coalitional framework, since a long-term goal is to exploit this approach to determine when control agents should negotiate their actions and when it does not translate into significant performance improvements in comparison with a decentralized operation. – Objective 5.2: Development of a multi-agent LMPC with collaborative learning to optimize repetitive tasks. Here we consider a set of MPC agents that must execute a finite set of tasks at their minimum possible cost. Our goal is to use the LMPC strategy to progressively optimize their performance, while using a data cloud as a means to enable collaboration (by data sharing) and keeping the computation of the inputs decentralized. In this context, we must also deal with changes in the constraints for executing these tasks and exploit similarities between them to accelerate the learning process. A key issue is retaining the theoretical guarantees of the LMPC formulation, particularly recursive feasibility, asymptotic stability, and a non-increasing evolution of the tasks’ costs. 24 Chapter 3. Summary of results and discussion 3.3 Supervised coalitional MPC schemes This section summarizes the works of [34,35]. Following Objective 3 in Chapter 2, [34] focuses on the application of coalitional MPC to traffic freeways, whereas [35] adopts a more theoretical perspective and aims at developing a coalitional MPC for tracking. 3.3.1 Coalitional MPC in traffic freeways During the past decades, traffic jams have become a daily hassle in many towns and cities worldwide. The latter has prompted research into optimal control methods to alleviate this problem and its consequences, which includes MPC-based approaches such as [135 – 137]. Following this line of research, and also motivated by the positive results of coalitional control in [77], the work of [34] focuses on the application of coalitional MPC to traffic freeways. It is assumed that the freeway under study counts with variable speed limits (VSL) and controllable on-ramp entries, which allow acting dynamically on the traffic flows to relieve traffic jams. Model METANET, which provides a fluid-like macroscopic description of the traffic behavior, is used to predict the density of vehicles and mean speed at different segments of the freeway. The main control goal is to minimize the total time spent by the drivers (TTS) on the road while reducing coordination efforts required to compute the VSL and ramp-metering rates. The nonlinearity of the traffic behavior and the mixed-integer nature of the control actions hinder the implementation of a centralized MPC approach for real time control. To deal with this issue, [34] proposes a top-down architecture (see Fig. 3.3) where the overall problem is dynamically broken into smaller pieces, which accordingly involve a reduced number of optimization variables in comparison with the centralized formulation. Indeed, the freeway is partitioned into stretches that are assigned to a set of distributed MPC agents. These agents constitute the bottom layer and are in charge of managing the VSL and ramp-metering signals that are located in their corresponding stretches. On the other hand, the top layer decides periodically which of them should merge into coalitions, thus fostering coordination in certain parts of the system. In this regard, each coalition can be seen as an independent entity where the involved agents solve cooperatively a constrained 2 Topology selection 134 5 6 Coal. 1 Bottom layer Top layer Measurements Control actions (VSL and ramp metering) Flow of vehicles Coal. 2 Coal. 3 Coal. 4 VSL VSL VSL Figure 3.3 Scheme of the top-down approach for traffic freeways implemented in [34]. 3.3 Supervised coalitional MPC schemes 25 Table 3.2 Summary of the simulation results in [34]. Decent. cFC-MPC 5.5 4.5 1 TTS (veh·h) 2682 2647 2632 2608 2588 % TTS Red. Links 1-3 8.90 10.60 10.77 10.49 9.71 Links 4-6 2.09 2.91 5.40 7.94 10.23 Global 4.01 5.27 5.80 6.63 7.38 TWT (veh·h) On-ramp 1 31.78 38.02 42.51 42.85 44.55 On-ramp 2 0.52 0 18.79 34.77 34.66 On-ramp 3 21.26 21.50 26.66 21.19 23.72 Comp. Time (min) 5.38 6.53 9.94 13.07 25.75 Figure 3.4 Evolution of the density of vehicles in veh/km along the road simulated in [34]. The left plot shows the results obtained in the no control case, and the right one refers to the coalitional MPC controller with c=4.5. MPC problem aiming at improving the aggregated traffic performance along their freeway stretches. A genetic algorithm is used to optimize the sequences of VSL and ramp-metering, and a heuristic criterion is proposed to decide the partitions into coalitions at the top layer. This heuristic criterion is designed so as to foster coordination between those controllers associated with stretches of road where the traffic situation is more critical. Our simulation results in [34] consider an example of a 15km freeway with three metered on-ramps, three speed limits, and one off-ramp. Likewise, the system is partitioned into six locally managed stretches, and it was possible to impose 13 different communication topologies between the corresponding agents. Two traffic jams were intentionally induced at different parts of the freeway through a proper setting of the mainstream traffic flow and the number of cars arriving at the on-ramps. As indicated in Table 3.2 ([34, Table III]), the proposed coalitional approach is shown to provide a balance between performance optimality and efficiency. In particular, the TTS and the computation times required for the simulations were in-between those obtained when imposing statically the decentralized topology and when using a fully cooperative MPC strategy. The results reported in Table 3.2 also show the total waiting times at the on-ramp queues (TWT), and consider different 26 Chapter 3. Summary of results and discussion values of parameter c , which penalizes the formation of coalitions. That is, the higher c is, the sparser the topologies chosen. In this regard, for c=1 the TTS global reduction was 6.63%, and for c=5.5 it was 5.27%, while the decentralized case led to 4.01% and full cooperation to 7.38%. For the case of c=4.5 , we also provide color maps illustrating the evolution of the density of vehicles in the no-control case, i.e., assuming that the VSL are always set at their maximum possible value and that the on-ramp accesses are not regulated, and when using the presented coalitional MPC controller (see Fig. 3.4, which corresponds with [34, Fig. 6 and 7]). Additionally, [34] includes graphical results illustrating the evolution of the vehicles’ mean speed in each stretch over time, the waiting queues at each on-ramp, the implemented control actions, and the chosen coalitions for two different values of c. 3.3.2 Coalitional MPC for tracking The main result of [35] is the introduction of a tracking strategy with theoretical guarantees into the field of coalitional control. In particular, [35] focuses on the tube-based MPC for tracking in [131], which presents a robust version of the controller proposed in [130]. Additionally, we adopt the approach in [132], where the goal is to admissibly drive the system state to a given target set rather than to a point. The chosen tracking formulation is characterized by the introduction of an artificial setpoint as optimization variable in the MPC problem, the use of an invariant set for tracking as terminal set, and the use of an offset cost function that penalizes the deviation between the artificial and the real setpoints. As presented in [130] for centralized systems, this type of controller allows enlarging the domain of attraction in comparison to that of the regulation problem to a certain setpoint, while providing persistent feasibility and convergence guarantees. In [35], we consider linear multi-agent systems with coupled dynamics that can be periodically organized into coalitions. In particular, each subsystem’s state is coupled to its neighbors’ states and inputs, and is also influenced by an external perturbation. Both the neighboring and external disturbances are assumed to be uncertain from a local standpoint, but confined within convex compact polytopes. The system is robustified against the latter by using the tube-based [87,88] approach proposed in [131]. In this context, the agents solve a nominal MPC problem formulated according to [131,132], where the constraint sets are shrunken as a function of the size of the uncertainty. Likewise, the resulting nominal input is corrected by an ancillary feedback law that keeps the real system evolution in a neighborhood of the nominal predictions. The selection of the agents partition is based on a quadratic function that is designed similarly to the upper bound on the cost-to-go used in [28,31,32]. Once a certain partition is imposed, within each resulting coalition the agents solve cooperatively a nominal MPC problem where the objective function is the sum of that of their local problems, and the constraints sets are adjusted to only consider as coupling uncertainty the effect from outsider subsystems. Likewise, the ancillary feedback gain is switched to match the communication constraints of the chosen topologies. Therefore, the formation of coalitions has three main effects: first, it allows the agents to use less restrictive constraints in their nominal MPC problem; secondly, the resulting nominal inputs seek to maximize 3.3 Supervised coalitional MPC schemes 27 the coalitional nominal performance, instead of the local one, which usually translates into global benefits; and, thirdly, the agents can use local and neighboring variables to define their ancillary feedback law. Finally, under certain conditions on the switching process, recursive feasibility, stability of the closed-loop system, and constraints satisfaction are guaranteed, and from these convergence to the target sets is also proven. We provide simulations results on a trucks system connected by spring and dampers similar to the one in [32]. However, here we have 12 trucks and it is possible to impose 1450 different communication topologies. The simulations considered a time period of 18s, with a change of the target set at 9s. Fig. 3.5 ([35, Fig.3]) shows the state trajectory of each subsystem i∈[1,2,...,12] on the plane given by its state components, which here are denoted as si (position) and vi (velocity). Also, Fig. 3.6 ([35, Fig.4]) illustrates the evolution of the coalitions over time. It can be seen how the agents are able to drive their states to their corresponding target sets while progressively disabling the communication links. Indeed, when all subsystems are in the vicinity of their target sets, the system uses the decentralized configuration. Additionally, [35] shows that the increased coupling uncertainty and lack of cooperation involved by the decentralized configuration leads to reduced constraints sets and greater losses of performance. In this regard, the proposed coalitional scheme is shown to achieve a reduction of 11.6% of the cumulative performance costs resulting from a static use of the decentralized topology ([35, Fig.6]). Finally, we also include graphical results to illustrate that the nominal and real states remain within the invariant tube ([35, Fig.7]). -1 0 1 -1 0 1 -4 -2 0 2 4 -1 0 1 -4 -2 0 2 4 Figure 3.5 State evolution of a 12-trucks system with the coalitional MPC for tracking presented in [35]. The figure illustrates the evolution of the system to two target sets. The first ones are shown in light blue and red, whereas the green and orange regions represent the second target sets. The subsystems states at the switching instants are indicated by star marks, and the initial and final states are marked respectively with a circle and a diamond. 28 Chapter 3. Summary of results and discussion Figure 3.6 Evolution of the network topology in the simulation example of [35]. 3.4 Methods for the coalitions selection The works of [36,37] are also framed within the supervised coalitional MPC framework. However, unlike [34,35], they focus on the problem of selecting the agents partition, i.e., finding which coalitions should be formed to provide the best trade-off between global performance and coordination effort. Both works consider the double sample rate strategy implemented in [31 – 35], where the system control inputs are updated by the coalitions at each time instant, whereas the agents partitions can only be changed periodically. 3.4.1 Binary quadratic program In [36], the clustering problem is formulated as a binary quadratic program (BQP) where the optimization variables are directly the state (enabled/disabled) of the communication links. The solution of this BQP establishes therefore a certain network topology, and thus partitions the agents into the corresponding coalitions (see Fig. 1.2). This approach avoids combinatorial explosion issues, since for a network with L links, we would have L binary variables, whereas if we formulate a decision problem directly on the agents’ partition, then we may have to deal with 2Ldecision variables. The BQP problem is derived for a class of linear input-coupled systems by analyzing how the decentralization of the local agents, i.e., the use of any topology different from the fully connected case, alters the overall MPC optimization problem. This MPC problem is built by aggregating all coalitions’ optimizations into a single formulation, which allows us to assess the global behavior. In this regard, the key to render the clustering problem as a BQP lies in the decomposition of the gradient vector of this overall MPC problem as a sum of a vector associated with the fully decentralized topology, and others terms that depend on the enabled links. By using the latter, we then assess how far the optimal centralized input sequence is from satisfying its optimality conditions. The underlying idea is that if it is close to being optimal, then the inputs obtained by decentralized coalitions will be close to that of the centralized case, thus leading to small performance losses. Also, we should remark that [36] first derives the proposed BQP considering the unconstrained case, and later introduces a heuristic approach to include decoupled input constraints. Moreover, theorems relating the performance cost function to the BQP problem are included. 3.4 Methods for the coalitions selection 29 5 10 15 20 25 30 35 40 0 5 10 15 20 25 30 35 Figure 3.7 Comparison in terms of computation time between different methods for selecting coalitions [36]. This figure was obtained considering a linear network with an increasing number of agents (see [36, Section V.A]). We provide numerical results comparing the proposed BQP approach in terms of computation time and performance with the solutions obtained by exhaustive search, following a greedy-based procedure, and implementing the genetic algorithm (GA). In this regard, [36] considers two cases of study: first, we use a linear network with a variable number of agents, where no constraints are considered; and secondly, an urban traffic network modeled as in [180], where decoupled input constraints are introduced. To evaluate the results, [36] uses the following indexes: %Loss of opt.(Λ) = 100J(x,U U U∗ Λ)−J(x,U U U∗ cen) |J(x,U U U∗ cen)|,and %Coord.(Λ) = 100|Λ| L,(3.2) where J(·) denotes the global performance cost function, U U U∗ Λ and U U U∗ cen are respectively the optimal input sequences obtained with topology Λ and the centralized one, |Λ| is the number of enabled links in Λ , and L is the total number of links as mentioned before. Note that the first index in (3.2) assesses the loss of performance optimality in comparison with the centralized approach for a given system state x , whereas the second index evaluates the percentage of enabled links and thus the coordination burden. In the linear network example, we progressively build larger systems by increasing the number of agents, and, for each of them, we solve the clustering problem for 200 different states. Fig. 3.7 ([36, Fig. 6]) illustrates the total time spent on the latter using the mentioned methods. It can be seen how the exponential growth of the number of topologies impacts the time required by the exhaustive search method and the GA, whereas the BQP scales gracefully for larger systems. This also reflects that, from a certain number of agents, the time required for implementing the proposed BQP is still lower than that to evaluate and compare all possible topologies. Nevertheless, faster decisions can be made by the greedy-based approach if the matrices involved in the MPC control law of each possible topology can be precomputed and stored offline. However, the latter increases the memory requirements and offline computation costs, which is why this approach is only simulated until 25 agents. Likewise, Table 3.3 provides the values of (3.2) incurred by the partitions 30 Chapter 3. Summary of results and discussion Table 3.3 Comparison of the BQP approach in [36] with other clustering methods in terms of performance optimality and coordination costs. Ex. search GA BQP Greedy Linear net. Av. loss of opt. [%]0.11 0.23 0.12 0.15 Av. coord. [%]69.80 67.72 69.85 67.34 UTN Av. loss of opt. [%]0.016 0.024 0.025 0.033 Av. coord. [%]54.55 54.55 54.55 54.55 selected for systems with 7 to 15 agents, showing that the BQP approach practically attains the optimal clustering and thus finds a balance between the centralized and decentralized topologies. In this regard, note that the optimal clustering is found by exhaustive search, and that the centralized configuration leads respectively to 0% of performance optimality losses and 100% of coordination burden. Also, the values associated with the decentralized case were in this example respectively 3.5% and 0%. In the second example, we simulate an UTN with 8 controllable junctions and 17 streets. It is assumed that there is a control agent assigned to each junction that should manage the traffic flows by acting on the corresponding traffic lights. Additionally, by using the proposed BQP, the communication topology between these agents can be periodically switched. At each switching instant, we compare the coalitions provided by the BQP with the methods used in the previous example as shown in Table 3.3. Likewise, the average required times required to make the topologies decisions were respectively 0.13s (BQP), 11.45s (exhaustive search), 84.06s (GA), and 0.29s (greedy). In this example, all methods led to very small losses of optimality and the GA slightly outperformed the BQP in this sense. However, note that the fact of dealing with constraints notably increased the time needed to optimize the input sequences, and hence to try out all topologies and to implement the GA, in comparison with that of the BPQ approach. Finally, it should be mentioned that the parameters set for the GA differed between the first and second examples. Whereas in the first case they were set to obtain comparable computation times to the other methods, in the second one we increased the maximum number of generations to show that it can attain close to optimal clustering at expense of increasing the computation times. 3.4.2 Multi-armed bandit problem On the other hand, [37] addresses the clustering problem using a contextual multi-armed bandit game approach [134,181]. Broadly speaking, multi-armed bandit (MAB) games are a kind of sequential games, where a decision entity, often referred to as player, selects one arm (i.e., one action) from a finite set of possibilities, and subsequently receives a loss that weights the suitability of its choice. The term contextual implies that the player counts with some relevant information in the form of a context vector for taking its decision. Based on this philosophy, the main result of [37] is a heuristic learning algorithm that leverages real-time observations to progressively learn the coalitional structure that best suits the needs of the system at a certain state. In particular, we define the set of arms 3.4 Methods for the coalitions selection 31 as the set of possible agents partitions and the player performs as a supervisory entity that can periodically select one of them, and thus change the number and composition of the coalitions. Likewise, the loss is defined as a bi-criteria function weighting the overall system performance and coordination efforts provided by the selected partition, and the context vector is given by the current system state. The player progressively builds a database that stores a set of historic system states, its selected topologies, and the observed losses. This information is then used as the basis to improve the coalitions selection over time by considering that the loss inflicted by a given topology at a certain state remains similar in its vicinity. The player’s goal is therefore to balance exploration and exploitation so as to optimize its decisions. To evaluate its performance, we use a regret function that weights the difference between the player’s cumulative loss and the loss of playing the best in hindsight. Additionally, we propose the use of multi-class classification techniques to build in real-time a model that matches the system states to suitable partitions by exploiting the collected data. To illustrate the proposed MAB approach, we use an academic example similar to the ones in [32,35], where 5 coupled trucks should admissibly regulate their displacement and speed to the origin, and 16 communication topologies may be imposed in the data network. For simplicity, [37] repeatedly simulates the system’s behavior from the same set of initial states, which increases the utility of the collected data to improve future decisions. Each of these simulations is called a cycle and involves the selection of a set of topologies for different states. In particular, in each cycle, once the system reaches the origin, the coalitional control scheme based on MAB is again simulated from a different initial point, until we finish with all in the mentioned set of initial states. As it is shown in Fig. 3.8, the proposed heuristic algorithm allows for progressively improving the selected coalitions, thus reducing regret over time. Note that the percentage of optimal decisions remains below 30% at the beginning of the simulations due to the lack of data, and progressively increases until reaching around 80%. Additionally, Fig. 3.9 illustrates how the multi-class classification allows deriving regions in the state space where one topology is expected to be optimal. To allow for a bi-dimensional representation, we show the 0 5 10 15 20 0 200 400 600 800 1000 Cumulative Regret / cycle 20 30 40 50 60 70 80 90 % Optimal topology choices Figure 3.8 Learning regret and percentage of optimal decisions per cycle with the MAB method described in [37]. 32 Chapter 3. Summary of results and discussion -4 -3 -2 -1 0 1 2 3 4 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 Figure 3.9 Regions derived from the multi-class classification of the states in [37]. Letter C is used to denote the coalitions proposed by the classifier. plane given by the state components of truck 1, considering that the rest subsystems states are equal to zero. It should be remarked that around the origin, the classifier selects the decentralized configuration (yellow region in Fig. 3.9), whereas topologies involving denser communication are preferred further from the origin. 3.5 Data-based methods in non-centralized MPC This section introduces the conference papers [38,39], which propose respectively the use of NNs in DMPC schemes and a cloud-based LMPC for multi-agent repetitive systems. Note that although [37] also uses a data-based approach, it has been introduced in the previous section due to its focus on the coalitional framework and the problem addressed. 3.5.1 Dual decomposition-based DMPC with NNs Dual decomposition allows attaining coordination between distributed MPC controllers by introducing in their optimization problems a set of Lagrange multipliers that seek to enforce an agreement on the coupled variables [23]. In this context, the MPC agents implement at each time step an iterative negotiation where their control inputs and Lagrange multipliers are repeatedly updated until convergence is attained. This procedure can provide centralized performance but demands increased computation and communication resources when the convergence rate is low. Considering the above, [38] proposes the use of NNs to accelerate the inputs computation in cooperative DMPC schemes based on dual-decomposition. In particular, the goal is to provide a NN-based warm start for the distributed negotiation that directly leads to the solution at convergence, and therefore to fully coordinated control inputs. The proposal is tailored for linear input-coupled systems where the performance function only depends on the current system state and the sequence of implemented inputs. The NN-based model is trained offline by using a pre-computed data set where the input variables are different system states and the target variables are given by the Lagrange multipliers obtained at 3.5 Data-based methods in non-centralized MPC 33 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 0 10 20 30 40 50 60 Centralized MPC DMPC with NN-based warm start Figure 3.10 Comparison between the DMPC based on dual decomposition with NN-based warm start proposed in [38] and the corresponding centralized MPC. The figure illustrates the cumulative performance costs in 20 simulations starting from different random states. convergence when implementing accordingly the distributed negotiation. Simulations on a 16-tanks benchmark are provided, which assess the performance of the proposed approach from two standpoints: first, we assume that the MPC agents directly implement the inputs obtained with the NN-based initialization of the Lagrange prices at each time step, i.e., they do not iterate; and, secondly, we consider that these Lagrange prices can be used as an initialization of the negotiation, but thereupon, the agents should proceed iterating until satisfying a convergence condition. In this regard, the proposal also suggests switching to a warm start based on the Lagrange prices obtained at the previous time instant when the system state remains approximately constant, since in this case, these values may be more accurate than the ones provided by the NN. Fig. 3.10 compares the first of the approaches above with the corresponding centralized MPC results. In particular, it shows the cumulative performance costs in 20 different simulations that start from random initial states. As can be seen, the resulting costs barely differ from the centralized outcome although no iterations are performed, which also reflects that the Lagrange prices provided by the NN are close to being optimal. Regarding the second approach, Table 3.4 ([38, Table II]) provides the cumulative costs obtained in a given simulation when considering different values of parameter ¯p , which denotes the maximum number of allowed iterations. In this regard, the step size in the Lagrange prices’ Table 3.4 Cumulative performance cost for different values of the maximum number of iterations ( ¯p). Cumulative cost ¯p=130.9123 ¯p=530.2270 ¯p=10 30.0063 Centralized MPC 29.9156 40 Chapter 4. Publications 4. P. Chanfreut, J. M. Maestre, A. Ferramosca, F. J. Muros and E. F. Camacho, “Distributed Model Predictive Control for Tracking: A Coalitional Clustering Approach,” in IEEE Transactions on Automatic Control, Early Access, 2022. https://doi.org/10.1109/TAC.2021.3133486 Summary: Section 3.3.2 5. P. Chanfreut, J. M. Maestre, T. Hatanaka and E. F. Camacho, “Fast Clustering for Multi-agent Model Predictive Control,” in IEEE Transactions on Control of Network Systems, Early Access, 2022. https://doi.org/10.1109/TCNS.2022.3158745 Summary: Section 3.4.1 6. J. M. Maestre, P. Chanfreut, J. Garc ´ ıa-Mart ´ ın, E. Masero-Rubio, M. Inoue and E. F. Camacho, “Control Predictivo de Sistemas Ciberf ´ ısicos,” in Revista Iberoamericana de Autom´ atica e Inform´ atica Industrial, vol. 19, no. 1., pp. 1-12, 2021. https://doi.org/10.4995/riai.2021.15771 Summary: Section 3.6 4.2 Conference articles The articles that have been accepted for their presentation at conferences, and which have also resulted from this thesis are given below. Likewise, we include a further contribution that has not been published yet but is under revision for the 61st Conference on Decision and Control (CDC 2022). 1. P. Chanfreut, J. M. Maestre, F. J. Muros and E. F. Camacho, “A Coalitional Control Scheme with Topology-Switchings Convexity Guarantees,” 58th IEEE Conference on Decision and Control (CDC), Nice, France, 2019. https://doi.org/10.1109/CDC40024.2019.9030238 Summary: Section 3.2.1 2. P. Chanfreut, T. Keijzer, R. Ferrari and J. M. Maestre, “A Topology-Switching Coalitional Control and Observation Scheme with Stability Guarantees,” 21st IFAC World Congress, Berlin, Germany, 2020. https://doi.org/10.1016/j.ifacol.2020.12.1792 Summary: Section 3.2.2 3. P. Chanfreut, J. M. Maestre, Q. Zhu and E. F. Camacho, “No-Regret Learning for Coalitional Model Predictive Control,” 21st IFAC World Congress, Berlin, Germany, 2020. https://doi.org/10.1016/j.ifacol.2020.12.1674 Summary: Section 3.4.2 4.2 Conference articles 41 4. P. Chanfreut, A. S ´ anchez-Amores, J. M. Maestre and E. F. Camacho, “Distributed Model Predictive Control based on Dual Decomposition with Neural-Network-based Warm Start,” 19th IEEE European Control Conference (ECC), Rotterdam, The Netherlands, 2021. https://doi.org/10.23919/ECC54610.2021.9655150 Summary: Section 3.5.1 5. P. Chanfreut, J. M. Maestre, E. F. Camacho and F. Borrelli, “Collaborative Learning Model Predictive Control for Repetitive Tasks,” submitted to the 61th IEEE Conference on Decision and Control (CDC), Canc´ un, Mexico, 2022. Summary: Section 3.5.2 5 Conclusions and future research directions As it has been seen in [30], a growing number of works and applications are consolidating the research area of distributed control with partial and varying communication topologies. In this context, this thesis reaffirms the potential that coalitional control has shown in previous works to provide a balance between global performance and coordination efforts, e.g., [29,77]. In particular, it has been seen that by focusing the communication and computation resources only on clusters of highly coupled control agents, it is possible to obtain a performance close to the centralized approach with lighter computations. This thesis contributes to the development of these methods by addressing the agents’ clustering problem, which, given its combinatorial nature, becomes a key issue for the real-time implementation of coalitional controllers [36,37]. Our simulation results on traffic applications, which include freeways and urban networks, provide illustrations of this strategy [34,36]. Additionally, by merging the results in [130,131] with the coalitional methodology, we have developed new theoretical results to provide this clustering strategy with robust and stability guarantees to track changing targets [35]. We have also started to pave the way to merge the coalitional strategy with machine learning techniques by developing different research efforts in this direction. In this regard, machine learning techniques offer potential solutions to enhance distributed control schemes by using data, and can also be exploited for this dynamic clustering approach. This thesis has focused on the use of NN to aid distributed negotiations [38], and on the development of a multi-agent LMPC based on a collaborative data collection [39]. The coalitional control strategy studied in this thesis, which in essence is a distributed strategy that allows a varying degree of coordination, presents significant challenges that are likely to persist, thus providing interesting research opportunities for future works. Some of the possible future research directions are briefly discussed below: • Integration of heterogeneous dynamics and plug-and-play events: Besides allowing dynamical changes of the control communication network, future coalitional control 43 44 Chapter 5. Conclusions and future research directions schemes should increase their flexibility to accommodate changes in the number and structure of the agents, and in their local dynamics. An illustrative application where this is interesting is that of vehicles’ platoons, which is the case study considered in [33]. In this context, new vehicles may unpredictably join and then leave the system, thus modifying the neighboring structure that the remaining vehicles should consider, and also their communication connections. Furthermore, the design of each vehicle determines its dynamic behavior, and therefore any change should be reflected in the prediction model employed to optimize the inputs. In addition, future work should extend the LMPC-based proposal in [39] to a more general framework where the agents’ dynamics and constraints are not identical, and therefore the data collected by some of them may not be directly exploited by the rest of the agents. • Switching instants: As mentioned in Section 1.1, most existing works on coalitional control use a double sample rate strategy where the network topology is changed periodically, whereas the control actions are computed at each time step according to the imposed partition. The latter intends to approximate the solution obtained by minimizing (1.1) while avoiding mixing in the same optimization problem variables associated with the network configuration and with the system control inputs. Nonetheless, further benefits could be obtained if the switching instants are optimized rather than predefined. For this reason, future work should also study whether there is possible to predict the optimal switching timing without incurring in computationally-expensive solutions. It could be of special interest to generate offline a model mapping system variables (e.g., the system state) with suitable topologies in the line of the multi-class classifier in [37] or the dominance regions described in [32]. If such a model is available, one could estimate the amounts of instants until the topology should be switched by using predictions of the system trajectory. Likewise, as proposed in [183], it is also interesting to integrate the switching of topologies in the prediction horizon so as to facilitate a gradual transition between communication configurations. • Clustering decisions: Artificial intelligence-based solutions could provide further benefits and new prospects to tackle the clustering problem in coalitional MPC. As proposed in [37], one can exploit data collected online to improve the decisions on the agents’ partition by using a reinforcement learning approach. For systems showing a repetitive behavior and involving a small number of topologies, this approach can find optimal partitions after a few repetitions. Nevertheless, the combinatorial growth of the number of partitions with the number of communication links enlarges the range of decisions that can be made and thus complicates the learning process. New heuristics and more sophisticated proposals are of interest to alleviate the latter issue. Moreover, it is interesting to study how supervised learning techniques can aid to address this problem. If it is possible to generate offline a database relating system variables to optimal partitions, neural networks can be a promising solution to precompute a model that approximates the optimal clustering. The latter can be subsequently integrated into the online operation as a decision-maker that selects a network topology by using real-time measurements of the corresponding system variables. Moreover, it may be worth exploring new reformulations of the clustering 45 problem, which, although not directly dealing with objective function (1.1) , can provide efficient agents’ partition while alleviating the inherent difficulty of this problem as intended in [36]. Finally, besides focusing on the trade-off between performance and coordination costs, coalitional schemes can incorporate further criteria to take the clustering decisions, such as the local benefits that merging into coalitions provides to each of the aggregated entities. • Theoretical properties: Although recent research efforts have provided coalitional controllers with theoretical guarantees [35,78,79], the lack of full information sharing and the switching nature of the control communication network complicates the derivation of basic properties such as recursive feasibility, stability, and robustness. Indeed, an inherent feature of coalitional schemes is that each cluster has to deal with uncertainty regarding the behavior of the rest of the system, which undermines the accuracy of their predictions and endangers the satisfaction of the constraints. Future work should study methods able to handle efficiently coupling uncertainties without resorting to overconservative approaches. Recent advances in this regard are reported in [79], which proposes a coalitional controller built upon tube-based MPC ideas [87,88], but which only relies on the implicit existence of the required control invariant sets. Given the difficulty to compute high-dimensional invariant sets, this approach can bring significant implementation benefits when dealing with large clusters. In particular, it may help to reduce the implementation complexity of the coalitional MPC for tracking presented in [35]. Additionally, the coalitional framework goes hand-in-hand with cooperative game theory, especially when the merger of local control units generates a certain utility that can be distributed in the form of local benefits. Therefore, the study of the properties of this game and the development of fair profit allocation mechanisms are also of interest in this field [75,78]. • Data-based solutions in coalitional schemes: To the best of the authors’ knowledge, in the coalitional schemes proposed so far the agents rely on the resolution of an optimization problem or on the evaluation of a precomputed law to find their control inputs. However, the use of learning-based local agents is yet to be explored. Different formulations can be tested in this regard. For example, we may consider a reinceforment learning approach [106] where the agents’ observations depend on the clusters they join, thus allowing them to gather or not neighboring information to optimize their actions. In addition, a similar idea to the safe sets in [101] may be useful to define terminal constraints aiming at providing persistent feasibility and stability properties in the coalitional switching framework. In this regard, we could collect points from which convergence to the target can be guaranteed even if the control communication network switches to a complete decentralized topology. Finally, the loss of overall performance caused by the decentralization of the control structure may be alleviated if the local controllers count with accurate predictions on the neighboring influence. As used in [184 – 187] for vehicles systems, data-based methods offer a potential ability to predict and find patterns in data, which can be exploited to estimate neighboring behaviours. 46 Chapter 5. Conclusions and future research directions • Cyber-security and fault tolerance: The networked architecture of distributed systems and the need for reliable information sharing introduce further security concerns [188 – 190]. In this regard, faulty subsystems and attackers can distort the measurements and actuation signals that are transmitted between the physical system and the controllers, as well as the data that are communicated between agents. The latter threatens not only the level of performance obtained but can also bring stability issues [191,192]. Moreover, as mentioned in [193] for micro-grid applications, cyber-attackers can even exploit vulnerabilities enabling the introduction of malicious information to gain economic benefits. For these reasons, future works should also include the study of methods able to detect and mitigate the effect of faults and cyber-attacks in distributed and coalitional MPC schemes. A possibility to be explored in this regard is the formation of coalitions of healthy controllers to deal with these issues. List of Figures 1.1 Scheme of a distributed system composed of 5 subsystems 2 1.2 Examples of topologies and coalitions for a network with 5 agents 4 1.3 Structure of sparse feedback gains in the coalitional approaches [28,74–76] 6 3.1 Pie chart with the methodologies considered in [30–40] 19 3.2 Boundaries between topologies in two simulation examples in [32] 22 3.3 Scheme of the top-down approach for traffic freeways implemented in [34] 24 3.4 Evolution of the density of vehicles along the road simulated in [34] 25 3.5 State evolution of a 12-trucks system with the coalitional MPC for tracking presented in [35] 27 3.6 Evolution of the network topology in the simulation example of [35] 28 3.7 Comparison in terms of computation time between different methods for selecting coalitions [36] 29 3.8 Learning regret and percentage of optimal decisions per cycle with the MAB method described in [37] 31 3.9 Regions derived from the multi-class classification of the states in [37] 32 3.10 Comparison between the DMPC with NN-based warm start proposed in [38] and the corresponding centralized MPC 33 3.11 Sketch of the approach proposed in [39]. 34 3.12 Position trajectories obtained with the LMPC controller in [39] in each repetition of five different tasks. 35 3.13 Evolution of the costs of performing different tasks in [39] with the number of repetitions 36 47 Bibliography [1] R. Scattolini, “Architectures for distributed and hierarchical model predictive control–a review,” Journal of process control, vol. 19, no. 5, pp. 723–731, 2009. [2] D. Ding, Q.-L. Han, Z. Wang, and X. Ge, “A survey on model-based distributed control and filtering for industrial cyber-physical systems,” IEEE Transactions on Industrial Informatics, vol. 15, no. 5, pp. 2483–2499, 2019. [3] J. M. Guerrero, M. Chandorkar, T.-L. Lee, and P. C. Loh, “Advanced control architectures for intelligent microgrids—part i: Decentralized and hierarchical control,” IEEE Transactions on Industrial Electronics, vol. 60, no. 4, pp. 1254– 1262, 2012. [4] M. Yazdanian and A. Mehrizi-Sani, “Distributed control techniques in microgrids,” IEEE Transactions on Smart Grid, vol. 5, no. 6, pp. 2901–2909, 2014. [5] F. Bullo, J. Cort ´ es, and S. Martinez, Distributed control of robotic networks. Princeton University Press, 2009. [6] J. Z. Hern ´ andez, S. Ossowski, and A. Garcıa-Serrano, “Multiagent architectures for intelligent traffic management systems,” Transportation Research Part C: Emerging Technologies, vol. 10, no. 5-6, pp. 473–506, 2002. [7] M. Kordestani, A. A. Safavi, and M. Saif, “Recent survey of large-scale systems: Architectures, controller strategies, and industrial applications,” IEEE Systems Journal, 2021. [8] W. Qi, J. Liu, and P. D. Christofides, “A distributed control framework for smart grid development: Energy/water system optimal operation and electric grid integration,” Journal of Process Control, vol. 21, no. 10, pp. 1504–1516, 2011. [9] R. R. Negenborn, P.-J. van Overloop, T. Keviczky, and B. De Schutter, “Distributed model predictive control of irrigation canals,” Networks & Heterogeneous Media, vol. 4, no. 2, p. 359, 2009. [10] H. Hu, Y. Liu, and M. Zhou, “Maximally permissive distributed control of large scale automated manufacturing systems modeled with petri nets,” IEEE Transactions on Control Systems Technology, vol. 23, no. 5, pp. 2026–2034, 2015. 49 56 Bibliography [97] C. J. Ostafew, A. P. Schoellig, and T. D. Barfoot, “Robust constrained learning-based nmpc enabling reliable mobile robot path tracking,” The International Journal of Robotics Research, vol. 35, no. 13, pp. 1547–1563, 2016. [98] J. M. Manzano, D. Limon, D. M. de la Pe ˜ na, and J.-P. Calliess, “Robust learningbased mpc for nonlinear constrained systems,” Automatica, vol. 117, p. 108948, 2020. [99] I. Lenz, R. A. Knepper, and A. Saxena, “Deepmpc: Learning deep latent features for model predictive control.” in Robotics: Science and Systems. Rome, Italy, 2015. [100] A. Carron, E. Arcari, M. Wermelinger, L. Hewing, M. Hutter, and M. N. Zeilinger, “Data-driven model predictive control for trajectory tracking with a robotic arm,” IEEE Robotics and Automation Letters, vol. 4, no. 4, pp. 3758–3765, 2019. [101] U. Rosolia and F. Borrelli, “Learning model predictive control for iterative tasks. a data-driven control framework,” IEEE Transactions on Automatic Control, vol. 63, no. 7, pp. 1883–1896, 2017. [102] C. Shang and F. You, “A data-driven robust optimization approach to scenario-based stochastic model predictive control,” Journal of Process Control, vol. 75, pp. 24–39, 2019. [103] K. P. Wabersich and M. N. Zeilinger, “Linear model predictive safety certification for learning-based control,” in 2018 IEEE Conference on Decision and Control (CDC). IEEE, 2018, pp. 7130–7135. [104] ——, “Safe exploration of nonlinear dynamical systems: A predictive safety filter for reinforcement learning,” arXiv preprint arXiv:1812.05506, 2018. [105] ——, “Scalable synthesis of safety certificates from data with application to learningbased control,” in 2018 European Control Conference (ECC). IEEE, 2018, pp. 1691–1697. [106] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018. [107] B. Kiumarsi, K. G. Vamvoudakis, H. Modares, and F. L. Lewis, “Optimal and autonomous control using reinforcement learning: A survey,” IEEE transactions on neural networks and learning systems, vol. 29, no. 6, pp. 2042–2062, 2017. [108] U. Rosolia, A. Carvalho, and F. Borrelli, “Autonomous racing using learning model predictive control,” in 2017 American Control Conference (ACC). IEEE, 2017, pp. 5115–5120. [109] Y. R. St ¨ urz, E. L. Zhu, U. Rosolia, K. H. Johansson, and F. Borrelli, “Distributed learning model predictive control for linear systems,” in 2020 59th IEEE Conference on Decision and Control (CDC). IEEE, 2020, pp. 4366–4373. [110] E. L. Zhu, Y. R. St ¨ urz, U. Rosolia, and F. Borrelli, “Trajectory optimization for nonlinear multi-agent systems using decentralized learning model predictive control,” in 2020 59th IEEE Conference on Decision and Control (CDC). IEEE, 2020, pp. 6198–6203. [111] N. Scianca, U. Rosolia, and F. Borrelli, “Learning model predictive control for Bibliography 57 periodic repetitive tasks,” in 2020 European Control Conference (ECC). IEEE, 2020, pp. 29–34. [112] C. Vallon and F. Borrelli, “Task decomposition for iterative learning model predictive control,” in 2020 American Control Conference (ACC). IEEE, 2020, pp. 2024–2029. [113] B. M. Akesson and H. T. Toivonen, “A neural network model predictive controller,” Journal of Process Control, vol. 16, no. 9, pp. 937–946, 2006. [114] M. Hertneck, J. K ¨ ohler, S. Trimpe, and F. Allg ¨ ower, “Learning an approximate model predictive controller with guarantees,” IEEE Control Systems Letters, vol. 2, no. 3, pp. 543–548, 2018. [115] B. Karg and S. Lucia, “Efficient representation and approximation of model predictive control laws via deep learning,” IEEE Transactions on Cybernetics, vol. 50, no. 9, pp. 3866–3878, 2020. [116] J. A. Anderson, An introduction to neural networks. MIT press, 1995. [117] A. Alessio and A. Bemporad, “A survey on explicit model predictive control,” in Nonlinear model predictive control. Springer, 2009, pp. 345–369. [118] A. Chakrabarty, V. Dinh, M. J. Corless, A. E. Rundell, S. H. ˙ Zak, and G. T. Buzzard, “Support vector machine informed explicit nonlinear model predictive control using low-discrepancy sequences,” IEEE Transactions on Automatic Control, vol. 62, no. 1, pp. 135–148, 2016. [119] H. Zhang, L. Du, and J. Shen, “Hybrid mpc system for platoon based cooperative lane change control using machine learning aided distributed optimization,” Transportation Research Part B: Methodological, 2021. [120] Y. L ¨ ohr, M. Klau ˇ co, M. Fikar, and M. M ¨ onnigmann, “Machine learning assisted solutions of mixed integer mpc on embedded platforms,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 5195–5200, 2020. [121] Y. Vaupel, N. C. Hamacher, A. Caspari, A. Mhamdi, I. G. Kevrekidis, and A. Mitsos, “Accelerating nonlinear model predictive control through machine learning,” Journal of Process Control, vol. 92, pp. 261–270, 2020. [122] S. Chen, Z. Wu, D. Rincon, and P. D. Christofides, “Machine learning-based distributed model predictive control of nonlinear processes,” AIChE Journal, vol. 66, no. 11, p. e17013, 2020. [123] S. Muntwiler, K. P. Wabersich, A. Carron, and M. N. Zeilinger, “Distributed model predictive safety certification for learning-based control,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 5258–5265, 2020. [124] C. Conte, M. N. Zeilinger, M. Morari, and C. N. Jones, “Robust distributed model predictive control of linear systems,” in 2013 European Control Conference (ECC). IEEE, 2013, pp. 2764–2769. [125] V. Javalera, B. Morcego, and V. Puig, “Distributed mpc for large scale systems using agent-based reinforcement learning,” IFAC Proceedings Volumes, vol. 43, no. 8, pp. 597–602, 2010. 58 Bibliography [126] D. Chilin, J. Liu, J. F. Davis, and P. D. Christofides, “Data-based monitoring and reconfiguration of a distributed model predictive control system,” International Journal of Robust and Nonlinear Control, vol. 22, no. 1, pp. 68–88, 2012. [127] I. Arel, C. Liu, T. Urbanik, and A. G. Kohls, “Reinforcement learning-based multiagent system for network traffic signal control,” IET Intelligent Transport Systems, vol. 4, no. 2, pp. 128–135, 2010. [128] L.-N. Zhao, H.-J. Ma, L.-X. Xu, and X. Wang, “Observer-based adaptive sampleddata event-triggered distributed control for multi-agent systems,” IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 67, no. 1, pp. 97–101, 2019. [129] L. Ding, Q.-L. Han, and X.-M. Zhang, “Distributed secondary control for active power sharing and frequency regulation in islanded microgrids using an eventtriggered communication mechanism,” IEEE Transactions on Industrial Informatics, vol. 15, no. 7, pp. 3910–3922, 2019. [130] D. Limon, I. Alvarado, T. Alamo, and E. F. Camacho, “MPC for tracking piecewise constant references for constrained linear systems,” Automatica, vol. 44, no. 9, pp. 2382–2387, September 2008. [131] ——, “Robust tube-based MPC for tracking of constrained linear systems with additive disturbances,” Journal of Process Control, vol. 20, no. 3, pp. 248–260, March 2010. [132] A. Ferramosca, D. Limon, I. Alvarado, and E. F. Camacho, “Cooperative distributed MPC for tracking,” Automatica, vol. 49, no. 4, pp. 906–914, April 2013. [133] P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,” Machine learning, vol. 47, no. 2-3, pp. 235–256, 2002. [134] T. Lu, D. P ´ al, and M. P ´ al, “Contextual multi-armed bandits,” in Proceedings of the Thirteenth international conference on Artificial Intelligence and Statistics, 2010, pp. 485–492. [135] A. Hegyi, B. De Schutter, and H. Hellendoorn, “Model predictive control for optimal coordination of ramp metering and variable speed limits,” Transportation Research Part C: Emerging Technologies, vol. 13, no. 3, pp. 185–209, 2005. [136] R. C. Carlson, I. Papamichail, M. Papageorgiou, and A. Messmer, “Optimal motorway traffic flow control involving variable speed limits and ramp metering,” Transportation Science, vol. 44, no. 2, pp. 238–253, 2010. [137] A. Muralidharan and R. Horowitz, “Computationally efficient model predictive control of freeway networks,” Transportation Research Part C: Emerging Technologies, vol. 58, pp. 532 – 553, 2015, special Issue: Advanced Road Traffic Control. [138] J. G. Martin, R. A. Garc ´ ıa, and E. F. Camacho, “Event-milp-based task allocation for heterogeneous robotic sensor network for thermosolar plants,” Journal of Intelligent & Robotic Systems, vol. 102, no. 1, pp. 1–13, 2021. [139] J. Martin, J. Maestre, and E. Camacho, “Spatial irradiance estimation in a thermosolar power plant by a mobile robot sensor network,” Solar Energy, vol. 220, pp. 735–744, 2021. [140] P. Van Overloop, J. Maestre, A. D. Sadowska, E. F. Camacho, and B. De Schutter, Bibliography 59 “Human-in-the-loop model predictive control of an irrigation canal [applications of control],” IEEE Control Systems Magazine, vol. 35, no. 4, pp. 19–29, 2015. [141] J. M. Maestre, P.-J. van Overloop, M. Hashemy, A. Sadowska, and E. F. Camacho, “Human in the loop model predictive control: An irrigation canal case study,” in 53rd IEEE Conference on Decision and Control. IEEE, 2014, pp. 4881–4886. [142] M. I. Fern ´ andez, P. Chanfreut, I. Jurado, and J. M. Maestre, “A data-based model predictive decision support system for inventory management in hospitals,” IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 6, pp. 2227–2236, 2020. [143] S. Shibasaki, M. Inoue, M. Arahata, and V. Gupta, “Weak control approach to consumer-preferred energy management,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 17 083–17 088, 2020. [144] M. C. Priess, R. Conway, J. Choi, J. M. Popovich, and C. Radcliffe, “Solutions to the inverse lqr problem with application to biological systems analysis,” IEEE Transactions on control systems technology, vol. 23, no. 2, pp. 770–777, 2014. [145] K. Hara, M. Inoue, and J. M. Maestre, “Data-driven human modeling: Quantifying personal tendency toward laziness,” IEEE Control Systems Letters, vol. 5, no. 4, pp. 1219–1224, 2020. [146] A. Ramadan, J. Choi, C. J. Radcliffe, J. M. Popovich, and N. P. Reeves, “Inferring control intent during seated balance using inverse model predictive control,” IEEE robotics and automation letters, vol. 4, no. 2, pp. 224–230, 2018. [147] R. Guicherd, P. A. Trodden, A. R. Mills, and V. Kadirkamanathan, “Superviseddistributed control with joint performance and communication optimisation,” International Journal of Control, pp. 1–12, 2020. [148] S. Riverso, M. Farina, and G. Ferrari-Trecate, “Plug-and-play decentralized model predictive control for linear systems,” IEEE Transactions on Automatic Control, vol. 58, no. 10, pp. 2608–2614, 2013. [149] ——, “Plug-and-play model predictive control based on robust control invariant sets,” Automatica, vol. 50, no. 8, pp. 2179–2186, 2014. [150] S. Riverso and G. Ferrari-Trecate, “Plug-and-play distributed model predictive control with coupling attenuation,” Optimal Control Applications and Methods, vol. 36, no. 3, pp. 292–305, 2015. [151] M. N. Zeilinger, Y. Pu, S. Riverso, G. Ferrari-Trecate, and C. N. Jones, “Plug and play distributed model predictive control based on distributed invariance and optimization,” in 52nd IEEE conference on decision and control. IEEE, 2013, pp. 5770–5776. [152] S. Bansal, M. N. Zeilinger, and C. J. Tomlin, “Plug-and-play model predictive control for electric vehicle charging and voltage control in smart grids,” in 53rd IEEE Conference on Decision and Control. IEEE, 2014, pp. 5894–5900. [153] S. Lucia, M. K ¨ ogel, and R. Findeisen, “Contract-based predictive control of distributed systems with plug and play capabilities,” IFAC-PapersOnLine, vol. 48, no. 23, pp. 205–211, 2015. 60 Bibliography [154] J. Zhou, D. J. Burns, C. Danielson, and S. Di Cairano, “A reconfigurable plug-and- play model predictive controller for multi-evaporator vapor compression systems,” in 2016 American Control Conference (ACC). IEEE, 2016, pp. 2358–2364. [155] B. Hou, S. Li, and Y. Zheng, “Distributed model predictive control for reconfigurable systems with network connection,” IEEE Transactions on Automation Science and Engineering, 2021. [156] B. Ding, L. Ge, H. Pan, and P. Wang, “Distributed MPC for tracking and formation of homogeneous multi-agent system with time-varying communication topology,” Asian Journal of Control, vol. 18, no. 3, pp. 1030–1041, 2016. [157] L. Kang, W. Tang, Y. Liu, and P. Daoutidis, “Control configuration synthesis using agglomerative hierarchical clustering: A graph-theoretic approach,” Journal of Process Control, vol. 46, pp. 43–54, 2016. [158] Y. Zheng, Y. Wei, and S. Li, “Coupling degree clustering-based distributed model predictive control network design,” IEEE Transactions on Automation Science and Engineering, vol. 15, no. 4, pp. 1749–1758, 2018. [159] D. Bauso, “Aggregate energy production in wind farms via dynamic robust coalitional games,” IEEE Control Systems Letters, 2021. [160] M. Marzband, R. R. Ardeshiri, M. Moafi, and H. Uppal, “Distributed generation for economic benefit maximization through coalition formation–based game theory concept,” International Transactions on Electrical Energy Systems, vol. 27, no. 6, p. e2313, 2017. [161] E. Cotilla-Sanchez, P. D. Hines, C. Barrows, S. Blumsack, and M. Patel, “Multiattribute partitioning of power networks based on electrical distance,” IEEE Transactions on Power Systems, vol. 28, no. 4, pp. 4979–4987, 2013. [162] S. Siniscalchi-Minna, F. D. Bianchi, C. Ocampo-Martinez, J. L. Dom ´ ınguez-Garc ´ ıa, and B. De Schutter, “A non-centralized predictive control strategy for wind farm active power control: A wake-based partitioning approach,” Renewable Energy, vol. 150, pp. 656–669, 2020. [163] A. Jain, A. Chakrabortty, and E. Biyik, “Distributed wide-area control of power system oscillations under communication and actuation constraints,” Control Engineering Practice, vol. 74, pp. 132–143, 2018. [164] A. La Bella, P. Klaus, G. Ferrari-Trecate, and R. Scattolini, “Supervised model predictive control of large-scale electricity networks via clustering methods,” Optimal Control Applications and Methods, 2021. [165] Y. Li, C. Tang, K. Li, S. Peeta, X. He, and Y. Wang, “Nonlinear finite-time consensusbased connected vehicle platoon control under fixed and switching communication topologies,” Transportation Research Part C: Emerging Technologies, vol. 93, pp. 525–543, 2018. [166] G. Lou, W. Gu, Y. Xu, M. Cheng, and W. Liu, “Distributed MPC-based secondary voltage control scheme for autonomous droop-controlled microgrids,” IEEE transactions on sustainable energy, vol. 8, no. 2, pp. 792–804, 2016. [167] L. Han, T. Morstyn, and M. McCulloch, “Incentivizing prosumer coalitions with Bibliography 61 energy management using cooperative game theory,” IEEE Transactions on Power Systems, vol. 34, no. 1, pp. 303–313, 2018. [168] A. M. Jadhav, N. R. Patne, and J. M. Guerrero, “A novel approach to neighborhood fair energy trading in a distribution network of multiple microgrid clusters,” IEEE Transactions on Industrial Electronics, vol. 66, no. 2, pp. 1520–1531, 2018. [169] X. Cheng and J. M. Scherpen, “Clustering approach to model order reduction of power networks with distributed controllers,” Advances in Computational Mathematics, vol. 44, no. 6, pp. 1917–1939, 2018. [170] C. Ocampo-Martinez, D. Barcelli, V. Puig, and A. Bemporad, “Hierarchical and decentralised model predictive control of drinking water networks: Application to barcelona case study,” IET control theory & applications, vol. 6, no. 1, pp. 62–71, 2012. [171] Z. Zhou, B. De Schutter, S. Lin, and Y. Xi, “Two-level hierarchical model-based predictive control for large-scale urban traffic networks,” IEEE Transactions on Control Systems Technology, vol. 25, no. 2, pp. 496–508, 2016. [172] Y. Zheng, S. E. Li, J. Wang, D. Cao, and K. Li, “Stability and scalability of homogeneous vehicular platoon: Study on the influence of information flow topologies,” IEEE Transactions on intelligent transportation systems, vol. 17, no. 1, pp. 14–26, 2015. [173] M. Nayeripour, H. Fallahzadeh-Abarghouei, E. Waffenschmidt, and S. Hasanvand, “Coordinated online voltage management of distributed generation using network partitioning,” Electric Power Systems Research, vol. 141, pp. 202–209, 2016. [174] L. Che, X. Zhang, M. Shahidehpour, A. Alabdulwahab, and A. Abusorrah, “Optimal interconnection planning of community microgrids with renewable energy sources,” IEEE Transactions on Smart Grid, vol. 8, no. 3, pp. 1054–1063, 2015. [175] A. S ´ anchez, A. Gallego, J. Esca ˜ no, and E. Camacho, “Thermal balance of large scale parabolic trough plants: A case study,” Solar Energy, vol. 190, pp. 69–81, 2019. [176] D. Zhang, S. K. Nguang, and L. Yu, “Distributed control of large-scale networked control systems with communication constraints and topology switching,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 47, no. 7, pp. 1746– 1757, 2017. [177] A. Clark, B. Alomair, L. Bushnell, and R. Poovendran, Submodularity in dynamics and control of networked systems. Springer, 2016. [178] M. Farina and R. Scattolini, “Distributed predictive control: A non-cooperative algorithm with neighbor-to-neighbor communication for linear systems,” Automatica, vol. 48, no. 6, pp. 1088–1096, June 2012. [179] S. B. Chabane, C. S. Maniu, T. Alamo, E. F. Camacho, and D. Dumur, “A new approach for guaranteed ellipsoidal state estimation,” IFAC Proceedings Volumes, vol. 47, no. 3, pp. 6533–6538, 2014. [180] L. B. De Oliveira and E. Camponogara, “Multi-agent model predictive control of signaling split in urban traffic networks,” Transportation Research Part C: Emerging 62 Bibliography Technologies, vol. 18, no. 1, pp. 120–139, 2010. [181] A. Slivkins, “Contextual bandits with similarity information,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. 2533–2568, 2014. [182] A. S ´ anchez, A. Gallego, J. Esca ˜ no, and E. Camacho, “Temperature homogenization of a solar trough field for performance improvement,” Solar Energy, vol. 165, pp. 1–9, 2018. [183] E. Masero, J. Maestre, M. Francisco, and E. Camacho, “Coalitional MPC with predicted topology transitions,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 3342–3347, 2020. [184] S. Su, K. Muelling, J. Dolan, P. Palanisamy, and P. Mudalige, “Learning vehicle surrounding-aware lane-changing behavior from observed trajectories,” in 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018, pp. 1412–1417. [185] N. Deo and M. M. Trivedi, “Multi-modal trajectory prediction of surrounding vehicles with maneuver based LSTM,” in 2018 IEEE Intelligent Vehicles Symposium (IV), 2018, pp. 1179–1184. [186] Y. Guo, Q. Sun, R. Fu, and C. Wang, “Improved car-following strategy based on merging behavior prediction of adjacent vehicle from naturalistic driving data,” IEEE Access, vol. 7, pp. 44 258–44 268, 2019. [187] R. Song and B. Li, “Surrounding vehicles’ lane change maneuver prediction and detection for intelligent vehicles: A comprehensive review,” IEEE Transactions on Intelligent Transportation Systems, 2021. [188] T. Arauz, P. Chanfreut, and J. Maestre, “Cyber-security in networked and distributed model predictive control,” Annual Reviews in Control, 2021. [189] Z. Li, M. Shahidehpour, and F. Aminifar, “Cybersecurity in distributed power systems,” Proceedings of the IEEE, vol. 105, no. 7, pp. 1367–1388, 2017. [190] W. Wang and Z. Lu, “Cyber security in the smart grid: Survey and challenges,” Computer networks, vol. 57, no. 5, pp. 1344–1371, 2013. [191] A. Alipour-Fanid, M. Dabaghchian, and K. Zeng, “Impact of jamming attacks on vehicular cooperative adaptive cruise control systems,” IEEE transactions on vehicular technology, vol. 69, no. 11, pp. 12 679–12 693, 2020. [192] P. Srikantha and D. Kundur, “Denial of service attacks and mitigation for stability in cyber-enabled power grid,” in 2015 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT). IEEE, 2015, pp. 1–5. [193] Y. Mo, T. H.-J. Kim, K. Brancik, D. Dickinson, H. Lee, A. Perrig, and B. Sinopoli, “Cyber–physical security of a smart grid infrastructure,” Proceedings of the IEEE, vol. 100, no. 1, pp. 195–209, 2011. Acronyms BQP Binary Quadratic Program. DMPC Distributed Model Predictive Control. GA Genetic Algorithm. LMI Linear Matrix Inequality. LMPC Learning Model Predictive Control. LSS Large-Scale System. MAB Multi-Armed Bandit. MPC Model Predictive Control. NN Neural Network. TTS Total Time Spent. TWT Total Waiting Time. UTN Urban Traffic Network. VSL Variable Speed Limit. 63