An attention model for the formation of collectives in real-world domains

Artificial Intelligence 328 (2024) 104064 Available online 9 January 2024 0004-3702/© 2024 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/). Contents lists available at ScienceDirect Artificial Intelligence journal homepage: www.elsevier.com/locate/artint An attention model for the formation of collectives in real-world domains Adrià Fenoyb,a, Filippo Bistaffab,∗, Alessandro Farinelli a aUniversity of Verona, Italy bIIIA-CSIC, Spain A R T I C L E I N F O A B S T R A C T Keywords: Attention models Reinforcement learning Collective formation Optimization We consider the problem of forming collectives of agents inherent in application domains aligned with Sustainable Development Goals 4 and 11 (i.e., team formation and ridesharing, respectively). We propose a general solution approach based on a novel combination of an attention model and an integer linear program (ILP). In more detail, we propose an attention encoder-decoder model that transforms a collective formation instance to a weighted set packing problem, which is then solved by an ILP. Results on collective formation problems inherent in the ridesharing and team formation domains show that our approach provides comparable solutions (in terms of quality) to the ones produced by state-of-the-art approaches specific to each domain. Moreover, our solution outperforms the most recent general approach for forming collectives based on Monte Carlo tree search. 1. Introduction In recent years, more and more scenarios require Collective Intelligence solutions enabling novel ways of social production, promoting innovation, and encouraging the exchange of ideas [1]. Such new forms of collaborative consumption and production pose complex, multi-faceted challenges, but they ultimately all rely on a common and fundamental task, i.e., the formation of collectives. In this work, we consider two real-world application domains, i.e., ridesharing and team formation (respectively aligned with UN Sustainable Development Goals 11 and 4 [2]), where agents complete tasks and achieve benefits through the formation of collectives to achieve cooperation. On the one hand, in ridesharing commuters can form groups and travel together with the objective of reducing transportation costs, mitigating pollutant emissions, and alleviating traffic congestion in urban environments [3,4]. Another prominent example can be found in modern educational institutions that aim at implementing cooperative and active learning techniques, which engage students in teams to participate in all learning activities in the classrooms. As recently shown by Andrejczuk et al. [5], collective formation approaches can improve the overall performance of the students by grouping them in teams that maximize the synergies among members. Due to the inherent complexity and specificity of each application domain, researchers usually tackle the formation of collectives by designing very specific sub-optimal approaches that can solve the associated large-scale optimization problem in a feasible runtime. However, unfortunately, one domain-specific approach usually cannot be applied in a different scenario. * Corresponding author. E-mail address: filippo.bistaff[email protected] (F. Bistaffa). https://doi.org/10.1016/j.artint.2023.104064 Received 22 June 2022; Received in revised form 20 December 2023; Accepted 28 December 2023 Artificial Intelligence 328 (2024) 104064 2 A. Fenoy, F. Bistaffa and A. Farinelli In contrast, in this paper, we propose a novel, general approach for the formation of collectives that is based on two fundamental steps. First, we apply deep reinforcement learning techniques to train an attention encoder-decoder model, with the objective of automatically generating a set of promising collectives based on the structure of the considered scenario. In the second step, we compile a weighted set packing (WSP) instance that, by only taking into account the promising candidates generated in the first step, can be solved by off-the-shelf ILP solvers in a manageable time budget. Thus, our approach does not require manually specifying any domain-specific knowledge, in contrast with the above-mentioned sub-optimal state-of-the-art approaches. Furthermore, by only considering a set of promising candidates rather than the entire set of possible collectives,1we reduce the complexity of the original problem by several orders of magnitude while producing a high-quality solution. As such, this paper advances the state-of-the-art as follows: •We propose a general approach for the formation of collectives in real-world domains based on the novel combination of an attention model and WSP formulation. •We proposed a novel training procedure for our attention model based on Maximum Entropy Reinforcement Learning. In contrast to previous approaches which use attention-based models for optimization [6], our solution achieves a wide variety of promising candidates. Such variety is a key feature that allows the ILP solver to compute a high-quality solution to the collective formation problem. •We evaluate our approach on collective formation problems inherent in two real-world domains (i.e., ridesharing and team formation) by comparing it with state-of-the-art approaches specific to each domain. Our results show that our approach can produce solutions of comparable quality without requiring any domain-specific knowledge. Moreover, we compare our approach with the most recent general approach for forming collectives based on Monte Carlo tree search (MCTS) [7],2showing that our solutions outperform (in terms of quality) the ones computed by the counterpart. 2. Background & related work In this section, we discuss the necessary background and the relevant literature on the formation of collectives. We then elaborate on previous attempts at using machine learning techniques to solve combinatorial optimization problems. 2.1. Formation of collectives of agents The problem of forming collectives of agents has been deeply studied from many different perspectives in the scientific literature. Depending on the context and the application domain, collectives of agents are also referred to as coalitions [8]or teams [5,9]of agents. Here we adopt the term “collective” to refer to the general concept of a “group” of agents that cooperate to complete tasks or obtain benefits, as we deem it more general and intuitive. More in particular, in this paper we focus on the optimization problem [10]of computing the best set of non-overlapping collectives (i.e., subsets) of agents belonging to a universal set 𝐴, to maximize the total value provided by a domain-specific utility function, e.g., the reduction in terms of cost or CO2emissions associated to the arrangement of a shared trip [3]or the improvement thanks to cooperation within a team of students [5]. Formally, we consider a set of 𝑛agents 𝐴 ={𝑎1, 𝑎2, … , 𝑎𝑛}and a utility function 𝑓∶(𝐴) →ℝ(also referred to as characteristic function) that maps every collective in the feasible set3of collectives (𝐴)to a real number. We formulate the formation of collectives as the problem of computing the best set ∗of non-overlapping subsets of 𝐴(also referred to as a coalition structure [8]) that maximizes the sum of the values associated to each collective 𝐶∈∗, i.e., ∗=argmax ∈∏(𝐴)∑ 𝐶∈ 𝑓(𝐶),(1) where ∏(𝐴)is the set of all partitions of 𝐴into non-overlapping feasible subsets. 2.1.1. Complete approaches By and large, the formation of collectives requires to solve a coalition structure generation (CSG) problem [13,8]or, equivalently, a set partitioning problem [14]. A wealth of complete approaches have been proposed to solve Equation (1)to optimality [13], depending on the properties of the utility function 𝑓. Complete CSG algorithms [15,16] usually make no assumptions on the utility function, which is treated as a black-box oracle. Unfortunately, the mere act of providing the input to the solution algorithm (without even considering the runtime of the CSG algorithm itself) requires enumerating a number of values that grow exponentially with the number of agents. 1The number of possible collectives grows exponentially with the number of agents, hence it is not manageable in real-world applications that involve more than a few tens of agents. 2By “general approach” here we mean an approach that can be applied across different domains without significant changes, such as MCTS in this case. 3Depending on the considered domain, such a set of feasible collectives can be the entire set of subsets of 𝐴or, for example, the set of all collectives that satisfy a given constraint (e.g., cardinality constraints [11]or graph-based constraints [12]), as explained in Section 2.1.2. Artificial Intelligence 328 (2024) 104064 3 A. Fenoy, F. Bistaffa and A. Farinelli For this reason, complete unconstrained CSG algorithms are limited to only 25–30 agents, i.e., a scale that is not sufficient for realistic applications involving hundreds of agents (such as the ones we consider in this paper), hence we will not consider these approaches as benchmarks in our experimental evaluation. 2.1.2. Approaches for constrained scenarios In some application domains, it is possible to exploit specific properties to improve the runtime of the solution algorithm by considering constraints that reduce the number of feasible collectives. Cardinality constraints that limit the maximum size of the collectives to 𝑘naturally arise in many realistic scenarios [3,5,11]. In addition, a strand of literature pioneered by Myerson [12]has investigated graph-restricted scenarios [17–20]where collectives can be formed only if they induce a connected subgraph of an initial graph defined over the set of agents (e.g., a social network). Despite considering these constraints can significantly reduce the number of total collectives (up to a polynomial number (|𝐴| 𝑘)= 𝑂(|𝐴|𝑘)if collectives are restricted to a maximum cardinality of 𝑘agents [11]), such a number remains prohibitively large for realistic applications involving hundreds of agents. Indeed, as observed by Bistaffa et al. [3]and Andrejczuk et al. [5], enumerating and computing the utility value for all the collectives of size up to 5can require hours, especially when the computation of the utility function is particularly demanding (e.g., in team formation it requires to solve a small task assignment problem [5]). Along these lines, the authors of [3,5]concluded that complete algorithms were not a viable solution for real-world scenarios—even constrained ones—, resorting to domain-specific approaches that can compute sub-optimal solutions of good quality in a manageable amount of time (see Section 2.1.5). 2.1.3. Approaches focusing on specific function representations Another strand of literature [21–23]has focused on alternative utility function representations, which allow one to reduce the computational complexity of the CSG problem by exploiting specific properties of the adopted representation. For example, Ieong and Shoham [21] proposed a concise representation called marginal contribution nets, or MC-nets, where the calculation of the utility is based on a collection of rules. Tran-Thanh et al. [22] proposed a representation called coalitional skill vector model, where there is a set of skills in the system, and each agent has a skill vector (a vector consisting of values that reflect the agents’ level in different skills). More recently, Bistaffa et al. [23]focused on the well-known induced subgraph game (ISG) representation originally introduced by Deng and Papadimitriou [24]and proposed a CSG algorithm based on graph-clustering that exploits the succinctness of the representation. By definition, these approaches can only be applied if the utility function 𝑓of the collective formation domain satisfies some specific properties (e.g., it can be represented as a combination of rules in the case of MC-nets, or it can be represented by a sum of the weights of the graph in case of ISGs). Unfortunately, these specific properties rarely hold in real-world application domains. Indeed, neither of the two considered real-world collective formation domains (i.e., ridesharing and team formation) can be modeled as one of the above-mentioned function representations. In contrast, our work goes into the opposite research direction, i.e., obtaining a general approach for collective formation that does not rely on any specific property. For this reason, we will not compare against these approaches in our experimental evaluation. 2.1.4. Team formation approaches Collective formation has also been widely studied in the context of team formation, in which such a problem has been studied from different perspectives. For instance, Gaston and desJardins [25]focused only on local optimization without considering any concept of global optimal solution, proposing a heuristic to modify the graph connecting the agents based on local autonomous reasoning. Lappas et al. [26]studied the complexity of finding a single group of agents who possess a given set of skills to minimize the communication cost within such a group, and proposed a heuristic algorithm to solve such a problem. Marcolino et al. [27]focused on forming a single group of agents that has the maximum strength in the set of world states, showing that a diverse team can outperform a team formed by uniform members, and proposing optimal voting rules for such a diverse team. Finally, Liemhetcharat and Veloso [28] tackled the task of modeling the values of the utility function based on observations, without considering any partitioning problem on top of it. Here we focus on the optimization problem of forming disjoint teams with the objective of maximizing the sum of the corresponding utility values. In this context, Andrejczuk et al. [5] proposed a local-search algorithm named SynTeam that heavily relies on the structure of the problem and the considered dataset to form proficient teams that are all assigned the same task (e.g., an English proficiency task, an arts and design task, etc.). This local-search approach was later extended by Georgara et al. [29,30]to account for multiple tasks. Präntare and Heintz [9], on the other hand, proposed an optimal solution algorithm for the same optimization problem, which involves solving a CSG and a task assignment problem at the same time. Since we consider the team formation scenario involving one single task, we only consider SynTeam [5]as a competitor among the above-discussed team formation approaches. 2.1.5. Heuristic approaches To overcome the scalability limitations discussed in previous sections, the formation of collectives in real-world domains is usually tackled by means of sub-optimal approaches that trade generality for scalability, i.e., that exploit the specific structure of the considered domain to compute good-quality solutions in a feasible amount of time. Note that, in this case, the domain knowledge that the approaches exploit is not necessarily related to the characteristic function representation (in contrast with the approaches discussed in Section 2.1.3), rather it is related to specific properties of the application domain. Artificial Intelligence 328 (2024) 104064 4 A. Fenoy, F. Bistaffa and A. Farinelli For instance, Bistaffa et al. [3] proposed a solution algorithm for large-scale ridesharing that, by heavily relying on the greedy nature of the domain, is capable of computing solutions of very good quality for hundreds of agents within one minute. Previously, Farinelli et al. [31] proposed an approach based on hierarchical clustering that also relies on a greedy heuristic to identify the most promising couple of coalitions that can be merged, until no beneficial merge can be executed. Unfortunately, these approaches cannot be applied in collective formation domains that are not characterized by such a greedy nature, e.g., the team formation domain discussed in [5]. More recently, Wu and Ramchurn [7] proposed a CSG solution algorithm based on MCTS that can be used in any collective formation domain, including ridesharing and team formation. Nonetheless, such a MCTS approach employs a simulation policy based on a greedy heuristic similar to the one proposed by Farinelli et al. [31]. Indeed, the authors report good performance on synthetic datasets that are characterized by a greedy nature.4 Along these lines, in our experimental evaluation, we compare against the approaches by Wu and Ramchurn [7]and by Bistaffa et al. [3]. 2.2. Machine learning for optimization The use of machine learning techniques to solve combinatorial optimization problems is a recent yet very active topic that has received a lot of attention during the last few years. According to Bengio et al. [32], machine learning can contribute to the optimization field in twofold ways: i) replace some heavy computations by building fast approximations, and ii) improve the optimization approach by learning domain-specific structure. Due to the wide diversity among ways of combining machine learning and combinatorial optimization, Bengio et al. [32] classify the different approaches along two axes. Along the first axis, depending on the structure of the overall approach, Bengio et al. [32] identifies two alternatives: i) end-to-end machine learning approaches that are able to directly construct a solution for optimization problems, and ii) mixed approaches that use machine learning as a subroutine of a classical optimization approach, either as a preprocessing step or used alongside the classical approach in an online scheme. On the other hand, the second axis concerns the adopted learning methodology, i.e., supervised, unsupervised, or reinforcement learning. With respect to this classification, in this paper we propose a mixed approach, where an attention model generates high-valued candidate collectives, which are then encoded as variables in an ILP. Moreover, the attention model is trained by means of reinforcement learning to learn the domain-specific structure of the application domain. Along these lines, in what follows we first provide an overview of the relevant literature on end-to-end approaches, and then we focus on mixed approaches. Afterward, we discuss approaches trained with different learning methods. 2.2.1. End-to-end approaches for optimization There is a wide variety of end-to-end approaches to optimization, such as Pointer Networks by Vinyals et al. [33], Graph Convolutional Networks by Joshi et al. [34], or Attention mechanisms by Kool et al. [6]. These methods have been applied to various classic problems in Operations Research and Optimization, such as the well-known traveling salesperson problem (TSP). Moreover, other end-to-end approaches have been devised for variants of the vehicle routing problem (VRP), such as the capacitated VRP [35] or the online capacitated VRP [36]. It is important to note that, while our attention mechanism is inspired5by the one by Kool et al. [6]—an end-to-end approach—, our overall approach is instead a mixed one, as mentioned in Section 2.2. More specifically, our attention model does not compute the final solution to the considered problem (i.e., the formation of collectives), but rather a set of candidate collectives that constitute input of the ILP, which then computes the final solution. This combination of Machine Learning and Optimization makes our approach “mixed”. In what follows, we discuss other relevant works in the category of mixed approaches. 2.2.2. Mixed approaches for optimization Mixed approaches aim at combining machine learning with classic optimization procedures. In this line of research, a strand of literature focuses on using machine learning as a preprocessing step before a classical optimization approach. In particular, the approach proposed by Ding et al. [37]solves 8 different classical problems, including TSP and VRP, by means of Graph Convolutional Neural Networks which predict the value of binary variables in a mixed-integer linear program (MILP) formulation of these problems. The solution is then found by employing a branch and bound approach which uses the values to guide the search. In the same way, Li et al. [38] tackle benchmark satisfiability and problems related to social networks by means of Graph Convolutional Neural Networks to select the nodes in a graph that are likely to appear in an optimal solution and then solve the problem for the reduced graph. Similar to neural network-based approaches, support vector machines (SVMs) also can play the role of a heuristic approach as a preprocessing step for optimization problems. Examples of this are the work of Xavier et al. [39], which uses k-Nearest Neighbors in addition to SVMs to significantly reduce the problem size for Security-Constrained Unit Commitment problem in power systems and electricity markets. Another example is the work of Sun et al. [40], which uses SVMs to find a reduction of the graph for the maximum 4According to the methodology reported in [7], the value of a coalition 𝐶is correlated with its cardinality |𝐶|, hence forming bigger coalitions is, on average, more beneficial. 5We focus on attention models because recent work [6] shows that they can significantly outperform Pointer Networks on routing problems. Moreover, since inputs to our model are sets of agents (which are permutation invariant), we prefer the use of an attention mechanism, whose output is guaranteed to be invariant to permutations of the input set. Artificial Intelligence 328 (2024) 104064 5 A. Fenoy, F. Bistaffa and A. Farinelli weight clique problem. Many of the approaches that use machine learning as a preprocessing step focus on reducing the optimization problem (i.e., making the problem smaller and computationally tractable). Despite problem reduction with machine learning does not ensure any optimality guarantee, these approaches can usually provide high-quality solutions thanks to the “backbone” structure of the optimization problems they tackle (according to the terminology adopted by [41]), i.e., optimal solution and high-quality solutions are likely to share a specific structure. Our work aims at following a similar problem reduction strategy by lowering the number of variables in an ILP formulation for the collective formation problem, but in contrast with the above-mentioned works (which build the entire problem instance and then reduces it), we directly generate the reduced ILP by generating collectives which are likely to be in the optimal solution. Another family of mixed approaches is the one that aims at integrating machine learning within a classical approach for optimization. For example, Hottung and Tierney [42]use an attention model to reconstruct solutions inside a Large Neighborhood Search setting for the Capacitated VRP. Another example is the work of Hottung and Tierney [42], in which neural networks are used as a heuristic to guide search inside a tree search approach for the container pre-marshaling problem. In this respect, our approach is similar to the one proposed by Bistaffa et al. [3], which uses a domain-specific heuristic as a preprocessing step for the formation of candidate collectives and then computes the final solution employing an ILP. However, instead of using an ad-hoc heuristic designed for a specific domain, we introduce an attention model that learns the domain-specific structure of a problem. Therefore, by not relying on any domain-specific component, our approach can be applied to structurally different collective formation problems. 2.2.3. Learning methods Depending on the adopted learning methodology, an approach can be classified as supervised, unsupervised, or based on reinforcement learning. Several approaches adopt supervised learning strategies to imitate the outputs of an already existing solution algorithm, hence, by following Bengio et al. [32]’s terminology, “replacing it by its ML approximation”. For example, the abovediscussed work by Li et al. [38]makes use of supervised learning to train Graph Convolutional Neural Networks for Maximal Independent Set, Minimum Vertex Cover, Maximal Clique, and SAT. Khalil et al. [43] propose an approach to learn from a precomputed dataset a ranking function which is then used as a branching heuristic for MILP. Gasse et al. [44]and Nair et al. [45]also aim at constructing a branching heuristic for solving MILP, but in their case, they directly learn to imitate the decisions made by an expert employing a cross-entropy loss. The majority of these approaches aim at selecting an option among several alternatives (e.g., variable selection in branch-and-bound approaches), some approaches use machine learning to decide if an action (which is usually very demanding from a computational point of view) needs to be performed or not. This is the case of the work by Kruber et al. [46]where they use a supervised learning approach to decide whether a Dantzig-Wolfe decomposition should be applied to a MIP instance to solve it faster. Unsupervised learning tries to capture patterns appearing in unlabeled data. Since its purpose does not pursue finding fast approximations and it is not obvious in which way it could improve the solution quality of current approaches, its use when combining machine learning with optimization is not very common. To the best of our knowledge, the only approach that applies an unsupervised machine learning technique to optimization is the work by Karalias and Loukas [47], where a probabilistic loss is proposed to train a Graph Neural Network for the Graph partitioning and maximum clique problems. Finally, reinforcement learning (RL) aims at learning a policy (i.e., a system that determines a course of action) that optimizes the sum of future expected rewards observed from the actions performed by the policy. The policy is usually modeled inside a Markov Decision Process and interacts with the environment by executing an action and receiving an observation and a reward signal. The use of RL is appealing in the context of mixing machine learning with optimization because classical algorithms used for this purpose usually involve sequential decisions, which can be modeled as a Markov Decision Process. Moreover, in contrast with supervised learning, RL does not require other approaches to learn from, since it learns purely from experience. Several approaches aim at replacing supervised learning with RL in the context of combinatorial optimization. For example, the work by Bello et al. [48]and the aforementioned work by Nazari et al. [35] which proposes RL as an alternative way to train Pointer Networks, previously trained with supervised learning [33]. Related to our work, the approach by Kool et al. [6]employs the REINFORCE algorithm, a RL approach introduced by Williams [49], to train an attention-based model for the Traveling Salesperson Problem and the VRP. Along these lines, we adopt RL instead of other learning approaches, since we aim at achieving a general approach that is not limited by the quality of the examples used in a supervised environment. Because of the importance of the REINFORCE algorithm, which constitutes the training framework of our attention-based model, we devote the following section to discussing this algorithm in more detail. 2.3. The REINFORCE algorithm The REINFORCE algorithm proposed by Williams [49]constitutes one of the pillars of RL and has inspired many modern RL approaches. The general idea of the algorithm is to update the parameters of a model utilizing gradient methods in the direction that reinforces actions with higher rewards. At a general level, a RL setup is characterized by a sequence of states, actions, and rewards, i.e., (𝑠1, 𝑎1, 𝑟1, 𝑠2, 𝑎2, 𝑟2, … , 𝑠𝐻, 𝑎𝐻, 𝑟𝐻). An agent in this setup is modeled by a parameterized policy 𝜋𝜽and it decides on the actions that are performed according to the observed states. Moreover, in the RL setup, the agent receives a reward as feedback from its actions, which reinforces actions leading to a desired behavior. The reward is defined according to the optimization goal. In practice, when considering RL for combinatorial optimization, the reward is assigned to be the utility function of the optimization domain. However, this reward is usually not defined for intermediate states. In such situations, an expected return 𝐺𝑡is used instead, which can be computed from the probabilities Artificial Intelligence 328 (2024) 104064 6 A. Fenoy, F. Bistaffa and A. Farinelli determined by the policy at each state or by estimating it through a simulated trajectory referred to as a rollout. In this work, we employ the rollout technique to estimate the expected return of intermediate states corresponding to incomplete coalitions. To update the policy parameters 𝜽, Williams [49] proposes to use gradient methods, which update the parameters in the direction that maximizes the expected return with 𝜽←𝜽+𝛼𝐺𝑡∇𝜽log𝜋𝜽,(2) where 𝛼is the size of the learning step. In this work, we adopt a technique from actor-critic approaches [50], which introduces a baseline to reduce the variance in the expected return. This technique consists of employing the increment of the expected return with respect to a baseline 𝑏𝑡, which will be defined in the following sections. Then, the update rule becomes 𝜽←𝜽+𝛼(𝐺𝑡−𝑏𝑡)∇𝜽log𝜋𝜽.(3) Previous work employs REINFORCE to train a policy to produce solutions to combinatorial optimization problems [48,6], such as the TSP and the VRP. For such applications, the characteristic function of the problem is used as a reward to guide the models towards generating solutions of continuously improving quality until they converge to close-to-optimal solutions. Here we employ the REINFORCE algorithm to train a policy to generate collectives of high quality. Thus, the quality of a collective, determined by the utility function of a specific collective formation domain, is used as the reward. Very recently, the use of entropy within REINFORCE has been independently proposed by the researchers of the team Ratel during the AI4TSP competition [51]. More specifically, Ratel’s approach employs entropy regularization “to enhance exploration” [51, Section 4.2.2] within RL for the solution of TSP. Similarly, in our collective formation approach, entropy is used to increase the diversity of the generated candidates (via enhanced exploration), as we discuss in Section 3.1.4. Notice that, in contrast to Ratel’s approach, we do not use machine learning to directly provide a solution to the considered optimization problem, but to provide a set of diverse candidate solutions that are then processed by a classical optimization approach (i.e., an ILP). After having discussed the RL setup, which will be part of our approach, we now proceed to present our solution approach for optimization problems involving the formation of collectives. 3. Our solution approach As seen in previous sections, collective formation approaches that aim at considering the entire set of possible collectives cannot be applied to realistic application domains since such a set is prohibitively large to enumerate. In this respect, it is crucial to avoid the generation of such an impractically large problem instance in the first place since it would be impossible to handle for any solution algorithm. Indeed, our approach follows this rationale and works by breaking the problem into two parts. First, by means of an attention model, we generate candidate collectives, one at a time, to construct a set of candidate collectives. Second, we encode these collectives into an ILP, which computes the final solution. To accommodate the discussion of our approach, we first reformulate the optimization problem in Equation (1)as an ILP: maximize ∑ 𝐶∈(𝐴) 𝑓(𝐶)⋅𝑥𝐶, subject to ∑ 𝐶∈(𝐴) 𝑏𝑖,𝐶 ⋅𝑥𝐶≤1,∀𝑎𝑖∈𝐴, (4) where 𝑥𝐶is a binary decision variable that encodes whether collective 𝐶is in the set and 𝑏𝑖,𝐶 is a binary value that encodes whether agent 𝑎𝑖∈𝐴belongs to the collective 𝐶. Notice that the ILP formulation in Equation (4) only considers the constraint that collectives must be non-overlapping. Depending on the considered application domain, it is also possible to impose that the formed collectives cover the entire set 𝐴by enforcing ∑ 𝐶∈(𝐴) 𝑏𝑖,𝐶 ⋅𝑥𝐶=1,∀𝑎𝑖∈𝐴. (5) The problem in Equation (4) can be easily recognized as a WSP, one of the original 21 Karp’s NP-complete problems [52]. Because of the computational complexity of such problems, the ILP can only be solved for small instances (i.e., 𝐴with less than a couple of tens of agents) by employing off-the-shelf solvers. On the other hand, in real-world scenarios, the generation of such an ILP (let alone its solution) can require hours of computation due to the necessity of enumerating all feasible collectives. This complexity is further increased in scenarios where determining each value 𝑓(𝐶)requires a significant computational effort, such as the synergistic value proposed by Andrejczuk et al. [5]. On the other hand, by exploiting the inherent structure of the domain, an expert might propose a reduced set of promising collectives (𝐴), from which a sub-optimal solution of high quality can be obtained. Following this approach, in this paper we propose an attention-based model that learns this structure to generate an ILP of manageable size, i.e., maximize ∑ 𝐶∈(𝐴) 𝑓(𝐶)⋅𝑥𝐶, subject to ∑ 𝐶∈(𝐴) 𝑏𝑖,𝐶 ⋅𝑥𝐶≤1,∀𝑎𝑖∈𝐴. (6) Artificial Intelligence 328 (2024) 104064 7 A. Fenoy, F. Bistaffa and A. Farinelli Fig. 1. Proposed approach for the formation of collectives. The attention model generates a reduced set of collectives from which an ILP solver computes the solution. Fig. 1illustrates the two-step approach we propose for the formation of collectives. In the first step, the attention model produces collectives, one by one, to build a set (𝐴)of candidate collectives. In the second step, Equation (6)is solved with the set (𝐴)as input to solve the collective formation problem. The attention model will be discussed in the following sections. Algorithm 1 Pseudocode of our approach for the formation of collectives. Input: set of agents 𝐴, overall time budget 𝑡 ∈ℝ+, generation portion 𝑘 ∈[0, 1] Output: the computed set of collectives 1: (𝐴) ←∅{Initialize empty set of candidates} 2: repeat 3: 𝐴′←𝐴{Local copy of input set of agents} 4: while |𝐴′| >0do 5: 𝑆←subset of 𝐴′generated by means of our attention model 6: (𝐴) ←(𝐴) ∪𝑆{Add 𝑆to the set of candidates} 7: 𝐴′←𝐴′⧵𝑆{Remove 𝑆from the local set of agents} 8: end while 9: until the time budget 𝑡 ⋅𝑘expires 10: Formulate the model in Equation (6)given (𝐴)computed in Lines 2–9 11: Solve such a model with an ILP solver given a time budget of (1 −𝑘) ⋅𝑡 Algorithm 1provides the pseudocode of our general approach for the formation of collectives. Following a standard practice [3], we assume that our entire approach is provided with a time budget 𝑡and we distribute such a time budget between the two phases illustrated in Fig. 1. More specifically, we devote a time budget of 𝑘 ⋅𝑡(with 𝑘 ∈[0, 1]) to the first phase, in which we generate the reduced set (𝐴)by repeatedly generating a collective 𝑆with our attention model and removing it from the initial set 𝐴until such a set has been entirely consumed. If the generation time budget has not been exhausted, we repeat the same procedure with another copy of the initial set. The remaining part (1 −𝑘) ⋅𝑡is devoted to the solution of the just-computed reduced ILP model in Equation (6), by providing such a time budget to the off-the-shelf ILP solver. In our experiments in Section 4we compute the value of the parameter 𝑘via IRACE [53], a widely used software for tuning algorithmic parameters. 3.1. Attention model Our attention model implements a decision-making process where collectives are built incrementally by selecting elements from the set of agents 𝐴, one at every decision step, and adding them to the collective 𝐶. During such a decision-making process, these two fundamental pieces of information (i.e., 𝐴and 𝐶) are internally maintained as the state of the model. Fig. 2illustrates an example of the process of forming a collective by means of the attention-based model in a ridesharing application domain. Initially, an empty collective and the set 𝐴of agents are provided as input to the model, which outputs a vector of probabilities, one for each agent plus one for the “end-of-sequence” (𝑒𝑜𝑠) token (indicating that no agent is selected and terminating the process of building the current collective). The probabilities, indicating the best agent to be selected, are used to sample one agent and add it to the collective. The collective is then updated, and a new iteration starts. A mask that forces the probability of selected agents to zero is used to avoid one agent being selected twice. The process finishes when the 𝑒𝑜𝑠 token is sampled. More formally, the internal state 𝑠of the model is represented by the tuple 𝑠 =(𝐴, 𝐶), where 𝐴is the set of agents and 𝐶is the collective currently being built. With a small abuse of notation,6our model receives the set of agents 𝐴as a list of 𝑑𝑎dimensional feature vectors, where 𝑑𝑎is the number of features. Following standard practice [54], our model assumes that an element (in our case an agent) can be represented as a vector of features (e.g., origin and destination locations in the ridesharing scenario, or students’ personality traits and competence levels in the team formation one). The model also receives a binary encoding of a collective 𝐶={𝑏1,𝐶 , 𝑏2,𝐶 , … , 𝑏𝑛,𝐶 }, where 𝑏𝑖,𝐶 are binary values determining whether the respective agents 𝑎𝑖are in the collective or not. Given a state 𝑠, we design an attention-based encoder-decoder model based on the one proposed by [6], which defines a stochastic policy 𝜋𝜽(𝑠)parameterized by a set of learnable parameters 𝜽representing the weights and biases of a neural network) that determine the probability for each element in the pool of agents 𝐴to be included in the collective 𝐶. Our encoder produces an embedding, i.e., a continuous representation of the input, for each element in the pool. Then, as illustrated in Fig. 3, the decoder receives the embedding and the collective to compute the probabilities. 3.1.1. Multi-head attention The main block of our encoder-decoder model is based on the attention mechanism by Vaswani et al. [55]. Within our approach, attention works as a mapping where the inputs are, following the standard nomenclature [55,6], the queries 𝑄and the keys 𝐾repre6Thus far, we considered 𝐴and 𝐶to be sets of agents. Now we redefine them respectively as a list of vectors and a list of binary variables. Artificial Intelligence 328 (2024) 104064 8 A. Fenoy, F. Bistaffa and A. Farinelli Fig. 2. An illustrative example of the process carried out by the attention model to form a collective in a ridesharing environment. In this example concerning the ridesharing domain, three agents (1: red, 2: green, and 3: blue) are the input of the model, together with the current collective, which is initialized empty. The collective is represented on a map to show the spatial relations between the origin (squares) and destination (triangles) locations. The model outputs a vector of probabilities, one for each agent plus one for the “end-of-sequence” (𝑒𝑜𝑠) token, which stops the formation of the collective. During the selection process, agents that have already been selected are masked out (indicated in gray in this example). Best viewed in colors. (For interpretation of the colors in the figure(s), the reader is referred to the web version of this article.) Fig. 3. General scheme of the encoder-decoder approach that computes the probability 𝜋𝜽for each agent in 𝐴to be added to the collective 𝐶. The sizes for input, output, and intermediate hidden states are specified for each element. The specific details of the probability computation are provided in Section 3.1.1. sented by a set of 𝑑𝑞and 𝑑𝑘dimensional vectors respectively. The outputs are the attention weights 𝑎𝑖𝑗 , which reflect the normalized compatibility of the query 𝒒𝑖with the key 𝒌𝑖. The first step to obtain the attention weights is to compute the compatibilities 𝑢𝑖𝑗 =(𝑊𝑞𝒒𝑖)𝑇(𝑊𝑘𝒌𝑗) √𝑑𝑞 ,(7) where 𝑊𝑞and 𝑊𝑘are two learnable linear transformations, and the output is scaled with a factor of 1 √𝑑𝑞 . The attention weights are then obtained by normalizing the compatibilities with a softmax: 𝑎𝑖𝑗 =𝑒𝑢𝑖𝑗 ∑𝑗′𝑒𝑢𝑖𝑗′.(8) Artificial Intelligence 328 (2024) 104064 9 A. Fenoy, F. Bistaffa and A. Farinelli Fig. 4. General scheme of our encoder architecture. Similar to the one proposed by Vaswani et al. [55], it combines several steps of multi-head attention, feedforward layers, layer normalization, and residual connections. The encoding step is repeated 𝑁times. These attention weights already capture the desired information (compatibility between queries and keys) and can already be used for a given purpose, as we do in the last step of the decoder, where we interpret these attention weights as probabilities for elements in 𝐴to be included in 𝐶; that is, the higher the compatibilities, the higher the probabilities will be. Nevertheless, the main use of the attention weights is to update a set of values 𝑉represented by a set of 𝑑𝑣dimensional vectors by computing a linear combination of the values weighted by the attention weights 𝒗′ 𝑖=∑ 𝑗 𝑎𝑖𝑗(𝑊𝑣𝒗𝑗),(9) where 𝑊𝑣is again a learnable linear transformation. In practice, according to the multi-head attention approach, it is beneficial to compute attention in parallel 𝑀times with different parameters 𝑊𝑞, 𝑊𝑘, and 𝑊𝑣, and combine the outputs at the end, i.e., 𝒗′ 𝑖= 𝑀 ∑ 𝑚=1 𝑊𝑜 𝑚𝒗′ 𝑖𝑚,(10) where 𝑊𝑜is again a learnable linear transformation. In our approach, we use the attention mechanism in the encoder to incorporate information from the set of agents 𝐴into the representation of an agent 𝑎𝑖by using agents in 𝐴as queries, keys, and values. Thus, the attention mechanism behaves as a messagepassing approach between consecutive neural layers in the attention-based model. This mechanism is known as self-attention since the elements in 𝐴are updated according to elements in the same set [55]. Further, in the decoder, we update the representation of a collective 𝐶with information about the agents 𝐴, i.e., we employ 𝐶as query and value, while 𝐴behaves as keys, following the standard nomenclature adopted for attention models [55,6]. 3.1.2. Encoder Our encoder is inspired by the one proposed by Vaswani et al. [55], but in contrast with the original model, we omit positional encoding since the order of the elements in the pool of agents is not relevant for the formation of collectives. Instead, we use an input feed-forward layer to encode elements in the pool of agents 𝐴from its 𝑑𝑎dimensional feature representation to a 𝑑ℎdimensional embedding before the main attention blocks. To get the encoded representation of the pool of agents 𝒉𝐴, the input embeddings are updated using 𝑁attention blocks depicted in Fig. 4, each one consisting of two sub-layers: a multi-head self-attention and a feed-forward layer. Each sub-layer adds a residual connection [56]and performs layer normalization [57]on its outputs, i.e., LayerNorm(𝑥 +sub-layer(𝑥)). To facilitate residual connections, all sub-layers in the encoder use the same dimensionality 𝑑ℎ. Artificial Intelligence 328 (2024) 104064 16 A. Fenoy, F. Bistaffa and A. Farinelli Data availability Data will be made available on request. Acknowledgements The authors gratefully acknowledge the computer resources at Artemisa, funded by the European Union ERDF and Comunitat Valenciana (through the 2014–2020 FEDER Operative Programme of Comunitat Valenciana, project IDIFEDER/2018/048) as well as the technical support provided by the Instituto de Fisica Corpuscular, IFIC (CSIC-UV). This work was supported by the “ACISUD” project (PID2022-136787NB-I00) funded by MCIN/AEI/10.13039/501100011033 and by the “YOMA Operational Research” project (OPE02570) funded by the Botnar Foundation. References [1] European Commission, Collective awareness platforms for sustainability and social innovation, https://ec .europa .eu /digital -single -market /en /collective - awareness, 2021. [2] United Nations, Sustainable development goals, https://www .un .org /sustainabledevelopment /sustainable -development -goals, 2015. [3] F. Bistaffa, C. Blum, J. Cerquides, A. Farinelli, J.A. Rodríguez-Aguilar, A computational approach to quantify the benefits of ridesharing for policy makers and travellers, IEEE Trans. Intell. Transp. Syst. 22 (2021) 119–130. [4] J. Alonso-Mora, S. Samaranayake, A. Wallar, E. Frazzoli, D. Rus, On-demand high-capacity ride-sharing via dynamic trip-vehicle assignment, Proc. Natl. Acad. Sci. 114 (2017) 462–467. [5] E. Andrejczuk, F. Bistaffa, C. Blum, J.A. Rodríguez-Aguilar, C. Sierra, Synergistic team composition: a computational approach to Foster diversity in teams, Knowl.-Based Syst. 182 (2019) 104799. [6] W. Kool, H. van Hoof, M. Welling, Attention, learn to solve routing problems!, in: Proceedings of the International Conference on Learning Representations, 2019, pp. 1–25. [7] F. Wu, S.D. Ramchurn, Monte-Carlo tree search for scalable coalition formation, in: Proceedings of the International Joint Conference on Artificial Intelligence, 2020, pp. 407–413. [8] G. Chalkiadakis, E. Elkind, M. Wooldridge, Computational Aspects of Cooperative Game Theory, Synthesis Lectures on Artificial Intelligence and Machine Learning, Morgan and Claypool Publishers, 2011. [9] F. Präntare, F. Heintz, An anytime algorithm for optimal simultaneous coalition structure generation and assignment, Auton. Agents Multi-Agent Syst. 34 (2020) 1–31. [10] J. Cerquides, A. Farinelli, P. Meseguer, S.D. Ramchurn, A tutorial on optimization for multi-agent systems, Comput. J. 57 (2014) 799–824. [11] O. Shehory, S. Kraus, Methods for task allocation via agent coalition formation, Artif. Intell. 101 (1998) 165–200. [12] R.B. Myerson, Graphs and cooperation in games, Math. Oper. Res. 2 (1977) 225–229. [13] T. Rahwan, T.P. Michalak, M. Wooldridge, N.R. Jennings, Coalition structure generation: a survey, Artif. Intell. 229 (2015) 139–174. [14] C.-H. Lin, Corporate tax structures and a special class of set partitioning problems, Ph.D. thesis, Case Western Reserve University, Cleveland, OH, USA, 1975. [15] N. Changder, S. Aknine, S. Ramchurn, A. Dutta, Odss: efficient hybridization for optimal coalition structure generation, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2020, pp. 7079–7086. [16] T. Michalak, T. Rahwan, E. Elkind, M. Wooldridge, N.R. Jennings, A hybrid exact algorithm for complete set partitioning, Artif. Intell. 230 (2016) 14–50. [17] T. Voice, M. Polukarov, N.R. Jennings, Coalition structure generation over graphs, J. Artif. Intell. Res. 45 (2012) 165–196. [18] F. Bistaffa, A. Farinelli, G. Chalkiadakis, S.D. Ramchurn, A cooperative game-theoretic approach to the social ridesharing problem, Artif. Intell. 246 (2017) 86–117. [19] T. Voice, S.D. Ramchurn, N.R. Jennings, On coalition formation with sparse synergies, in: Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems, 2012, pp. 223–230. [20] F. Bistaffa, A. Farinelli, J. Cerquides, J. Rodríguez-Aguilar, S.D. Ramchurn, Algorithms for graph-constrained coalition formation in the real world, ACM Trans. Intell. Syst. Technol. 8 (2017) 1–24. [21] S. Ieong, Y. Shoham, Marginal contribution nets: a compact representation scheme for coalitional games, in: Proceedings of the ACM Conference on Electronic Commerce, 2005, pp. 193–202. [22] L. Tran-Thanh, T.-D. Nguyen, T. Rahwan, A. Rogers, N.R. Jennings, An efficient vector-based representation for coalitional games, in: Proceedings of the International Joint Conference on Artificial Intelligence, 2013, pp. 383–389. [23] F. Bistaffa, G. Chalkiadakis, A. Farinelli, Efficient coalition structure generation via approximately equivalent induced subgraph games, IEEE Trans. Cybern. 52 (2021) 5548–5558. [24] X. Deng, C. Papadimitriou, On the complexity of cooperative solution concepts, Math. Oper. Res. 19 (1994) 257–266. [25] M.E. Gaston, M. desJardins, Agent-organized networks for dynamic team formation, in: Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems, 2005, pp. 230–237. [26] T. Lappas, K. Liu, E. Terzi, Finding a team of experts in social networks, in: Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2009, pp. 467–476. [27] L.S. Marcolino, A.X. Jiang, M. Tambe, Multi-agent team formation: diversity beats strength?, in: Proceedings of the International Joint Conference on Artificial Intelligence, 2013, pp. 279–285. [28] S. Liemhetcharat, M. Veloso, Weighted synergy graphs for effective team formation with heterogeneous ad-hoc agents, Artif. Intell. 208 (2014) 41–65. [29] A. Georgara, J.A. Rodríguez-Aguilar, C. Sierra, Towards a competence-based approach to allocate teams to tasks, in: Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems, 2021, pp. 1504–1506. [30] A. Georgara, J.A. Rodríguez-Aguilar, C. Sierra, O. Mich, R. Kazhamiakin, A. Palmero Approsio, J.-C. Pazzaglia, An anytime heuristic algorithm for allocating many teams to many tasks, in: Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems, 2022, pp. 1598–1600. [31] A. Farinelli, M. Bicego, F. Bistaffa, S.D. Ramchurn, A hierarchical clustering approach to large-scale near-optimal coalition formation with quality guarantees, Eng. Appl. Artif. Intell. 59 (2017) 170–185. [32] Y. Bengio, A. Lodi, A. Prouvost, Machine learning for combinatorial optimization: a methodological tour d’horizon, Eur. J. Oper. Res. (2020). [33] O. Vinyals, M. Fortunato, N. Jaitly, Pointer networks, Adv. Neural Inf. Process. Syst. (2015) 2692–2700. [34] C.K. Joshi, T. Laurent, X. Bresson, An efficient graph convolutional network technique for the travelling salesman problem, preprint, arXiv :1906 .01227, 2019. [35] M. Nazari, A. Oroojlooy, L. Snyder, M. Takác, Reinforcement learning for solving the vehicle routing problem, Adv. Neural Inf. Process. Syst. 31 (2018). [36] J. James, W. Yu, J. Gu, Online vehicle routing with neural combinatorial optimization and deep reinforcement learning, IEEE Trans. Intell. Transp. Syst. 20 (2019) 3806–3817. Artificial Intelligence 328 (2024) 104064 17 A. Fenoy, F. Bistaffa and A. Farinelli [37] J.-Y. Ding, C. Zhang, L. Shen, S. Li, B. Wang, Y. Xu, L. Song, Accelerating primal solution findings for mixed integer programs based on solution prediction, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2020, pp. 1452–1459. [38] Z. Li, Q. Chen, V. Koltun, Combinatorial optimization with graph convolutional networks and guided tree search, Adv. Neural Inf. Process. Syst. 31 (2018). [39] A.S. Xavier, F. Qiu, S. Ahmed, Learning to solve large-scale security-constrained unit commitment problems, INFORMS J. Comput. 33 (2021) 739–756. [40] Y. Sun, X. Li, A. Ernst, Using statistical measures and machine learning for graph reduction to solve maximum weight clique problems, IEEE Trans. Pattern Anal. Mach. Intell. 43 (2019) 1746–1760. [41] P. Kilby, J. Slaney, T. Walsh, et al., The backbone of the travelling salesperson, in: Proceedings of the International Joint Conference on Artificial Intelligence, 2005, pp. 175–180. [42] A. Hottung, K. Tierney, Neural large neighborhood search for the capacitated vehicle routing problem, in: Proceedings of the European Conference on Artificial Intelligence, 2020, pp. 443–450. [43] E. Khalil, P. Le Bodic, L. Song, G. Nemhauser, B. Dilkina, Learning to branch in mixed integer programming, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2016, pp. 724–731. [44] M. Gasse, D. Chételat, N. Ferroni, L. Charlin, A. Lodi, Exact combinatorial optimization with graph convolutional neural networks, Adv. Neural Inf. Process. Syst. (2019) 15580–15592. [45] V. Nair, S. Bartunov, F. Gimeno, I. von Glehn, P. Lichocki, I. Lobov, B. O’Donoghue, N. Sonnerat, C. Tjandraatmadja, P. Wang, et al., Solving mixed integer programs using neural networks, preprint, arXiv :2012 .13349, 2021. [46] M. Kruber, M.E. Lübbecke, A. Parmentier, Learning when to use a decomposition, in: Proceedings of the International Conference on the Integration of Constraint Programming, Artificial Intelligence, and Operations Research, 2017, pp. 202–210. [47] N. Karalias, A. Loukas, Erdos goes neural: an unsupervised learning framework for combinatorial optimization on graphs, Adv. Neural Inf. Process. Syst. (2020) 6659–6672. [48] I. Bello, H. Pham, Q.V. Le, M. Norouzi, S. Bengio, Neural combinatorial optimization with reinforcement learning, preprint, arXiv :1611 .09940, 2016. [49] R.J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Mach. Learn. 8 (1992) 229–256. [50] V. Konda, J. Tsitsiklis, Actor-critic algorithms, Adv. Neural Inf. Process. Syst. 12 (1999). [51] Y. Zhang, L. Bliek, P. da Costa, R.R. Afshar, R. Reijnen, T. Catshoek, D. Vos, S. Verwer, F. Schmitt-Ulms, A. Hottung, T. Shah, M. Sellmann, K. Tierney, C. Perreault-Lafleur, C. Leboeuf, F. Bobbio, J. Pepin, W.A. Silva, R. Gama, H.L. Fernandes, M. Zaefferer, M. López-Ibáñez, E. Irurozki, The first AI4TSP competition: learning to solve stochastic routing problems, Artif. Intell. 319 (2023) 103918. [52] R.M. Karp, Reducibility Among Combinatorial Problems, Springer, 2010. [53] M. López-Ibáñez, J. Dubois-Lacoste, L.P. Cáceres, M. Birattari, T. Stützle, The IRACE package: iterated racing for automatic algorithm configuration, Oper. Res. Perspect. 3 (2016) 43–58. [54] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016. [55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Adv. Neural Inf. Process. Syst. (2017) 5998–6008. [56] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778. [57] J.L. Ba, J.R. Kiros, G.E. Hinton, Layer normalization, preprint, arXiv :1607 .06450, 2016. [58] A.G. Barto, R.S. Sutton, C.W. Anderson, Neuronlike adaptive elements that can solve difficult learning control problems, IEEE Trans. Syst. Man Cybern. (1983) 834–846. [59] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, preprint, arXiv :1707 .06347, 2017. [60] Y. Kwon, J. Choo, B. Kim, I. Yoon, Y. Gwon, S. Min, POMO: policy optimization with multiple optima for reinforcement learning, Adv. Neural Inf. Process. Syst. 33 (2020). [61] D.P. Kingma, J. Ba Adam, A method for stochastic optimization, preprint, arXiv :1412 .6980, 2014. [62] T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, Continuous control with deep reinforcement learning, preprint, arXiv : 1509 .02971, 2015. [63] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: Proceedings of the International Conference on Machine Learning, 2018, pp. 1861–1870. [64] B.W. Silverman, Density Estimation for Statistics and Data Analysis, Routledge, 2018. [65] E. Liscio, R. Lera-Leri, F. Bistaffa, R.I.J. Dobbe, C.M. Jonker, M. Lopez-Sanchez, J. Rodríguez-Aguilar, P.K. Murukannaiah, Value inference in sociotechnical systems, in: Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems, 2023, pp. 1774–1780.