Embedding Graph Convolutional Networks in Recurrent Neural Networks for Predictive Monitoring

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 1 Embedding Graph Convolutional Networks in Recurrent Neural Networks for Predictive Monitoring Efr´ en Rama-Maneiro, Juan C. Vidal, and Manuel Lama Abstract—Predictive monitoring of business processes is a subfield of process mining that aims to predict, among other things, the characteristics of the next event or the sequence of the next events. Although multiple approaches based on deep learning have been proposed, mainly recurrent neural networks and convolutional neural networks, none of them really exploit the structural information available in process models. This paper proposes an approach that simultaneously learns spatio-temporal information from both the event log and the process model by combining recurrent neural networks with graph convolutional networks. Thus, common patterns from process models, such as loops or parallels, can be learned while avoiding overwriting information during the encoding phase. An experimental evaluation of real-life event logs shows that our approach is more consistent and outperforms the current state-of-the-art approaches. Index Terms—Process mining, Predictive business monitoring, Deep Learning, Graph Neural Networks, Recurrent Neural Networks ✦ 1 INTRODUCTION Process mining [1] is a discipline that aims to describe the future and past behavior of a business process, given the information available in an event log. Multiple analyses can be applied to event logs, such as discovering a process model as an abstract representation of the underlying business process (process discovery), improving a process model using the information from the event log (process enhancement), or comparing a process model with an event log to check its degree of conformance (process conformance) [1]. However, it not only focuses on the description of the current process behavior but also on predicting its future behavior. Thus, predictive monitoring is a subfield of process mining concerned with forecasting how an ongoing case is going to unfold in the future [2]. These predictions may involve information such as the next activity or sequence of activities, when the next event will happen, or how much time is left until the end of the case. Many machine learning techniques have been applied to predictive monitoring, although approaches based on deep learning are the ones that perform better [3]. Recurrent Neural Networks (RNNs), are the most popular in this domain due to the sequential nature of traces in processes [4]. However, autoencoders [5], [6], generative adversarial networks (GANs) [7], convolutional neural networks (CNNs) [8], [9], or other types of RNNs [10], [11], [12] have also been used. •Efr´en Rama-Maneiro, Juan C. Vidal, and Manuel Lama are with the Centro Singular de Investigaci´on en Tecnolox´ıas Intelixentes (CiTIUS), Universidade de Santiago de Compostela, Santiago de Compostela, Spain. Juan C. Vidal is also with the Departamento de Electr´onica e Computaci´on, Universidade de Santiago de Compostela, Galicia, Spain. email: {efren.rama.maneiro, juan.vidal, manuel.lama}@usc.es Manuscript received April 22, 2023 revised X Almost all deep learning approaches in predictive monitoring build a predictive model exclusively from the information available in the traces of the processes [5], [6], [7], [8], [9], [10], [11], [13], [14], [15], [16], [17]. These approaches disregard the explicit structural information available in the process models, i.e., behavioral patterns such as loops and parallels may be missing since the neural networks might not be able to detect them with only the trace information. On the other hand, the approaches that rely on process models as input [18], [19], [20] obtain information from the connectivity between the model activities [19], [20] or by performing a token replay over its model [18]. However, these model-based approaches are unable to fully leverage the structural and temporal information available in the event log. They disregard the order in which the events appear in the prefix, are subjected to overwrite features if any activity appears repeated in the prefix, and most of them [19], [20] rely on Directly Follows Graph (DFGs) process models, which are less expressive than Petri nets [21]. In this paper, we hypothesize that the performance of predictive models can be improved by combining the information from both the traces and the process model and leveraging the information on how the execution of the process model behaves over time. Thus, we propose to capture this execution using graph neural networks (GNNs) and RNNs by taking into account the full sequence of states and not only the state of the replay previous to the prediction. This should facilitate the detection of some behavioral patterns common in process models, such as loops and parallels, in addition to time-related behavioral patterns. Parallels, in which a set of activities might appear in any order in a trace, are harder to detect without considering the process model, so knowing their presence beforehand could ease the training phase of the neural network. Regarding loops, the neural network might benefit from This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 2 knowing beforehand where the loop is going back, which activities compose the loop, and whether inner loops exist. Moreover, using the process model as an additional input could help the neural network focus more on other event log information since the relationships between the activities are already explicitly available from the process model. We validated our approach using ten publicly available datasets against the main state-of-the-art approaches, showing that our approach improves the accuracy of predictions in almost all cases. In summary, the main contributions of our graph recurrent neural model are as follows: •A neural model that simultaneously learns spatiotemporal information from both the event log and the process model by combining recurrent neural networks and graph convolutional neural networks, which, unlike other state-of-the-art approaches, learns the process model structure explicitly. Furthermore, the proposed model not only considers the structural information of the model but explicitly takes into account the information extracted from the events of the prefix. •A process model encoding that allows learning the dependencies between the loops and parallels, while retaining memory efficiency during training. This encoding, unlike other encodings proposed in the state of the art, retains the full information of the prefix, without overwriting information during the encoding phase. •An evaluation of the proposal on a wide variety of publicly available event logs, including a comparison with 10 state-of-the-art proposals, an ablation study on both the structural components of the architecture and the information extracted from the prefix, an analysis of the approach performance w.r.t. the presence of loops in the event logs, and an analysis of the hyperparameter effect on the approach results. This paper is structured as follows, Section 2 shows the state of the art in deep learning-based techniques for predictive monitoring, Section 3 shows some background needed to build the approach, Section 4 describes the proposed approach, Section 5 shows the evaluation of the proposed approach in real-life event logs, and Section 6 highlights the conclusions of the paper and future work. 2 RELATED WORK The first works in predictive monitoring with deep learning heavily relied on RNNs to learn predictive models. For example, [14] use LSTM neural networks to predict both the next activity and the next timestamp in an ongoing process instance. The activities are encoded as a one-hot vector, and they also consider time features such as the time passed since the previous event or the beginning of the case. [13] also aims to predict the next activity by encoding the activity through embeddings instead of a one-hot vector, thus reducing the dimensionality of the input. Other types of architectures based on recurrent neural networks have also been explored. [11] uses a Differentiable Neural Computer [22], which is a type of neural network that has an external memory to enhance the representation of longer-term dependencies in a sequence. They define an encoder-decoder that is trained to predict either the next activity or the next timestamp. [23] also uses an encoderdecoder network, but they rely on an attention mechanism to take into account every hidden state of the LSTM that learns from the input sequence. These kinds of architectures may yield better results at the expense of increasing the complexity of the network, or more training difficulties than simpler types of RNN. Some predictive monitoring approaches propose novel ways to encode the attributes of the event log. [15] clusters the resources available in the event log into roles and learn an LSTM that simultaneously predicts the next activity, next timestamp, and next role. [10] also clusters the events based on their attributes and uses these clustering labels as additional information in a recurrent neural network. The main advantage of these types of encodings is reducing the complexity of the input by grouping attributes in fixed categories, however, at the risk of discarding potentially relevant information. Some other approaches are based on CNNs. [9] uses CNNs to predict the next activity. They rearrange the prefixes in a grid-like fashion using an encoding in which, for each event, they count the number of occurrences present in the prefix. These prefixes are fed to a two-dimensional CNN, which is used to predict the next activity. [8] also employs CNNs by adapting the Inception model [24] to predict the next activity, outperforming LSTMs in some event logs. In [12], Gated Convolutional Neural Networks [25] and Key-Value-Predict [26] Attention Networks are introduced. The former combines convolutional networks with a gating mechanism, similar to the one used in LSTMs and GRUs, while the latter tries to learn the correlation between pairs of elements that belong to the input sequence by means of an attention mechanism. The main advantage of convolutional neural networks is that they are computationally more efficient than RNNs. Nevertheless, they are more complex to design since they have more variation, such as whether and how to stack pooling layers and what size of the convolution filter shall be used. In predictive monitoring, RNNs seem to outperform CNNs due to this cause [4]. Furthermore, some approaches are testing novel architectures for predictive monitoring. [7] rely on GANs [27] to predict the next activity and timestamp of an ongoing process instance, hypothesizing that this type of neural network would alleviate the need for high amounts of training data. [16] are focused on implementing Transformers [28] architectures for predictive monitoring, a type of neural network that relies exclusively on attention mechanisms. Thus, they can avoid predictive performance degradation when the RNN faces long sequences and improves the learning phase training and inference speed. Even though the results of these approaches look promising, there are still doubts about whether they can perform reliably across a wide variety of datasets, and, even so, they do not use the potential information available in process models. Finally, some works rely on explicit process models to help encode the prefixes. [18] builds feature vectors by performing a token replay of a prefix over a process model. These feature vectors include the information about the This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 3 most recent activation on a Petri net, the time of activation, which is a decay function, the number of tokens, and the attributes available in the event log. These vectors are then fed to a deep feed-forward neural network. In [20], the usage of Gated Graph Neural Networks is explored. Their adjacency matrix can be based on the events of a prefix, the activities of the prefix, or the Directly Follows Graph (DFG) extracted from the event log. The edges in their graph convey information about the prefix. [19] investigates the usage of GCNs for predicting the next activity. Their adjacency matrix is based on the DFG, and they focus on the differences in various methods of encoding this adjacency matrix (binary, weighted, with a Laplacian transform, . . . ). However, the model-based approaches are still lacking in terms of taking full advantage of the structural and temporal information available in the event log. These approaches disregard the order in which the events appear in the event log. For example, [18] only takes into account this order by means of a decay function associated with each of the places of the Petri net, only accounting for its most recent activation, while [19] and [20], only consider the event order using time features extracted from the event log, such as the time since the last event. Structurally speaking, [18] relies on MLPs for their architecture, so the process model is not explicitly considered. Both [19] and [20] rely on GNNs but they use DFGs as their process model, which is less expressive than Petri nets. Furthermore, every model-based approach only retains the most recent activation for an activity or place, so they are prone to overwrite information when a loop occurs. Our approach relies on embeddings to encode the categorical features, similarly to [13], because we use the same attributes as in [4] and some logs have a high number of different attributes; therefore, using a one-hot encoding would increase the memory consumption dramatically [4]. Furthermore, our approach both learns structural properties explicitly from a process model using GCNs and learns temporal information from the order of the events within the trace. Thus, unlike the other model-based approaches from the state of the art, we capture the whole time dimension from the prefix, without being affected by overwriting features when an activity is repeated, as is the case with [18], [19], [20]. Moreover, our approach is based on Petri nets, which are more expressive than DFGs. In addition, the latter is unable to represent parallels or distinguish OR from XOR splits [21]. Therefore, we can capture more complex relationships between the events, unlike other model-based approaches. 3 PRELIMINARIES In this section, we present the main concepts needed to understand our approach for predicting the next activity of a running case. A symbol table is highlighted in TABLE 1, for ease of reference. 3.1 Definitions Definition 1 (event). Let AC be the universe of activities, Cthe universe of cases, T I the time domain, and D1, . . . , Dmthe universes of each of the attributes of the events of the event log, with m≥0. An event e∈Eis a tuple (a, c, t, d1, . . . , dm)where a∈AC,c∈C,t∈T I and di∈ {Di∪ϵ}with i∈[1, m]and ϵbeing the empty element. Definition 2 (projection function). Let πAC ,πC,πT I , and πDibe functions that map an event to an activity, a case identifier, a timestamp, and an attribute, that is, πAC (e) = a,πC(e) = c,πT I (e) = t, and πDi(e) = di. Definition 3 (trace). Let Sbe the universe of traces, then a trace σ∈Sis a non-empty sequence of events σ= ⟨e1, . . . , en⟩, which holds that ∀ei, ej∈σ;i, j ∈[1, n] : j > i ∧πC(ei) = πC(ej)∧πT I (ej)≥πT I (ei)where |σ|=n. Definition 4 (event log). An event log is a set of traces, B= {σ1, . . . , σl}such as B={σi|σi∈S∧i∈[1, b]}where |B|=b. Let TABLE 2 be an example of an event log from a loan application process [29]. In this event log, there are four different activities, each representing the task performed by a resource to achieve a goal [30]. The sequence of activities belonging to the same trace identifier is called a trace, and a partial trace, which is used as an input to a predictive model, is called a prefix. In this example, there are two different traces: “214364” and “173712”. Note that the activities in a trace must appear ordered chronologically according to their execution time, represented by the column “timestamp” in this case. Event logs can optionally have additional information, such as the resource that executes each activity. Trace ID Activity Timestamp Resource 173712 W Afhandelen leads (WAL) 01/10/2011 08:45:13 Gordon 173712 W Completeren aanvraag (WCA) 02/10/2011 09:26:51 Alex 173712 W Completeren aanvraag (WCA) 03/10/2011 10:39:04 Barney 173712 W Completeren aanvraag (WCA) 04/10/2011 11:51:33 Adrian 214364 W Afhandelen leads (WAL) 06/10/2011 09:10:22 Joseph 214364 W Completeren aanvraag (WCA) 06/10/2011 15:13:22 Joseph 214364 W Nabellen offertes (WNO) 08/10/2011 10:48:11 Enrico 214364 W Valideren aanvraag (WVA) 08/10/2011 11:10:32 Harambe TABLE 2: Potential example from the BPI-2012-W-C event log. Symbol Description AC Universe of activities (a∈AC) CUniverse of cases (c∈C) T I Time domain (t∈T I) DnUniverse of the attribute n(dn∈Dn) πXProjection function to a universe X SUniverse of traces (σ∈S) P T Petri net PSet of places from a Petri net (p∈P) TSet of transitions from a Petri net (t∈T) FSet of directed arcs from a Petri net VSet of vertices of the place graph ESet of edges of the place graph AAdjacency matrix of a graph INIdentity matrix NNumber of nodes of a graph HHidden size QSet of node features XFeature matrix (unspecified) RSet of attribute features LLength of the longest trace of the event log ⊕Concatenation operator TABLE 1: List of symbols used in the paper This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 4 Definition 5 (prefix). Let σbe a trace such as σ=⟨e1, . . . , en⟩ and k∈[1, n]be any positive integer. The event prefix of length k,hdkcan be defined as follows: hdk(σ) = ⟨e1, . . . , ek⟩. The activity prefix can be defined as the application of πAC to the whole event prefix, being πAC (hdk(σ)) = ⟨πAC (e1), . . . , πAC (ek)⟩, respectively. In this paper, we focus on predicting the next activity of a given prefix. Formally: Definition 6 (next activity prediction). Let hdk(σ)be an event prefix such as hdk(σ) = ⟨e1, . . . , ek⟩, and e′be a predicted event by a function Ω, then, the next activity prediction problem can be defined as learning a function ΩAC such as ΩAC (hdk(σ)) = πAC (e′ k+1). In this paper, we leverage information from process models using deep learning. Our business process models are represented by a place/transition Petri net [31] because it allows a richer representation of the behavior of a business process than other approaches such as the DFG. A Petri net is a bipartite graph composed of two types of nodes: places, which can contain tokens, and transitions. More formally, a Petri net can be defined as follows: Definition 7 (Petri net). A Petri net is a tuple PT = (P, T, F) where: •Pis a finite set of places. •Tis a finite set of transitions. •P∩T=∅ •F⊆(P×T)∪(T×P)is a set of directed arcs. Transitions can be classified into observable transitions, which are associated with a process activity, and silent transitions, which are related to control flow routing. Thus, the order of the execution of the activities within a prefix is modeled through the flow of the tokens across the network. The places contain the tokens that are produced/consumed by transitions, while the marking of a Petri net is the number of tokens in a given place. These tokens are produced and consumed when a transition is executed. When this happens, a token is removed from each input place of the transition and is inserted in every output place of it. This process can only be performed when a transition is enabled, i.e., when the input places of a transition have at least one token. Recall Fig. 1, which shows the execution semantics of a process model represented as a Petri net mined from the BPI-2012-W-C from TABLE 2. In this model, places are represented as circles, observable and silent transitions are represented as white and black rectangles, respectively, and red transitions represent each fired transition for each event replayed in the model. Thus, the replay of the trace with the case identifier “214364” in the process model would produce the execution semantics depicted in the figure. Our model reduces the Petri net of the process model to aplace graph that is easier to handle in our deep learningbased solution. Definition 8 (Place graph). A place graph is a directed graph G= (V, E)with vertices Vand edges Ethat represents a Petri net PT = (P, T, F )where: •v∈V⇐⇒ p∈P. (a) (b) (c) (d) Fig. 1: Execution semantics from the Petri net mined from the event log of TABLE 2. •Each edge e∈Econnects two vertices v∈Vif and only if a transition interconnects two places, i.e., let t∈T;p1, p2∈P; and, v1, v2∈V; then (v1, v2)∈ E→(p1, t)∈F∧(t, p2)∈F. The place graph is represented in our approach by a binary adjacency matrix, i.e., A∈R|P|×|P|. Modeling the Petri net as a place graph has two advantages. First, the interactions between the activities of a Petri net can be represented only by the tokens consumed or produced in the places because each event is associated with a snapshot of the execution state of the Petri net. Second, the memory consumption is reduced by approximately a factor of two, which is essential since the dimensions of every matrix related to the graph depend on the number of its nodes. 3.2 Graph Recurrent Neural Networks GNNs are a type of neural network that operates on the graph structure, capturing dependencies between each node of a graph. Graph convolutional networks (GCN) [32] are a type of GNN architecture that is defined as follows: gθ⋆ x ≈θ(IN+D−1 2AD−1 2x)(1) Where INis the identity matrix, Ais the adjacency matrix of the graph, and Dis the degree matrix of the graph. Equation 1 can lead to exploding/vanishing gradients, so the renormalization trick IN+D−1 2AD−1 2→˜ D−1 2˜ A˜ D−1 2is introduced in [32] by adding self-loops to the graph, i.e, ˜ A=A+INand ˜ Dii =Pj˜ Aij. Thus, we achieve the following GCN operator: GCN(A, X) = ˜ D−1 2˜ A˜ D−1 2XΘ(2) Where Θis a matrix of learnable filter parameters. This model is unsuitable for predictive monitoring for two reasons. First, it does not take into account directionality in the input place graph due to the symmetric normalization ˜ D−1 2˜ A˜ D−1 2. Second, it does not fully consider the information available in the whole sequence of execution states related to the events since it can only compute the last state. Thus, this model is prone to the problem of loop overwriting (detailed in section 4.2) and, by extension, to the loss of information contained in the prefix. This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 5 To alleviate the first issue, we modify the GCN operator to use the random walk normalized Laplacian with the renormalization trick. Thus, the directionality of the graph could be taken into account as follows [33]: GCN(A, X) = ˜ D−1˜ AXΘ(3) To solve the second issue, we propose to apply a recurrent operation to the sequence of token replays, using a Gated Recurrent Unit (GRU) as a foundation block, because they obtain similar results to an LSTM but are computationally less expensive: zt=σ(Wzxt+Uzht−1+bz) rt=σ(Wrxt+Urht−1+br) ˜ ht=tanh(Whxt+Uh(rt◦ht−1) + bz) ht=zt◦ht−1+ (1 −zt)◦˜ ht (4) Then, the product between the learnable weights and the inputs is substituted — or the hidden state — by a graph convolution, leading to a Graph Recurrent Neural Network (GRNN) [34]. zt=σ(GCN(xt, A) + GCN(ht−1, A) + bz) rt=σ(GCN(xt, A) + GCN(ht−1, A) + br) ˜ ht=tanh(GCN(xt, A) + rt◦GCN(ht−1, A) + bz) ht=zt◦ht−1+ (1 −zt)◦˜ ht (5) This substitution allows performing the graph convolution operation simultaneously with the recurrent operation for each timestep. In this way, the neural network can learn the spatiotemporal dependencies of the input easier than if we used a GCN first and then a GRU to take into account the time dimension (as described in subsection 4.2). Furthermore, note that we apply the GCN operation to the hidden state of the GRU so as to force this state to depend on the process model, represented, in this case, by the normalized random walk laplacian matrix. Note that the expression 5 can only deal with graphs with a fixed structure because the underlying process model never changes. 3.3 Readout The output of the GRNN operation is the last hidden state of the GRU, which is a matrix of dimensions R|P|×H. Since the objective of the approach is to predict the next activity of an ongoing process, we will assign a unique label to a whole graph (graph-level task). Thus, a summary operation over the graph is needed to project the output of a graph convolution operation into a single vector. This operation is called readout, and it reduces the graph outputs in the dimension of the nodes, i.e., the result of a readout operation is a vector RH. Many readout operations are available such as maximum, average, sum, or weighted average using attention. In this paper, we use the maximum operation for its simplicity and because it is suitable for classification problems [35]. Formally, let h∈R|P|×Hbe the matrix resulting from applying a graph convolution operation, then the maximum readout operation is defined as follows: hG=max(hv)|v∈G(6) Where hGis the summarized representation of the graph G,max is the element-wise max-pooling operation, and hv is the learned embedding for the node v. 4 APPROACH This section introduces our approach, “RecurrentGraph Convolutional Process Predictor” (TACO), which aims to address a multiclass classification problem by mapping a label to a given graph structure and its corresponding replay. Fig. 2 illustrates the data flow of our approach. The starting point is an event log and a prefix, which serve as the basis for the prediction. Given the event log, the Split Miner is applied to generate a set of process models, each corresponding to a specific hyperparameter configuration. Subsequently, the best process model, based on its fitness on the validation set, is selected. This model is then converted into a place graph that encapsulates the relationships between the Petri net places. The graph is represented as an Case ID Activity Timestamp Case 1123 Activity A 10/02/2020 19:30:02 Case 1123 Activity B 11/02/2020 11:54:34 Case 1123 Activity C 12/02/2020 09:56:12 Case 1123 Activity A 15/02/2020 10:15:09 Activity Timestamp Activity A 10/02/2020 19:30:02 Activity B 11/02/2020 11:54:34 A B C D A A B C D Case ID Activity Case 1123 Activity A Case 1123 Activity B Case 1123 Activity C Case 1123 Activity A B C D Fig. 2: Pipeline of the data flow in TACO. The sizes of the matrices are highlighted at each step. This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 6 adjacency matrix and is further normalized into a randomwalk normalized Laplacian serving as one of the inputs for the Graph Recurrent Neural Network (GRNN). As the process model is common to every prefix of the log, its discovery is performed only once, just before beginning to replay any prefix. Upon obtaining a process model, we replay the prefix on it, generating a node feature matrix that represents the activations of the places and transitions for that prefix. In addition, an attribute feature matrix containing additional information, such as the executed activities, event-related temporal data, and other attributes is created. As a result, the inputs to the GRNN consist of (i) the adjacency matrix and (ii) the node feature matrix —sequence of the activations in the Petri net after replaying the prefix—. The output of the GRNN operation, concatenated to the attribute feature matrix, is fed into an LSTM, whose outputs are subsequently passed to a softmax classifier to make the next activity prediction. To ease the comprehension of our approach, Algorithm 1 details the steps needed to mine the best process model and obtain the process metrics that are required in order to build the feature matrices. In line 1, the best model in terms of fitness is mined using the Split Miner [36] process discovery algorithm. To do that, a grid search is performed following the same approach of [18] where the frequency threshold ϵ —which controls the filtering process— and the parallelism detection parameter ηare optimized. Note that TACO works with any discovery algorithm and does not require that the process model has perfect fitness. Thus, if a prefix cannot be replayed because of a trace misalignment, the tokens will be added to the input places of the corresponding transition in order to be fired. After we obtain the best model from the Split Miner, in lines 2 to 5, the following values to build the node and attribute feature matrices are calculated: (i) the number of places, (ii) the maximum number of transitions, (iii) the maximum trace length, and (iv) the maximum number of attributes. Then, in line 6, the place graph is built, which, in turn, is represented by an adjacency matrix in line 7. In line 8, this adjacency matrix is transformed into a random walk normalized Laplacian. Algorithm 2 builds the feature matrices used to train the neural model. The algorithm starts in line 1, where an empty node matrix is created using the dimensions related to the maximum number of transitions and the number of places. In lines 2 and 3, the lists that will contain the sequence of node matrices and the attribute matrices are created. In lines 5 to 10, we iterate over the set of events of the prefix, building the node matrix (lines 6 and 7) and the attribute matrix of each event (lines 8 and 9). On the one hand, the function defined in line 11 builds the node matrix using an event, the best-mined process model, a previous node matrix, and the current marking of the Petri net, which is set to the initial marking in line 4 of the algorithm (when this function is called for the first time). Line 12 replays the event in the previously mined process model using the current net marking, thus obtaining the activated places, activated transitions, and the net marking after replaying the event. Then, in lines 12 to 16, the node matrix is updated with this information. Finally, in line 17, the function returns the updated node matrix and the Algorithm 1: Preprocessing algorithm for TACO Input: lwhole process log, ltrain - training set of the event log Output: bestModel - best model mined, initialMarking - initial marking, Nnumber of places of the best process model, Tmax - maximum fireable transitions in the process model, kmaximum trace length in the event log, maxattr - maximum number of attributes in the event log, normalizedLaplacian - normalized laplacian of the place graph. 1bestModel, initialMarking = modelHyperparameterSearch(ltrain); 2N= getNumberPlaces(bestModel); 3Tmax = getMaxTransitions(bestModel, ltrain); 4k= getMaxTraceLength(l); 5maxattr = maxAttributes(l); 6placeGraph = buildPlaceGraph(bestModel); 7adjacencyMatrix = getAdjMatrix(placeGraph); 8normalizedLaplacian = normalize(adjacencyMatrix); Fig. 3: Node feature matrix after encoding the prefix shown on TABLE 2 using the process model from Fig. 1. updated marking. On the other hand, the function defined in line 18 builds the attribute matrix using an event and the previously calculated maximum number of attributes. In line 19, an empty vector of the size of three plus the number of maximum attributes (|R|= 3+m) is created. Then, in lines 20 to 22, the time features are calculated and converted into categorical values. Finally, in lines 23 to 30, the matrix is filled with information about the activities, time, and attributes. 4.1 Encoding In this paper, we distinguish between features specific to each node of the place graph and features inherent to each event of the prefix. The former features belong to a node feature matrix, whereas the latter ones belong to an attribute feature matrix. A node feature matrix is defined as a matrix of dimensions RL×|P|×|Q|that captures the interactions of all events of the prefix with the process model in such a way that each matrix R|P|×|Q|represents the interactions of an event with the process model. If the number of events in the prefix is less than L, we apply prepadding (padding on the left side of the matrix). Let Fig. 3 be an example of a node feature matrix for a single event. The first column of this matrix designated as F1in Fig. 3, shows which place has been activated for a given place, while each subsequent column This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 7 Algorithm 2: Algorithm for building the feature matrices of TACO Input: outputs of Algorithm 1 Output: attributeMatrix - List of attribute feature matrices, nodeMatrixList - List of node feature matrices 1nodeMatrix = newMatrix(N,1+Tmax); 2nodeMatrixList = []; 3attributeMatrixList = []; 4marking = initialMarking; 5for ein pdo 6nodeMatrix, marking = buildNodeMatrix(e, bestModel, nodeMatrix, marking); 7nodeMatrixList.add(nodeMatrix); 8attributeMatrix = buildAttributeMatrix(e, maxattr); 9attributeMatrixList.add(attributeMatrix); 10 end 11 Function buildNodeMatrix(e, bestModel, nodeMatrix, marking): 12 actS,actT, marking = replayEvent(bestModel, e, marking); 13 nodeMatrix[actN, 1] = actN; 14 for i= 1;i < len(actT);i=i+ 1 do 15 nodeMatrix[actN,1 + i] = actT[i]; 16 end 17 return nodeMatrix, marking; 18 Function buildAttributeMatrix(e, maxattr): 19 attributeMatrix = newMatrix(3 + maxattr); 20 timeSincePrev, timeSinceStart = getTimeFeatures(e[”timestamp”]); 21 qTimeSincePrev = discretizeTime(timeSincePrev); 22 qTimeSinceStart = discretizeTime(timeSinceStart); 23 attributeMatrix[1] = e[”activity”]; 24 attributeMatrix[2] = qTimeSincePrev; 25 attributeMatrix[3] = qTimeSinceStart; 26 i= 1; 27 for attribute in edo 28 attributeMatrix[3 + i] = e[attribute]; 29 i+=1; 30 end 31 return attributeMatrix; points out the set of transitions activated for a given event. To know which places and transitions are activated at each moment, we perform a token-replay of each event of the prefix over the process model. The features are accumulated in the node feature matrix, e.g., each row of the matrix in Fig. 3 represents the execution of an event. Therefore, we retain the information from the token replays of previous events of the one being processed. Note that the number of columns of the node feature matrix depends on the number of transitions that can be fired concurrently in the Petri net, including every possible firing of silent transitions. In our example, only one transition is activated between every observable transition, so a single column, F2, is present in the matrix. This number is Fig. 4: Attribute feature matrix after encoding the prefix from TABLE 2. calculated by replaying the whole log and accounting for the maximum number of possible fired transitions between two observable transitions. Note that the first column, F1, which indicates which place has had a token after a transition firing, has the same meaning as each row of the node feature matrix, so it could be thought of as redundant. However, we still retain it since it helps the convergence of our GRNN model. Furthermore, TACO also uses an attribute feature matrix, which encodes information about the entire prefix and has dimensions RL×|R|. This matrix contains the following features for each event: (i) the event log activity whose firing generates a token in the places; (ii) the encoded bucket of the time since the previous event; (iii) the encoded bucket of the time since the first event of the prefix; and (iv) the m attributes associated with the event. Note that, similarly to the node feature matrix case, we apply prepadding to this matrix if necessary. Fig. 4 shows the example of the attribute feature matrix for the trace number “214364” of the log of TABLE 2, whose model was depicted in Fig. 1. In particular, we use the time since the previous event (F1), the time since the first event (F2), the event log activity (F3), and the resource (F4). Note that all features used by our approach are categorical. To convert continuous features into categorical ones, first, we calculate every possible time difference —time between events and time since start— in the training and validation sets of the event log. Then, a quantile-based discretization is applied to this list of values, calculating equal-sized time value intervals that will be used to discretize a given value. Therefore, the interval values will span from 0 to the maximum calculated value in the training-validation event log. If a value is bigger than this latter value, it would be assigned to the highest possible interval. In the case of Fig. 4, the calculated intervals for the time since the previous event (F1) are [0,1000),[1000,20000),[20000,150000),[150000,+∞), and the calculated intervals for the time since the first event are [0,50000),[50000,100000),[100000,200000),[200000,+∞). Rather than converting the categorical features to one-hot encoding, we separately embed each of the categorical features so that the dimensionality of Xis independent of the number of activities of the event log and the memory consumption of the neural network is reduced. Also, note that the values for each column of the matrix are the accumulative values obtained by the token replay from the first event of the prefix to the current event. This means that the place information from previous events is not removed. This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 8 4.2 Loop overwriting It could be thought that both the attribute feature matrix and the node feature matrix could be joined in the same matrix so that each element of the matrix, Xij, represents the jfeature from the place i. Thus, the GCN model could simultaneously process node and event-level information. However, this approach has the problem of overwriting the features of nodes when a loop occurs in a set of places. This is especially relevant when the loop consists of only a single activity because, in this case, the loss of information is maximum due to the low number of places involved in these types of loops. Fig. 5 exemplifies the replay process of the trace “173712” of the BPI-2012-W-C log from TABLE 2 using the place graph depicted also in Fig. 5. As it can be seen, the information of the feature matrix X3is overwritten in the feature matrix X4, due to the presence of a loop in the activity W Completeren aanvraag. Let us suppose a GCN model, which can only process the last state of a replay without fully taking into account the evolution of the trace, the only inputs would be the normalized randomwalk laplacian matrix and the X4feature matrix. Therefore, the information available in X3about the resources cannot be effectively used by the GCN model. This encoding problem could be avoided by extending the feature matrix along with the feature dimension Qup to the maximum loop size present in the event log (L) to consider older events that occurred in the same place, i.e., a feature matrix Xwith dimensions RP×(Q·L). However, some event logs contain loops with a maximum length that would make this approach computationally infeasible. A naive solution to the problem of loop overwriting would be applying the GCN operation independently to each node matrix X1,· · · , Xnfor each prefix event. Then, performing the readout operation after each convolution and feeding that information to an LSTM. However, this solution poses another problem: the structural properties of the input —the sequence of node feature matrices— and its temporal properties —the sequence of attribute feature matrices— are processed independently and, thus, some spatio-temporal interactions could be missed, such as any dependency within a loop. Therefore, in our approach, we propose to use a GRNN, as explained in Section 3.2, to simultaneously tackle the spatial and temporal features from the replay of the prefix over the process model. Fig. 5: Loop overwriting problem exemplified. Encoding for places P6 to P10, temporal and attribute features omitted for the sake of clarity. 5 EVALUATION 5.1 Experimental setup To perform the evaluation of TACO, we relied on the experimentation from [4] under the same conditions: a 5-fold cross-validation is performed, splitting each training fold into an 80%-20% trace distribution to obtain a validation set. The validation set is used to find the best-performing model, in terms of the epoch with the highest validation accuracy. Furthermore, we used the same attributes and compared the approaches using the “accuracy” metric, as reported in [4]. Moreover, we also used the same event logs, but “Sepsis” and “Nasa” because they cannot be applied to [15] as these logs do not contain the resources that perform the events. The characteristics of these event logs are depicted in TABLE 3, where most of them have high variability in both the temporal characteristics of the process execution —duration of events and traces— as well as the length of the traces. There are two modes of executing the approach of [18], so the two configurations are reported. Furthermore, we have extended the experimentation of [4] by adding the approaches of [16], [19] tested under the same conditions. We used the “weighted” variant of [19] due to having the most consistent results across all the datasets. Regarding the mined process models, TABLE 4 shows the average statistics of the mined models for the crossvalidation train and validation sets. The reported metrics for the process models, from left to right in the TABLE, are Event log Traces Activities Events Avg. case length Max. case length Avg. event duration Max. event duration Avg. case duration Max. case duration Variants Helpdesk 4580 14 21348 4.66 15 11.16 59.92 40.86 59.99 226 BPI-2012 13087 36 262200 20.04 175 0.45 102.85 8.62 137.22 4366 BPI-2012-C 13087 23 164506 12.57 96 0.74 30.92 8.61 91.46 4336 BPI-2012-W 9658 19 170107 17.61 156 0.7 102.85 11.69 137.22 2621 BPI-2012-W-C 9658 6 72413 7.5 74 1.75 30.92 11.4 91.04 2263 BPI-2012-O 5015 7 31244 6.23 30 3.28 69.93 17.18 89.55 168 BPI-2012-A 13087 10 60849 4.65 8 2.21 89.55 8.08 91.46 17 BPI-2013-C-P 1487 7 6660 4.48 35 51.42 2254.84 178.88 2254.85 327 BPI-2013-I 7554 13 65533 8.68 123 1.57 722.25 12.08 771.35 2278 Env-permit 1434 27 8577 5.98 25 1.09 268.97 5.41 275.84 116 TABLE 3: Statistics of the event logs used for benchmarking. Time-related measures are shown in days. This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 9 Event log Loops Edges Transitions Places Max. rep. loop Avg. rep. loop BPI-2012 18404807.6 381.2 190.6 68 3 2 BPI-2012-A 5 73.6 36.8 19 0 0 BPI-2012-C 18404807.6 384.4 192.2 69.6 56 4.73 BPI-2012-O 9 48 24 13 0 0 BPI-2012-W 15.6 95.6 47.8 28.0 3 2 BPI-2012-W-C 15.6 96 48 28.2 57 4.66 BPI-2013-C-P 50 96.4 48.2 28.2 5 2.16 Helpdesk 571.6 92.8 186.4 45.6 5 2.1 BPI-2013-I 128.4 65.2 32.6 16 19 2.07 Env-permit 83187.2 255.2 127.6 52.2 3 3 TABLE 4: Average statistics of the process models. the average number of loops, edges, transitions, places, and length and maximum length value of the loops that contain exclusively the same activity. This TABLE shows that the mined process models vary in complexity. The most simple process models are the ones mined from the event logs BPI2012-A and BPI-2012-O since they have the lowest number of loops, transitions, and places; while the most complex event logs are the BPI-2012,BPI-2012-C and Env-permit. Note that the Complete (C) variants of the BPI-2012 logs have more loops than their non-complete counterparts since we treat each combination of lifecycle:transition and activity as a separate activity. We configured our approach with the same hyperparameters for every event log. We used 256 hidden units for both the LSTM and the GRNN and the Adam optimizer with a learning rate of 1e-3 with a cosine annealing warm restart scheduler. We also used ten buckets to quantize the time features and an embedding dimension of size 32. We trained on 100 epochs using an early stopping criterion on the validation accuracy —training is stopped if the validation accuracy does not improve during 10 epochs after reaching the maximum—. No hyperparameter tuning was performed since our approach is very resilient to a wide array of different hyperparameters. All experiments have been carried out in an Intel Xeon Gold 5220 equipped with a Tesla V100S. We implemented our approach in PyTorch 1.8.1, relying on Pm4Py [37] for processing the event logs. We performed a two-stage statistical comparison to reduce the number of statistical tests to make and to ease the interpretation of the results. We decided to use Bayesian statistical tests instead of the classical Null Hypothesis Statistical Tests (NSHT) because they are easier to interpret and are more powerful in quantifying the differences between the approaches. First, a Bayesian approach to rank models based on the Plackett-Luce model [38] is applied. Then, a Bayesian hierarchical test [39] between our approach and the other two best approaches according to the previous ranking is performed. For this latter statistical test, the region of practical equivalence (ROPE) in which two approaches are statistically equal, must be defined. Following [39], a 1% accuracy value has been used as ROPE. Furthermore, in this test, we consider that one approach significantly outperforms another if the probability of obtaining the best result is greater than 95%. Note that the usefulness of the hierarchical Bayesian test resides in that the test uses the full accuracy results from the individual folds of the crossvalidation testing technique, so it accounts for the fact that one approach can perform badly in one fold but very well on the rest. We rely on the R library scmamp to perform these statistical tests [40]. 5.2 Results TABLE 5 shows the results of the evaluation. TACO obtains the best result in 7 of the tested event logs, the second-best result in one, and the third-best in two. In particular, the highest differences with the best approach for each event log are achieved in BPI-2013-I,BPI-2013-C-P, and in BPI-2012W. In the other logs, such as BPI-2012,BPI-2012-C, and Envpermit, the difference is less noticeable. On the other hand, TACO underperforms in the BPI-2012-A log, BPI-2012-O, and in the BPI-2012-W-C. TABLE 6 shows the Bayesian Plackett-Luce model results for ranking the algorithms. Our approach obtains the best rank overall and the highest probability of being the best approach, with a difference of 24.8percentage points over the second. Fig. 6 shows the credible intervals —5% and 95% quantiles— as well as the expected probability of winning for every tested approach. Note that a non-overlapping pair of approaches means that they are statistically different. The credible interval of the GRNN approach does not overlap any other one, which shows that our approach is statistically different in terms of ranking from the non-overlapping ones. TABLE 7 highlights the differences between TACO and the other approaches from the state of the art, where a positive/negative difference means that TACO outperforms/underperforms the other state-of-the-art approach with which is compared to. In order to make the comparison consistent with the hierarchical statistical tests, we adopt the same ROPE strategy in which a difference of less than 1% accuracy is considered as if the approaches were equal. In general, the differences are overwhelmingly positive towards TACO. In particular, TACO outperforms the other approaches from the state of the art in 96 cases, is equal in 10 cases, and worse only in 4 cases, which correspond to the approaches of Theis (w/ attr) [18] and Hinkka [10]. On the one hand, Theis (w/ attr) outperforms TACO in BPI-2012-W-C by 9.64%, but TACO outperforms it in every other dataset, obtaining the largest differences in BPI-2013-I (26.22%) and BPI-2012-A (13.41%). Furthermore, Theis (w/o attr) outperforms TACO in BPI-2012-W-C by 5.86%, but fall short in every other dataset, especially in BPI-2013-I (18.31%) and BPI-2012-A (13.58%). On the other hand, our approach outperforms Hinkka [10] in 6 event logs, obtains an equal result in BPI-2012 and BPI-2012-C, and underperforms [10] in BPI-2012-A and BPI-2012-O. In general, TACO outperforms the other approaches, but in event logs with a simple process model, namely, the BPI-2012-A and the BPI-2012-O. In these cases, a recurrent network-based model is sufficient to capture the dependencies between the events. These results confirm that the graph-based approach is better suited to capture more complex behavioral patterns that are usually not identified when the learning is performed by shallower models such as LSTMs. This feature improves the prediction of the next activity when either loops or parallels are present in the log. This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 16 [22] A. Graves and et al., “Hybrid computing using a neural network with dynamic external memory,” Nature, vol. 538, no. 7626, pp. 471–476, oct 2016. [23] A. Jalayer, M. Kahani, A. Beheshti, A. Pourmasoumi, and H. R. Motahari-Nezhad, “Attention mechanism in predictive business process monitoring.” IEEE, oct 2020. [24] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2015. [25] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” in Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 2017, pp. 933–941. [26] M. Daniluk, T. Rockt¨ aschel, J. Welbl, and S. Riedel, “Frustratingly short attention spans in neural language modeling,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. [27] I. J. Goodfellow and et al., “Generative adversarial nets,” in Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, 2014, pp. 2672–2680. [28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS 2017), 2017, pp. 5998–6008. [29] B. van Dongen, “Bpi challenge 2012,” 2012. [30] M. Kirchmer, High Performance Through Business Process Management. Springer International Publishing, 2017. [31] J. Desel and W. Reisig, “Place/transition petri nets,” in Lectures on Petri Nets I: Basic Models. Springer Berlin Heidelberg, 1998, pp. 122–173. [32] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in 5th International Conference on Learning Representations, ICLR 2017, 2017. [33] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, and M. Welling, “Modeling relational data with graph convolutional networks,” in The Semantic Web. Springer International Publishing, 2018, pp. 593–607. [34] L. Ruiz, F. Gama, and A. Ribeiro, “Gated graph recurrent neural networks,” IEEE Transactions on Signal Processing, vol. 68, pp. 6303– 6318, 2020. [35] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. [36] A. Augusto, R. Conforti, M. Dumas, M. L. Rosa, and A. Polyvyanyy, “Split miner: automated discovery of accurate and simple business process models from event logs,” Knowledge and Information Systems, vol. 59, no. 2, pp. 251–284, may 2018. [37] A. Berti, S. J. van Zelst, and W. M. P. van der Aalst, “Process mining for python (pm4py): Bridging the gap between processand data science,” 2019. [38] B. Calvo, J. Ceberio, and J. A. Lozano, “Bayesian inference for algorithm ranking analysis,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion. ACM, jul 2018. [39] A. Benavoli, G. Corani, J. Demsar, and M. Zaffalon, “Time for a change: a tutorial for comparing multiple classifiers through bayesian analysis,” J. Mach. Learn. Res., vol. 18, pp. 77:1–77:36, 2017. [40] B. Calvo and G. Santaf´ e, “scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems,” The R Journal, vol. 8, no. 1, pp. 248–256, 2016. [41] N. Srivastava and et al., “Dropout: a simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929–1958, 2014. This article has been accepted for publication in IEEE Transactions on Knowledge and Data Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TKDE.2023.3286017 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: UNIVERSIDADE DE SANTIAGO. Downloaded on June 17,2023 at 19:18:01 UTC from IEEE Xplore. Restrictions apply.