Exploring the balance between interpretability and performance with carefully designed constrainable Neural Additive Models

Information Fusion 99 (2023) 101882 Available online 12 June 2023 1566-2535/© 2023 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Contents lists available at ScienceDirect Information Fusion journal homepage: www.elsevier.com/locate/inffus Full length article Exploring the balance between interpretability and performance with carefully designed constrainable Neural Additive Models Ettore Mariottia,∗, José María Alonso Morala, Albert Gattb aCentro Singular de Investigacion en Tecnoloxias Intelixentes (CiTIUS), Universidade de Santiago de Compostela, Santiago de Compostela, Spain bUtrecht University, Utrecht, The Netherlands ARTICLE INFO Keywords: Generalised additive models Explainable Artificial Intelligence Interpretable modelling Neural additive models Interpretability Explainability ABSTRACT The interpretability of an intelligent model automatically derived from data is a property that can be acted upon with a set of structural constraints that such a model should adhere to. Often these are in contrast with the task objective and it is not straightforward how to explore the balance between model interpretability and performance. In order to allow an interested user to jointly optimise performance and interpretability, we propose a new formulation of Neural Additive Models (NAM) which can be subject to a number of constraints. Accordingly, our approach produces a new model that is called Constrainable NAM (or just CNAM in short) and it allows the specification of different regularisation terms. CNAM is differentiable and is built in such a way that it can be initialised as a solution of an efficient tree-based GAM solver (e.g., Explainable Boosting Machines). From this local optimum the model can then explore solutions with different interpretability-performance tradeoffs according to different definitions of both interpretability and performance. We empirically benchmark the model on 56 datasets against 12 models and observe that on average the proposed CNAM model ranks on the Pareto front of optimal solutions, i.e., models generated by CNAM exhibit a good balance between interpretability and performance. Moreover, we provide two illustrative examples which are aimed to show step by step how CNAM works well for solving classification tasks, but also how it can yield insights when considering regression tasks. 1. Introduction Today’s society is accumulating more and more data whose use can optimise existing processes and bring wealth and knowledge. This motivates a need for tools that can process these data according to the specific tasks at hand. Machine Learning (ML) and Artificial Intelligence (AI) recently gained lots of success thanks to greater performance with respect to traditional approaches on many benchmark problems [1,2]. The ML and AI approach consists in specifying the end goal and constraints to be satisfied and the programme automatically adjust itself to satisfy those. Unfortunately these models, while effective, are typically not understandable by humans. This is because their main job is only to maximise predictive generalisation, without taking into account the intelligibility of the model itself. The lack of interpretability of some AI-based systems is an issue for a number of applications for which quality assurance, trust, legal liability and adherence to ethical principles is a must. Motivated by this, a lot of work has been done in the field of eXplainable Artificial Intelligence (XAI) [3,4], a collective effort to explain the behaviour ∗Corresponding author. E-mail address: [email protected] (E. Mariotti). and predictions of ML models to allow debugging, oversight, auditing, knowledge discovery, and safe democratisation of these new powerful information-processing technologies in many domains [5,6]. In this paper we define explainability as the ability of a system to give an explanation to the user, that is a report of (part of) the causal reasoning that lead to a particular outcome. We refer to interpretability instead of the less demanding property of a system of being inspectable in its parts in a meaningful way. In this sense, interpretability is a prerequisite for explainability. XAI has evolved in different directions based on the requirements stated by the stakeholders of the explainable systems. For example, in order to build tools that are as widely applicable as possible some researchers have prioritised the development of ‘‘black-box explainers’’ [7]. In this context, no assumption is made on the model and we are only left with the input–output relationship. In that case we have to rely on post-hoc approaches. These typically involve approximating the original model in an appropriate neighbourhood of data with an interpretable model (a so-called white-box) and then inspecting that to provide explanations, as suggested by [8,9] (this approach is also https://doi.org/10.1016/j.inffus.2023.101882 Received 17 October 2022; Received in revised form 2 May 2023; Accepted 6 June 2023 Information Fusion 99 (2023) 101882 2 E. Mariotti et al. known as surrogating). The surrogation paradigm is a very powerful one as it allows in principle to explain any black-box model, even deep neural networks, as long as the surrogating models are powerful enough to act like them. In virtue of this, if we want to enhance the explanation of black boxes, we first need good interpretable models by design so that we can later use them for surrogating. To put it in other terms: •we are interested in explaining black box models as these are widely used; •interpretability is a prerequisite for explainability, by the argument stated above; •we should aim for models which are interpretable because these make ideal surrogate models for explaining other models. This opens up the question of what it really means for a model to be interpretable. While with an imperative-style approach (where the programme is hard-coded by a human) the interpretability is kept in each sub-component, with the ML-style approach (as it is usually employed) the only solutions explored are those that solve the task regardless of the final form of the model. A way of leveraging the power of the ML approach is to guide the search of models not only based on their performance but also on some other targets that reflect their interpretability. In this sense we can imagine that an appropriate regularisation term into the target loss could constrain the optimisation of both dimensions (i.e., interpretability and performance). Just as the appropriate measure for predictive performance varies from task to task, the appropriate constraints for interepretability are also application-dependent (usually penalise complexity and favour sparse representations) and should iteratively be refined with domain experts [10]. Generalised Additive Models (GAM) and Neural Additive Models (NAM) are well-studied classes of models that have powerful predictive performance while retaining an interpretable structure. This work addresses a new formulation for fitting GAM, named Constrainable Neural Additive Model (CNAM), such that appropriate interpretable constraints can be enforced to jointly maximise performance and interpretability. In order to benchmark this model against other approaches we use a novel interpretability metric that is called SHAP-Length [11], which exploits the well-known SHapley Additive exPlanations (SHAP) first introduced by [9]. The main contributions in this work are as follows: •a novel formulation of NAM, CNAM, such that interpretablilty constraints can be enforced and jointly optimised alongside taskrelated performance metrics; •an extensive benchmark of CNAM on 56 binary classification datasets against 13 different models; •an illustrative use-case of CNAM on a classification dataset; •an illustrative use-case of CNAM on a regression dataset. The rest of the manuscript is organised as follows. In Section 2, we discuss related work in the field. In Section 3, we describe the structure and properties of CNAM. In Section 4, we experiment first with a benchmark study for explicitly exploring CNAM interpretability-performance tradeoff on binary classification tasks and then go deeper with two illustrative use cases: an astrophysics classification task and a socioeconomic regression task. Finally in Section 6, we draw conclusions and delineate future work. 2. Related work The history of learning from data can be traced back to early work in the 19th century where, motivated by the desire of predicting astronomical data and minimising reconstruction errors, theoretical foundations and closed-form solutions for Linear Models (LM) were developed, which minimised the Mean Squared Error (MSE) as proposed by [12,13]. The target 𝑦is modelled as 𝑦 from a set of features 𝑥𝑖and the task is to find a set of real coefficients 𝛽𝑖such that 𝑦 =∑ 𝑖 𝛽𝑖𝑥𝑖 and the MSE= 1∕𝑁∑𝑁 𝑗(𝑦𝑗−𝑦𝑗)2, is minimised. Much later, [14] unified some scattered views of modelling a certain target with a LM and introduced Generalised Linear Models (GLM), where the target 𝑦is transformed with the so-called link function 𝑔(⋅) with static coefficients 𝛼𝑖such that 𝑔(𝑦) = ∑ 𝑖 𝛼𝑖𝑥𝑖 This new modelling allows (among other things) to have linear classifiers by choosing the logistic sigmoid as the link function, leading to what is known as Logistic Regression (LR). A further generalisation of LM was developed at Bell Labs by [15, 16] where the condition of having a static coefficient 𝛼𝑖is relaxed and the relationship is allowed to be a non-linear function 𝑓𝑖(⋅)of the univariate feature such that 𝑔(𝑦) = ∑ 𝑖 𝑓𝑖(𝑥𝑖) This new formulation is appealing because it is more expressive than GLM while keeping a relatively simple and understandable structure. When the model is fitted one can indeed easily visualise each 𝑓𝑖 (also called a shape function) as a function of the 𝑥𝑖values, naturally providing both a global view of the behaviour of the model on a dataset and a local explanation for the prediction of a single data point. From a practical point of view, initially the 𝑓𝑖were based on Regression Splines of degree 𝑑of the form 𝑓𝑖(𝑥𝑖) ∶= ∑𝑑 𝑘=1 𝛽𝑘𝑏𝑗(𝑥𝑖) (Spline-GAM) or other Kernel Expansions of the feature 𝑥𝑖. On the one hand, this allowed the injection of expert knowledge when designing the model. On the other hand, the fitting procedure was slow and sometimes did not converge properly. These fitting methods were then shown to be outperformed by [17] with a procedure of bagging and boosting binary decision trees leading to a class of models called Explainable Boosting Machines (EBM). For optimisation of speed and memory EBMs compress the representation of the function 𝑓𝑖with a lookup table, a data structure that bins the feature values and maps each bin to the height of the shape function. EBMs were then further generalised to allow also pairwise interactions (EB2M) of the form 𝑓(𝑥𝑖, 𝑥𝑗)by [18] and proved to be of value with an application in healthcare [19]. A parallel development has attempted to model the shape functions with neural networks. [20] pioneered the work with the so-called Generalised Additive Neural Networks (GANN), where each 𝑓𝑖was represented as a small neural network. The optimisation process followed an iterative approach and did not make use of backpropagation. GANN were successfully used for example with the aim of improving the performance of credit scoring applications [21]. Recently [22] proposed Neural Additive Models (NAM), a modern reinterpretation of GANN with more neurons, a new activation function ExU and a sophisticated training procedure that included dropout, weight decay, output penalty and feature dropout. They reported competitive task performance compared to EBMs and other models on four datasets. It is worth noting that NAM can be a building block for other models, for example as the backbone of autoencoders. The main drawback of NAM is the need for a careful setup of many hyperparameters and the long training time that is required for the convergence of these models. Moreover with their formulation it is more difficult to introduce expert knowledge in the system and to explicitly require a part of a shape function to have a specific form. In an effort to put the human in the loop, [23] built GAM-Changer, a framework where fitted GAM can be inspected and changed when deemed appropriate. This enables a domain expert to interact with the system and give some tools on how to correct the model where few Information Fusion 99 (2023) 101882 3 E. Mariotti et al. Table 1 Short summary of related models in the literature. CNAM is the new model proposed in this work. LM LR GLM Spline-GAM EBM EB2M NAM CNAM* Classification tasks x x x x x x x Regression tasks x x x x x x x Can be backbone of Autoencoders x x x Can use prior knowledge on target x x x x x x Can use prior knowledge on data x x Pairwise interactions x Arbitrary initialisation x x x x x Constrainable x x x Explicitly balancing x interpretability-performance Fig. 1. Schematic view of the structure of CNAM (for classification) and how constraints are inserted. For regression the model is the same with the difference of not having a logistic sigmoid non-linearity at the end. statistics are available but human knowledge can be of help. While the user does receive feedback on the performance of the modified model, there is currently no way of fine-tuning the changed model keeping fixed the human interventions. With the aim of overcoming drawbacks of current state of the art models, in this work we introduce the new CNAM, a way of building GAM based on neural networks but with a specific structure such that it is possible to initialise each shape function as an arbitrary curve and to enforce constraints on parts of the network. This is important because it allows us to: 1. leverage the power and speed of EBM which quickly converges to a good solution; 2. leverage the expressiveness of differentiable programming, opening the possibility of jointly optimising performance and interpretability-related metrics. To conclude this section, Table 1 summarises the main properties of the most outstanding related models in the literature. As we will see in the rest of the paper, CNAM is a promising tool for human-in-the-loop solutions and can in principle be integrated into GAM-Changer enabling a complete feedback-loop between a human and a machine. 3. Model structure We can graphically represent CNAM as illustrated in Fig. 1. The important novelty of CNAM is the construction of the shape functions 𝑓𝑖as a sum of carefully initialised differentiable versions of the unit step, which we will call Differentiable Step (DS). It is worth noting that only a few shape functions are expected to take non-zero values, thus effectively discarding some features from the final score by Regularisation (Few Features Selected). The parametrisation of the DS is what will allow CNAM to be initialised to any arbitrary shape, in particular this property will be used to initialise the model to the solution of EBM (more on this in Section 3.2). Once initialised, the structure of CNAM is transparent enough that each component has a clear interpretation and this facilitates the formal definition of appropriate regularisers for the problem at hand. As introduced in Section 1, having a model that is flexible enough to enforce different constraints (e.g., sparsity) is key for targeting specific interpretability needs of different use cases. An enumeration of different desirable constraints will be the objective of the following section. 3.1. Desirable constraints We now proceed to list desirable constraints that could be useful in different situations. These can be appropriate when prior knowledge exists and can help the model to generalise better where data is scarce: •Monotonicity of the shape functions (e.g., in a banking context we might want to enforce that the bigger the ‘‘Loan amount term’’ the lower should be the probability of granting the loan). More formally: for any 𝑥𝑖,𝑥𝑗such that 𝑥𝑖≤𝑥𝑗we have that 𝑓(𝑥𝑖)≤𝑓(𝑥𝑗)(positive monotonicity) or 𝑓(𝑥𝑖)≥𝑓(𝑥𝑗)(negative monotonicity). •Smoothness of the shape functions. This could be useful as a smooth function is simpler to describe compared to a jagged one (jagged representations are also more likely to be influenced by noise in the data). A user might be interested to explore if a smooth relationship between the feature and its target reasonably model the task at hand. Moreover, since a smooth curve is easier to describe in words than a jagged one, such constraint becomes then useful when the objective is to use natural language in order to provide explanations [24,25]. Information Fusion 99 (2023) 101882 4 E. Mariotti et al. Fig. 2. The effect of changing the parameters of the differentiable steps and how they can form a 𝑓𝑖. •Few shape functions (useful for problems where the number of features is so large that would be impractical to manually assess every shape function). In other words this entails having a sparse representation following the principle of Ockam’s razor: ‘‘if two models describe the same event, the model with less assumptions (i.e., the model that relies on fewer variables) is preferable’’. It is possible to mathematically enforce some sparseness of a linear combination of terms 𝑥parametrised by the dot product with 𝑤resulting in 𝑤.𝑥 by minimising the 𝐿1Norm of 𝑤, that is: 𝑚𝑖𝑛 ∑𝑖|𝑤𝑖| •Local freezing some part of the shape function (e.g., to help optimise fairness measures). For example if we had a categorical variable describing some protected attribute one might want to enforce that the score for the protected attribute match some formal description of fairness. It is important to note that the desirable constraints listed above are not an exhaustive list and are context-dependent. Furthermore, some of these constraints may conflict with maximising task performance or even among themselves. For instance, while enforcing monotonicity of a shape function may improve the model’s interpretability, it may negatively impact its task performance. Similarly, enforcing sparsity of shape functions may help with interpretability, but at the cost of some loss in predictive accuracy. Another approach that could be considered is to measure the agreement of feature importances as indicated by a human expert with what the model currently predicts. While this could help the model quickly converge to a well-generalising solution, it should be kept in mind that human ranking could introduce biases that are not present in the data. Developing a suitable metric for measuring such agreement is a topic for future research. Despite these considerations, the main motivation behind this novel approach is that once the desirable constraints are defined, the optimisation procedure can explore different solutions that aim to satisfy all the given constraints. In the following section, we will discuss how to precisely define each component of CNAM and how to define appropriate regularisers that enforce the desirable constraints. 3.2. Model construction CNAM specifies the structure of how each shape function can be constructed in a rather straightforward way. Intuitively, we develop the idea that each shape 𝑓𝑖is a sum of 𝑗step functions, an approximation of indicator functions 𝑓𝑖(𝑥𝑖) = ∑ 𝑗 𝑠𝑡𝑒𝑝𝑗(𝑥𝑖) This way the model can be rewritten as 𝑔(𝑦) = ∑ 𝑖∑ 𝑗 𝑠𝑡𝑒𝑝𝑗(𝑥𝑖) As introduced previously, we want 𝑠𝑡𝑒𝑝(⋅)(the DS) to be a soft version of an indicator function that adds or subtracts a specific (learnable) quantity at a certain position in the shape function. We parametrise DS with three parameters: 𝑝𝑜𝑠𝑥,𝑝𝑜𝑠𝑦,𝑠𝑝𝑒𝑒𝑑. DS can then be defined as follows: 𝑠𝑡𝑒𝑝(𝑥𝑖) ∶= 𝑝𝑜𝑠𝑦∗𝜎(𝑠𝑝𝑒𝑒𝑑 ∗ (𝑥𝑖−𝑝𝑜𝑠𝑥)) where •𝜎(⋅)is the Logistic Sigmoid function, that is: 𝜎(𝑥) = 1 1+𝑒−𝑥; •𝑝𝑜𝑠𝑥is the position on 𝑥where the shape is centred (see Fig. 2(a)); •𝑝𝑜𝑠𝑦is the maximum height of the step (see Fig. 2(b)); •𝑠𝑝𝑒𝑒𝑑 is a parameter that specifies how quickly the step grows (see Fig. 2(c)). Information Fusion 99 (2023) 101882 5 E. Mariotti et al. When many carefully initialised DS are added up they can form a curve of arbitrary shape (see Fig. 2(d)). Then, without loss of generality, we can aggregate all the shape functions 𝑓𝑖with a linear layer parametrised by a vector 𝛼to produce the final prediction (log-odds in case of classification), as follows: 𝑔(𝑦) = ∑ 𝑖 𝛼𝑖⋅𝑓𝑖(𝑥𝑖) The previous formulation can be useful for enforcing some regularisation like the 𝐿1norm on 𝛼𝑖in order to favour a sparse representation (i.e., when few shape functions are desired). This formulation is well-suited for binary classification tasks and regression tasks. However, we acknowledge that extending our model to other tasks requires additional considerations. One possible approach for non-binary classification is to use a one-vs-rest fashion, where a separate model is trained for each class. This approach increases the complexity of the system, but it can be effective for multiclass classification problems. 3.3. Enforceable constraints With this formulation we can now begin to impose some of the constraints discussed in Section 3.1 by specifying additional loss terms to be jointly optimised or by imposing some structural changes. Here’s how the list of possible enforceable constraints enumerated in Section 3.1 can be implemented: •Monotonicity of the shape functions: This can be achieved by imposing 𝑝𝑜𝑠𝑦and 𝑠𝑝𝑒𝑒𝑑 to be always greater (or smaller) than zero. This can be done by applying the function 𝑚𝑎𝑥(𝑥, 0), also known as ReLU [26,27], to them. •Smoothness of the shape functions: This can be achieved by incentivising small values for the 𝑠𝑝𝑒𝑒𝑑 parameters of the various DS of a given shape function. This can be done by minimising the 𝐿2norm of the 𝑠𝑝𝑒𝑒𝑑, that is to add to the final loss function the component 𝜆√∑𝑖𝑠𝑝𝑒𝑒𝑑2 𝑖. The parameter 𝜆controls the amount of regularisation. Notice that with this formulation we can have smooth functions that are still able to model big jumps if necessary, something that spline-based GAM failed to achieve because the splines implicitly encode a prior of global smoothness. On the other hand, enforcing an 𝐿2on the 𝑠𝑝𝑒𝑒𝑑 still allows local speeds to be very high if appropriate. •Few shape functions: this can be enforced by imposing sparsity (i.e., 𝐿1regularisation) on the final linear layer that aggregates the shape functions 𝑓𝑖. That is, we can represent without loss of generality the models as 𝑔(𝑥) = 𝛼1∗𝑓1(𝑥1) + ⋯+𝛼𝑝∗𝑓𝑝(𝑥𝑝) and then incentivise a sparse representation by adding to the final loss the 𝐿1norm of 𝛼:𝜆2∑𝑝 𝑖|𝛼𝑖|, where 𝜆2controls the amount of regularisation. This is similar to what [28] proposed with Lasso, but it is now possible to apply it to close-to-optimal EBM solutions, something that up to now was not possible to do. •Local freezing: we can exclude parts of the model from the optimisation process by setting the gradient of a certain step to 0. This prevent any further updates on that parameter and thus any change on that part of the function. 3.4. Parameter initialisation As already anticipated earlier, one of the key features of CNAM is the ability to be initialised as solutions of faster solvers (e.g. EBM) easily. This is important because if we try to optimise the model from a typical random initialisation (e.g., following the principles delineated by [29]) we observed empirically that the training can be unstable and result in suboptimal solutions with respect to EBM. On the other hand if we optimise the model while starting from the solution found by EBM, then CNAM is able to end up finding solutions that have similar Fig. 3. Example of applying algorithm 1to some random dummy data with the aim of initialising CNAM as a good approximation of EBM. or higher performance while also optimising the interpretability constraints (thus finding a solution that exhibits a better balance between performance and interpretability). This will be shown later with the experiments in Section 4. For the specific initialisation we set the initial value of speed to a large value (we found 100 to be heurisically a good guess) to better approximate the unit step. The translation parameter 𝑝𝑜𝑠𝑥can be set as the 𝑥-positions of the lookup table of the shape function of EBM. The height of the shape function (described by 𝑝𝑜𝑠𝑦) is finally set to the 𝑦-position of the EBM by first initialising all the 𝑝𝑜𝑠𝑦to 0and then adjusting the 𝑝𝑜𝑠𝑦from the right-most to the left-most using Algorithm 1. An illustration of how this algorithm works can be seen in Fig. 3. That said, the random initialisation can still be useful for some use-cases where it is not straightforward to train an EBM (e.g. as components of an Autoencoder). We still suggest to initialise the parameter 𝑝𝑜𝑠𝑥for shape 𝑖as the quantiles of the univariate distribution of feature 𝑖. On the other hand for classification and regression purposes we consistently found in our preliminary experiments that EBM already converges to a close-to-optimal solution and initialising CNAM to that vastly improves the final model across both performance and intelligibility metrics. Algorithm 1 Initialise CNAM as EBM Require: 𝑦𝐸𝐵𝑀 𝑦𝐶𝑁𝐴𝑀 ←𝑦𝐸𝐵𝑀 ∗ 0 ⊳Initialise all as zero 𝑦𝐶𝑁𝐴𝑀 [−1] ←𝑦𝐸𝐵𝑀 [−1] for 𝑘= 2, 𝑘 ≤𝐿𝑒𝑛𝑔𝑡ℎ(𝑦𝐸𝐵𝑀 )+1, 𝑘 + + do 𝑦𝐶𝑁𝐴𝑀 [−𝑘]←𝑦𝐸𝐵𝑀 [−𝑘]⊳Set the height to the desired one 𝑦𝐶𝑁𝐴𝑀 [−𝑘+ 1] ←𝑦𝐶𝑁𝐴𝑀 [−𝑘+ 1] − 𝑦𝐶𝑁𝐴𝑀 [−𝑘]⊳fix the effect on the following bin end for return 𝑦𝐶𝑁𝐴𝑀 3.5. Optimisation Once we have initialised the CNAM parameters, we can optimise the final loss, which includes both task-performance and taskinterpretability terms, using a gradient-based optimiser such as ADAM Information Fusion 99 (2023) 101882 6 E. Mariotti et al. Fig. 4. A benchmark of the performance-interpretability tradeoffs regarding different models on 56 datasets. The 𝑥-axis represents the relative Rank of interpretability (measured as SHAP-Length) while the 𝑦-axis corresponds to the relative Rank of performance (measured as ROC AUC, Accuracy, F1-Score). The best models lie in the lower-left part of the plots. The dotted blue lines highlight the Pareto front, i.e. the set of non-dominated solutions. Various instantiations of CNAM on average lie on that Pareto front. Interestingly, even without regularisation, CNAM scores better than EBM (without pairwise interactions). Fig. 5. A box plot of the results of the benchmark for the different models across the different statistics. Information Fusion 99 (2023) 101882 7 E. Mariotti et al. Fig. 6. Performance-Interpretability tradeoff on the MAGIC Telescopes dataset. CNAM configurations are coloured by the amount of 𝜆1regularisation, with blue indicating high regularisation and green representing low regularisation. More interpretable models with lower SL (x-axis) and better performance (y-axis) are situated in the upperleft corner of the figure. It is important to note that 𝜆2also influences the results, but we have omitted its visualisation for the sake of clarity. [30]. This technique allows us to iteratively adjust the parameters of CNAM towards a solution that minimises the target loss, even without an explicit closed-form formula for it. To control for overfitting, we periodically evaluate against a validation set and employ an early stopping criterion [31]. 4. Experiments CNAM can be used both for binary classification or regression. We first benchmark the model with different regularisation parameters on 56 binary classification datasets (see Section 4.1). Subsequently, we perform a deep dive into two specific datasets, a classification task (see Section 4.2) and a regression task (see Section 4.3), to provide a more in-depth analysis of the model’s behaviour. We implemented CNAM as an open-source software (available online at https://gitlab. nl4xai.eu/ettore.mariotti/cnam), using Python and pytorch [32] as the deep learning backend infrastructure, structuring the code using pytorch-lightning library. In our experiments, we explore the tradeoff between performance and interpretability by varying the weights of the weighted loss function according to the specific application domain. Although we lack theoretical convergence guarantees, our empirical observations suggest that by trying different coefficients, we can obtain a diverse set of solutions that effectively navigate the performance-interpretability tradeoff. Later in the subsections, we demonstrate this practically with different choices of regularisation parameters as the added term for the loss. 4.1. Binary classification benchmark In this section we focus our attention to the task of binary classification and we benchmark CNAM against other classifiers across 56 tabular datasets taken from [33]. We measure both task-performance and task-interpretability scores. The datasets are related to different domains and include both real-data as well as simulated-data and spans both in size and class imbalance. The basic properties of each dataset are reported in Table 2. In order to compare the results we rank the models for each dataset sorting them with respect to the specific metric value they scored. Once this ranking is obtained, we average the rank across all the datasets. We selected three measures of classification performance: •Accuracy: the fraction (percentage) of correctly classified instances. •F1-Score: the geometric mean of precision and recall. •ROC AUC: the area under the receiver operating characteristic curve (i.e., under the ROC curve), that is the curve of true positives versus false positives at all classification thresholds. For measuring interpretability in terms of model complexity we opted for the so-called SHAP-length metric [11] which is a modelagnostic metric which allows us to compare heterogeneous models. This metric returns the number of SHAP attributions of each data point such that the set of attributions 𝜙𝑖captures a given fraction (we settled at 90% here) of the overall explanation mass ∑𝑖|𝜙𝑖|. More formally: 𝑆𝐿90% ∶= the smallest 𝑖such that ∑𝑖sorted(|𝜙𝑖|) ∑𝑖|𝜙𝑖|≤0.9. This intuitively captures the length of the ‘‘compressed’’ explanation of each prediction, that is what a user would have to read in order to get a rough sense of what is the impact of different features. These lengths are then averaged across all the data points of the dataset in order to return a single interpretability score per dataset-model pair. It is important to note that SL might not always be the ideal measure of interpretability. Indeed, the best interpretability metric depends on the context, user preference, and the specific domain of the task. A model with a lower SL will typically have more sparse explanations, so the lower the SL, the less cognitive load burden on the user. This is a key consideration when choosing an interpretability metric, as the goal is to provide users with concise yet meaningful explanations that help them understand the model’s behaviour and decision-making process. As stated before, given the scores of each model on each dataset we can rank each model on the same dataset according to the various metrics. Once we have computed the ranking of all the metrics for all the models across all the datasets we can average them and delineate the Pareto front with the best solutions. The Pareto front is the set of all the Pareto-efficient solutions, that is all those solutions in multiobjective problems where no other solution is better than them in one of the objectives. We tested CNAM against a pool of 12 different models, most of them implemented by the scikit-learn package [34]: Explainable Boosting Machines with pariwise interactions (EB2M) and without interactions (EBM-GAM), rf100 (Random Forest with 100 trees), rf1k (Random Forest with 1000 trees), svm (Support Vector Machine), XGB (eXtremely Gradient Boosting trees), lr (Logistic Regression), small-tree (a decision tree with maximum tree depth of 4), big-tree (a decision tree unconstrained), 3-nnc, 5-nnc, 10-nnc (respectively 3-, 5-, and 10nearest neighbours classifiers). In order to evaluate the capabilities of CNAM we evaluated 3 different variations of the proposed model, each with different regularisation coefficients. The regularisation was set up such that a sparsity constraint (𝐿1norm on the 𝛼) is added to the classical cross entropy loss: 𝑙𝑜𝑠𝑠 =𝑐𝑟𝑜𝑠𝑠𝑒𝑛𝑡𝑟𝑜𝑝𝑦 +𝜆1∗∑𝑖|𝛼𝑖|. This way 𝜆1becomes a macroscopic hyperparameter that favours model simplicity at the potential cost of task performance. We benchmarked CNAM with 𝜆1= 0,𝜆1= 0.1, 𝜆1= 1 in order to show how CNAM behaves with no regularisaion, small regularisation and strong regularisation. Every model is evaluated with 5-fold stratified cross validation. For each single dataset the models are ordered from the best to the worst thus producing a ranking (lower values thus correspond to better models). The ranking is then averaged across all the datasets and the results are reported in Figs. 4 and 5. Fig. 4 reports for the sake of clarity and readability only the mean of the computed rankings. In Fig. 5 the interested reader can see further details in the form of box plots that help understand what is the distribution of the rankings. The results suggest that CNAM unconstrained is equivalent to (if sometimes marginally better than) EBM-GAM. This makes sense as by design CNAM is initialised as a good approximation of EBM-GAM. Information Fusion 99 (2023) 101882 8 E. Mariotti et al. Table 2 Basic properties of each dataset tested in our benchmark study. Dataset # Instances # Features Class imbalance analcatdata aids 50 4 0.000000 analcatdata asbestos 83 3 0.011758 analcatdata bankruptcy 50 6 0.000000 analcatdata boxing1 120 3 0.090000 analcatdata boxing2 132 3 0.005739 analcatdata creditscore 100 6 0.211600 analcatdata cyyoung8092 97 10 0.255181 analcatdata cyyoung9302 92 10 0.344518 analcatdata fraud 42 11 0.145125 analcatdata japansolvent 52 9 0.001479 analcatdata lawsuit 264 4 0.732840 appendicitis 106 7 0.364543 australian 690 14 0.012132 backache 180 32 0.521605 biomed 209 8 0.079691 breast 699 10 0.096375 breast cancer 286 9 0.164507 breast cancer wisconsin 569 30 0.064940 breast w 699 9 0.096375 buggyCrx 690 15 0.012132 bupa 345 5 0.000412 clean1 476 168 0.016966 cleve 303 13 0.007940 colic 368 22 0.068053 corral 160 6 0.015625 credit a 690 15 0.012132 crx 690 15 0.012132 german 1000 20 0.160000 glass2 163 9 0.004554 heart c 303 13 0.007940 heart h 294 13 0.077792 heart statlog 270 13 0.012346 house votes 84 435 16 0.051795 hungarian 294 13 0.077792 irish 500 5 0.012544 labour 57 16 0.088950 lupus 87 3 0.038182 molecular biology promoters 106 57 0.000000 monk1 556 6 0.000000 monk2 601 6 0.098895 monk3 554 6 0.001577 mux6 128 6 0.000000 parity5 32 5 0.000000 pima 768 8 0.091254 prnn crabs 200 7 0.000000 prnn synth 250 2 0.000000 profb 672 9 0.111111 saheart 462 9 0.094470 sonar 208 60 0.004530 spect 267 22 0.345762 spectf 349 44 0.207560 threeOf9 512 9 0.004944 tic tac toe 958 9 0.094181 vote 435 16 0.051795 wdbc 569 30 0.064940 xd6 973 9 0.114332 Instead, when a regularisation of the form discussed above is applied, we can see how the different models actually loose classification performance but gain in interpretability, returning models on the Pareto front of the best tradeoffs. This is important because it is an indication that the model cannot only retain competitive task performance but can successfully explore the tradeoff between competing constraints in a way that the result lie on the set of Pareto-efficient solutions. Moreover, CNAM reg 1 turns up as the second best from the point of view of interpretability (see the plot related to ‘‘Explanation Length’’ on the bottom right side of Fig. 5), only behind small-tree while it is much better from the performance viewpoint (i.e., CNAM reg 1 is always better ranked than small-tree in the other pictures in Fig. 5). Finally, it is worth noting that in a specific application a user should experiment with different regularisation parameters (and also specific regularisations fit to the specific domain) according to his or her needs. 4.2. Illustrative example on how to address a classification task: MAGIC dataset In order to gain more intuition of the behaviour of CNAM on a specific dataset we provide as an application the use of CNAM for the MAGIC Telescopes classification dataset [35], a real-world dataset (taken from the UCI repository [36]) that is much larger than all of those included in the set of the previous benchmark study. In this dataset the task is to distinguish between two different kind of particles (gamma-like vs hadrons) given a representation of the image captured by the telescope. From a physical point of view, a particle enters the atmosphere and produce an elliptic flash of light that is captured by the camera of the telescope. The image is then described by the following 10 numerical features, some of which describe an ellipsoidal shape fitted on the pixels: Information Fusion 99 (2023) 101882 9 E. Mariotti et al. Fig. 7. Visual representation of each shape function of CNAM on the MAGIC Telescopes dataset. Each shape function demonstrates the contribution of a given feature value to the final prediction, which is the sum of the partial scores. The blue solid line represents the most interpretable CNAM instance (CNAM best SL), identified as having the shortest Shap Length (SL), while the dotted red line indicates the best-performing instance (CNAM best ROC AUC), which has the highest AUC (Area Under The Curve) ROC (Receiver Operating Characteristics) curve score among models trained with varying lambda1 and lambda2 regularisation parameters. The data distribution is visualised as a histogram in the background, with the counts reported on the right-axis of the plot. •fLength: major axis of ellipse •fWidth: minor axis of ellipse •fSize: log 10 of the sum of content of all pixels (photon count) •fConc: the ratio of sum of two highest pixels over fSize •fConc1: the ratio of highest pixel over fSize •fAsym: distance from highest pixel to centre, projected onto major axis •fM3Long: 3rd root of third moment along major axis •fM3Trans: 3rd root of third moment along minor axis •fAlpha: angle of major axis with vector to origin •fDist: distance from origin to centre of ellipse The dataset has 19020 instances (12332 gamma and 6688 hadron), with a class imbalance of 0.542. In this task the Accuracy metric is not meaningful as classifying a hadron (background) as a gamma (signal) is worse than vice versa (classifying the signal as background). For comparing different classifier a more sensible metric is ROC AUC, as it allows the evaluation of the performances at different classifying thresholds. For illustrative purposes we will fit CNAM with different regularisation parameters, namely 𝜆1(encouraging sparsity, that is the shape functions are more often to 0) and 𝜆2(encouraging smooth shapes), drawing them randomly from a log-uniform distribution 𝑌= log(𝑋) where 𝑋=𝑈(−6,1). Each candidate configuration is evaluated with a 5-fold stratified cross-validation with the ROC AUC score (performance) and 𝑆𝐿90% (interpretability). For comparison we will also fit other competitive models: EBM-GAM (as it is CNAM initialisation), EB2M and Random Forest with 1000 trees (as both models demonstrated high performances on the benchmark study). In Fig. 6 is shown how each model scores in the performance-interpretability space.