Node co-activations as a means of error detection : Towards fault-tolerant neural networks

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ Node co-activations as a means of error detection : Towards fault-tolerant neural networks © 2022 the Authors Published version Myllyaho, Lalli; Nurminen, Jukka K.; Mikkonen, Tommi Myllyaho, L., Nurminen, J. K., & Mikkonen, T. (2022). Node co-activations as a means of error detection : Towards fault-tolerant neural networks. Array, 15, Article 100201. https://doi.org/10.1016/j.array.2022.100201 2022 Array 15 (2022) 100201 Available online 10 June 2022 2590-0056/© 2022 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Contents lists available at ScienceDirect Array journal homepage: www.elsevier.com/locate/array Node co-activations as a means of error detection—Towards fault-tolerant neural networks Lalli Myllyaho a,∗, Jukka K. Nurminen a, Tommi Mikkonen b aUniversity of Helsinki, Finland bUniversity of Jyväskylä, Finland A R T I C L E I N F O Keywords: Machine learning Fault tolerance Neural networks Error detection Concept drift Dependability A B S T R A C T Context: Machine learning has proved an efficient tool, but the systems need tools to mitigate risks during runtime. One approach is fault tolerance: detecting and handling errors before they cause harm. Objective: This paper investigates whether rare co-activations – pairs of usually segregated nodes activating together – are indicative of problems in neural networks (NN). These could be used to detect concept drift and flagging untrustworthy predictions. Method: We trained four NNs. For each, we studied how often each pair of nodes activates together. In a separate test set, we counted how many rare co-activations occurred with each input, and grouped the inputs based on whether its classification was correct, incorrect, or whether its class was absent during training. Results: Rare co-activations are much more common in inputs from a class that was absent during training. Incorrectly classified inputs averaged a larger number of rare co-activations than correctly classified inputs, but the difference was smaller. Conclusions: As rare co-activations are more common in unprecedented inputs, they show potential for detecting concept drift. There is also some potential in detecting single inputs from untrained classes. The small difference between correctly and incorrectly predicted inputs is less promising and needs further research. 1. Introduction Machine learning (ML) models are statistical approximations, whimsical and capricious in nature, and often made for environments that evolve over time. In such approximations, a 99% accurate model – something that is practically always correct – is wrong 1% of the time. What should be done if that 1% happens and causes errors in your system? Are there ways to mitigate the risk and prepare a software system for the inevitable ‘‘bad days’’ of your model? The trustworthiness of ML systems has been improved, for example, by establishing patterns for fault tolerance, but tools for measuring whether a model’s results are and remain trustworthy can still be improved [1]. Furthermore, such detection should ideally not only be an afterthought, but detection should occur in real time while the model is running. During computation runs, one approach to mitigating the risk and making the system more fault-tolerant could be monitoring the model’s own inner structure. The inner structure of a neural network – a currently common ML technique – is sometimes compared to the structure of biological neural circuitry (e.g. Abiodun et al. [2]). Like biological neural circuitry and ∗Corresponding author. E-mail addresses: [email protected] (L. Myllyaho), [email protected] (J.K. Nurminen), [email protected] (T. Mikkonen). neurons, neural networks in computing consist of layers of interconnected nodes. These nodes are tiny computational units that receive an input, and either activate and pass on an output to the following nodes or remain dormant with an output of 0, having no effect on the computations made by the following nodes. We know from previous research on neural networks that after network training, specific groups of nodes tend to be responsible for specific outcomes and thus often activate concurrently [3]. For example, in an image recognition model, certain groups of nodes can be expected to activate when the image of a dog is shown, while at least a partially different group should activate for the image of a cat. Activations have been studied in the context of testing neural networks (e.g. [3–6]), but use in mitigating risks during runtime has been scarce [1]. However, what if the activating nodes are suddenly ones that usually do not activate together and thus do not belong to a same group (cf. Fig. 1)? Can something be inferred from this? Do these rare coactivations within a neural network indicate that the computation result is incorrect or that the input has never been seen before? If so, could rare co-activations be used to detect errors in neural networks, https://doi.org/10.1016/j.array.2022.100201 Received 29 March 2022; Received in revised form 30 May 2022; Accepted 3 June 2022 Array 15 (2022) 100201 2 L. Myllyaho et al. Fig. 1. Illustration of activation patterns in a simple neural network. Colouring indicates an activated node. Mutated activation pattern on the far right. Picture of the dog courtesy of Helen Lopez https://www.pexels.com/photo/short-coated-tan-dog2253275/. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.) prevent them from propagating and causing failures, and, thus, increase the fault tolerance of ML systems to mitigate inherent risks? To test our hypotheses, we train four neural networks. For every node in a neural network, we calculate how often each node activates concurrently with every other node using the training data: i.e. how probable it is that two nodes activate concurrently. This way, we obtain a metric of which nodes often contribute together to the network output and thus belong to one or more of the same groups. Using a separate test data set, we count the number of rare co-activations happening within the neural network for each input and mark down whether the network’s output was correct, incorrect, or if the input belonged to a class that was not present in the training set. Once the tests have been run, we determine whether the number of rare co-activations differ statistically between the following scenarios: (1) test cases for which the output was correct, (2) cases where the output was incorrect, and (3) cases where the network was not trained for the input. Based on our data, we then estimate how well rare co-activations would fit in mitigating three specific major risks that are often present in ML systems: drift in incoming data, single inputs the model cannot handle, and inaccurate predictions [1]. Large numbers of rare co-activations indicate problems in predictions. Rare co-activations are, on average, much more common in inputs from untrained classes than in inputs the model has been trained for. Thus, rare co-activations show good potential in detecting drift in incoming data: should the average number of rare co-activations increase, drift is most likely imminent. However, inputs from trained classes contained outliers with a high occurrence number of rare coactivations as well, and some untrained inputs have a low number of occurrences. Thus, detecting inputs that the model cannot handle and preventing them from being used further down in the system is more problematic. Considering the difference in the average number of occurrences, it may be possible to find systems and contexts where using it is feasible, but the system should be able to deal with some false positives and negatives. Additionally, rare co-activations tended to be more common in incorrectly predicted inputs than in correctly predicted ones, but the difference was both smaller and statistically less significant. Thus, detecting single inaccurate predictions may not be feasible based on the number of occurrences alone, but the approach should at least be fine-tuned to find the most indicative co-activations. This paper is organized as follows: Section 2describes key concepts of system dependability, fault tolerance, and neural networks, along with previous work on activation patterns in neural networks and their utilization in testing and monitoring the networks. Section 3introduces the novel concepts in detail and describes our goals and research questions. Section 4describes our experimental set-up, how data was Fig. 2. A simple neural network, more precisely, a multi-layer perceptron. collected and the methods of analysis. Results can be found in Section 5. Section 6discusses the results, while Section 7discusses the study validity. Section 8concludes the paper. 2. Background 2.1. Dependability and fault tolerance The dependability of a system means its trustworthiness [7]. Dependability is usually assessed by evaluating a system’s reliability, availability, and maintainability. Essentially, a dependable system – at the very least – delivers correct service consistently, does not suffer from long periods of downtime, and is easily corrected and altered. System dependability is threatened by failures,errors, and faults [8]. Failures are deviations from a desired service. They are caused by propagating errors made by the system, i.e. incorrect functioning of the system. Errors are caused by faults that are defects in system components (software or hardware), activated by given inputs in a given state. Fault tolerance is one tool for diminishing these threats [8]. Fault tolerance aims for a system design that can prevent occurring errors from propagating in the system and causing failures by detecting the error and handling it before further damage is done. The need for fault tolerance in ML systems has lately been recognized more [1]. This is in no small part due to the nature of the ML models themselves. According to Myllyaho et al. [1], the problems in ML systems often originate from inaccuracies that the models hold, along with their proneness to socalled concept drift. That is, an ML model is trained and, ideally, it can generalize what it has learned to all data that are similar to the training data. However, the generalization rarely, if ever, is successful enough to reach a 100% prediction accuracy in new data in the first place, and the model rarely is able to handle data that are vastly different from the training data. Thus, ML systems can be seen as inherently faulty because of their approximate nature [9], and an initially adequate system can erode and become faulty over time [10]. To add insult to injury, erroneous behaviour is often difficult to detect [1], which is why we focus on the detection phase of fault tolerance in this paper. Research on fault tolerance in ML systems has mainly focused on various input and output observers and model redundancy [1]. This means that, for example, changes in inputs and outputs can be monitored, unacceptable values are handled differently, and the system may contain multiple models that handle inputs with some orchestration. The inner workings and structures of the models, however, are rarely utilized in error detection to achieve fault tolerance. 2.2. Neural networks The structure of neural networks consists of node layers [11] (see Fig. 2). The previous layer is connected to the next one. That is, when a node receives an input, it either activates and passes on an output to Array 15 (2022) 100201 3 L. Myllyaho et al. the next layer of nodes or remains dormant and, in practice, outputs 0, thus having no effect on the following computations. The technique responsible for the activation is an activation function [12]. A rectified linear unit (ReLU) is a commonly used activation function. ReLU very closely follows the philosophy of either activating or remaining dormant. Mathematically ReLU is usually formulated as 𝑓(𝑥) = max(0, 𝑥). In practice, this means that if the input a node receives is negative or 0, it actually does not have an effect on the following computations, but if the inputs are very strong, the effect the node has on the following layer is also strong. According to Sharma et al. [12], ReLU has proved to be very effective and is one of the most used activation functions today. 2.3. Related work In neural networks, various groups of nodes tend to take responsibility of different outcomes [4]. In their work, Tian et al. showed that different groups of nodes in a neural network for autonomous driving tended to activate based on whether the neural network proposed turning to the left or to the right. Xie et al. [5] also suggest that transforming a test input too much will lead to a deformed activation pattern and a wrong result, suggesting that the mutated pattern is related to the incorrect result. The activations have been used in research concerning the testing of neural networks (e.g. [3–6]). Usually this means finding nodes that have not activated during testing or exploring improved methodologies for creating test cases to find such nodes. This is rooted in the idea that so-called ‘‘neuron coverage’’ is related to code and statement coverage in traditional software: if the neuron has not activated during testing, the effect of that neuron is not known [3]. However, activation-related error detection measures have not been used widely and consistently in practice to achieve fault tolerance [1]. That is, activations have been monitored to initially test the model prior to deployment but not to continuously validate the system during runtime. The idea has raised some interest in practitioners, but how the activations should actually be monitored to detect errors and what conclusions should be made based on them has remained unclear [1]. Numerous attempts have also not been made on the research side, as we are aware of only one paper attempting to build fault tolerance by specifically utilizing activation monitors. In their work, Cheng et al. [13] form a pattern from the activations of the penultimate layer for each class. If the model prediction differentiates too much from the previously established pattern, the output is flagged as potentially erroneous. This shows promise in detecting some misclassifications. However, they only focus on the penultimate layer and a certain subset of the nodes they consider to be the core nodes affecting the outputs. Thus, they do not entertain the idea of how activations in the earlier layers or outside the core set behave. Also, the focus is on immediate error detection, and how the activations behave across various scenarios (i.e. whether the output was correct, incorrect, or something the model was not trained for) is not addressed. Thus, which types of failures the activation monitors are effective against remains uncertain, as does whether they could also be utilized when monitoring concept drift. 3. Concepts and goals In this section, we introduce the concept of rare co-activations, a novel approach to estimate the typicality of activation patterns in a neural network. Our goal is to show that correctly predicted inputs differ from problematic inputs with regards to rare co-activations within the network. Thus, atypical activation patterns would indicate untrustworthy predictions. If this is the case, monitoring rare co-activations would show potential in error detection in neural networks. First, we describe the concept of rare co-activations in detail in Section 3.1. Then, we discuss the motivation of our research goal and present our research questions in Section 3.2. 3.1. Co-activation rate & Rare co-activations As described in Section 2, nodes in neural networks either activate or remain dormant, and these activations form patterns responsible for certain outputs. If two nodes belong to one or more of the same patterns, they could be expected to activate together a fair share of the time. To estimate whether two nodes do not belong to any of these patterns, we present the idea of co-activation rate: how likely is it that node 𝑚activates when node 𝑛activates. More specifically, for every node 𝑛in a neural network, we calculate its co-activation rate with node 𝑚in a set of inputs 𝐼as 𝑟𝑎𝑡𝑒(𝑛, 𝑚) = 𝐼 ∑ 𝑖=1 𝑛𝑖∩𝑚𝑖 𝐼 ∑ 𝑖=1 𝑛𝑖 , where 𝑟𝑎𝑡𝑒(𝑛, 𝑚)is the co-activation rate of node 𝑛with node 𝑚, and 𝑛𝑖, 𝑚𝑖= 1 if nodes 𝑛and 𝑚activate with input 𝑖and otherwise 𝑛𝑖, 𝑚𝑖= 0. Algorithmically, the co-activation rates for a set of inputs 𝐼can be calculated with Algorithm 1. Using the algorithm, we will end up with a two-dimensional array rates, from which the co-activation rates for nodes 𝑛and 𝑚can be found as, in fact, rates[𝑛][𝑚] = 𝑟𝑎𝑡𝑒(𝑛, 𝑚). The time complexity of Algorithm 1is 𝑂(𝑖𝑛2), where 𝑖is the number of inputs in 𝐼, and 𝑛is the number of nodes in a neural network 𝑁𝑁. Algorithm 1 Co-activationRates(𝑖𝑛𝑝𝑢𝑡𝑠,𝑁𝑁) inputs: Set of inputs in which the co-activation rates are calculated NN: Neural network for which the co-activation rates are calculated 1: rates = [][]: an array in which to store the co-activation rates 2: for all 𝑖in 𝑖𝑛𝑝𝑢𝑡𝑠 do 3: for all node 𝑛in 𝑁𝑁 do 4: if 𝑛activates with 𝑖then 5: for all node 𝑚in 𝑁𝑁 do 6: if 𝑚activates with 𝑖then 7: rates[𝑛][𝑚]+= 1 8: end if 9: end for 10: end if 11: end for 12: end for 13: for all node 𝑛in 𝑁𝑁 do 14: for all node 𝑚in 𝑁𝑁 do 15: rates[𝑛][𝑚] = rates[𝑛][𝑚] / rates[𝑛][𝑛] 16: end for 17: end for 18: return 𝑟𝑎𝑡𝑒𝑠 To produce meaningful results, the set of inputs used to calculate the co-activation rates should be chosen appropriately. The approach we chose was to calculate the co-activation rates after the networks were trained and use the same training set that was used to train them. In this way, the co-activation rates should represent the activation patterns of the input classes that the network should be able to generalize to. Thus, co-activation rates describe the inner workings of the neural networks in cases where the network can reasonably be expected to handle correctly, whereas cases that have no representation in the training set may not produce good results. The activation pattern we study in this paper is derived from coactivation rate: a rare co-activation occurs when two nodes with a low co-activation rate activate within a neural network during a prediction. Thus, a rare co-activation is an indication of such computations that normally do not occur during a prediction. The more these rare coactivations occur, the more disjointed the activation pattern is from Array 15 (2022) 100201 4 L. Myllyaho et al. Table 1 Models used to test hypotheses. Model Filtered class Number of outputs Accuracy in test set without the filtered class CNN-ankle boot 9 (ankle boot) 10 91.8% CNN-ankle boot9 9 9 91.7% CNN-shirt 6 (shirt) 10 95.6% MLP-ankle boot 9 10 88.4% activation patterns that have occurred within the network before, and more atypical it is. In this study, we are looking into the connection between the problematic predictions and the number of rare co-activation occurring when the prediction is made. 3.2. Research goal and questions The goal of our research is to show that atypical activation patterns indicate untrustworthy predictions. To build dependable systems, a general need currently exists for fault tolerance in ML systems. However, approaches utilizing node activations to detect errors in neural networks have not been extensively studied regardless of their role in the computation process. The reasoning we have here is that by showing that activation patterns – rare co-activations in our case – behave differently in correct and problematic predictions, we can argue that observing the activation pattern has potential in error detection. In this paper, we study activations in the context of a classification problem, where certain classes are excluded from the training set but remain present in the separate test set. More specifically, we explore how activation patterns behave in the following scenarios: 1. Test cases for which the output is correct; 2. Test cases for which the output is incorrect despite its class being present in the training set; 3. Test cases where the input does not belong to any class in the training set. Based on this, we aim to assess whether the activation patterns we study can be used to improve fault tolerance, especially by detecting erroneous outputs, problematic inputs, and potential concept drift. Henceforth, we will address cases in the scenarios as correctly predicted inputs,incorrectly predicted inputs, and untrained inputs, respectively. The pattern we study is rare co-activations introduced above in Section 3.1. As the activations tend to form patterns [4], it makes sense that nodes in shared groups often activate together. If often activating together implies being in one or more of the same patterns, it may not be unreasonable to think that the disjointed and atypical patter manifested in rare co-activations implies a broken pattern and an untrustworthy prediction. Thus, we try and show whether there is a utilizable connection between rare co-activations and untrustworthy predictions. More specifically, we aim to answer the following research questions: •RQ1: Does the number of rare co-activations statistically differ in the above scenarios? •RQ2: Can rare co-activations be used to detect erroneous behaviour when building fault-tolerant ML systems, and how? The aim of RQ1 is to explore whether the idea is valid in the first place. Only a statistically significant difference in the number of rare co-activations allows us to argue that our approach has any potential in building fault-tolerant ML systems. If the distributions between cases where the neural network made a correct prediction and cases where the prediction was wrong or the input never appeared in the training set are not statistically different, we cannot claim that any meaningful conclusions can be drawn from the number of rare co-activations. As for RQ2, if the distributions actually differ in the various scenarios, we aim to detect what types of misbehaviour [1] could be addressed by abusing the rare co-activations. Differences in the number of rare co-activations in and of itself does not mean that the result is useful in error detection and fault tolerance as is. Also, as not every form of fault tolerance is suitable for every type of misbehaviour [1], we must consider how the results could link the approach to known misbehaviour types. In this case, the potential to tackle some forms of misbehaviour must be deduced from how the rare co-activations manifest in various scenarios. Specifically, we consider three misbehaviour types that pose a major risk to some systems and that we believe could potentially reveal themselves in the rare co-activations: untrustworthy predictions, inputs that could be problematic for the network, and drift in the incoming data [1]. 4. Experimental set-up In this section, we describe how the experiments were conducted. First, Section 4.1 describes the neural networks from which we gathered the data about the rare co-activations, along with how the networks differ from each other and why. Then, in Section 4.2, we describe what kind of data about rare co-activations in those networks was gathered and how. Finally, in Section 4.3 we describe how the data was analysed to draw conclusions and to ensure statistical significance of our findings. This combination of data triangulation [14] across networks and statistical rigour [15] raises confidence in our findings. 4.1. Neural networks To test the approach, four neural networks were built. The networks are intended to represent mundane neural networks in which we observe how the rare co-activations behave in the three different scenarios considered (correctly predicted, incorrectly predicted and untrained inputs, see Section 3.2). The differences between networks were designed to catch differences often present in neural networks (see descriptions below) and add data triangulation [14]. A general description of the models is presented in Table 1. All networks were trained using the Fashion-MNIST [16] data set. Fashion-MNIST consists of 28 ×28 size greyscale images depicting pieces of clothing.1The training set and test set for Fashion-MNIST contain 60 000 and 10 000 images, respectively, both divided into 10 evenly sized classes. The classes are – in order of labels from 0 to 9 – T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, and Ankle boot. The networks were trained for 15 epochs using Keras2 machine learning library. To make the results easier to reproduce, a fixed random seed (3) was chosen with a throw of a d20 die. To mimic Scenario 3 from Section 3.2 (a class of inputs was missing from the training set, but appears after training), one class was excluded from the training set, similarly to Ackerman et al. [17]. This way, in the testing phase with a separate test data set, we have both inputs that belong to classes the network was trained to recognize, along with inputs that the neural network should not have extensive knowledge of. Thus, the excluded class represents situations where all inputs do not resemble the data that the neural network was trained with. The networks achieved an accuracy ranging from 88.4% to 95.6% (see Table 1) in the test set, excluding the filtered out class. Omitting the filtered out class when measuring the accuracy gives a better estimate of how well the neural networks perform in tasks they should know. This, in our opinion, better estimates the model’s ability to learn the data set than by including the class it was not trained with in the first place. We note that the used networks are not fine-tuned to the 1Examples of Fashion-MNIST can be found at: https://github.com/ zalandoresearch/fashion-mnist. 2https://keras.io/. Array 15 (2022) 100201 5 L. Myllyaho et al. Fig. 3. A simple convolutional neural network. In the convolutional layers, every part of the input goes through a convolution, on which the activation is applied, after which the strongest activations of every small area are gathered by a pooling layer. Convolutional layers are followed by fully connected layers. maximum nor have they been pushed to their limits through training time. This decision is twofold. First, even though it were possible to push a neural network to basically label every input in Fashion-MNIST correctly (e.g. Kayed et al. in [18]), this would leave us with very little data to address mislabelled classes present in the training data (Scenario 2 in 3.2). This, in turn, would risk statistical significance and our ability to meet the research goals we have set. Second, reallife ML models may not reach such high levels of accuracy in their respective data sets (e.g. in [19]). Thus, not pushing the models to their limit makes them more on par with their industrial counterparts, representing them better. With these two reasons combined, our networks are, in a sense, intentionally broken, but also ‘‘good enough’’. In other words, they handle most cases correctly, showing some strength in their behaviour, and yet, make some errors in order to leave us with enough data to answer our research questions. This clearly poses some threats to the validity of the study, which are addressed in Section 7. Next, we go through all models in more detail. CNN-ankle boot: As the name suggests, CNN-ankle boot is a convolutional neural network [20] (cf. Fig. 3). The ‘‘ankle boot’’ in the name refers to the class (9, Ankle boot) that is filtered out from the training set for this neural network. The output node for the specific class is still present in the network, even if it is filtered out in the training phase. The structure of CNN-ankle boot begins with three convolutional layers. The convolutional layers consist of 3 ×3 -sized filters with a stride length of 1 and the same padding. The three convolutional layers have 32, 64, and 128 filters, respectively. Each convolutional layer is accompanied with a batch normalization layer [21], ReLU activation, and a 2 ×2 -sized max pooling layer [22]. The convolutional layers are followed by two fully connected layers. The fully connected layers also utilize ReLU activation, and consist of 64 and 128 neurons, respectively. Finally, the output layer consists of 10 neurons, utilizing the Softmax activation function [12]. CNN-ankle boot9: CNN-ankle boot9 shares most features with CNN-ankle boot except for the number of nodes on the output layer. The filtered class is the same, along with the hidden layers in the neural network. The difference is that the output layer has no reserved output node for the filtered class. This naturally results in only having nine nodes on the output layer. The reasoning behind this is that they represent two different situations in training a neural network. In the case of CNN-ankle boot9, the imaginary developers are unaware that ankle boots exist and do not reserve an output node for it. With CNN-ankle boot, however, the developers know ankle boots exist, they just do not have enough data for them, and they are left underrepresented in the training set. This adds variety to the results, as CNN-ankle boot9 works ‘‘as intended’’ by the imaginary developers, and begins receiving unexpected data, whereas CNN-ankle boot is left broken by the training data and begins receiving appropriate data only after the training is complete. CNN-shirt: CNN-shirt is structurally identical to CNN-ankle boot. The difference is that the class filtered out from the training set is class 6 (Shirt) instead of class 9 (Ankle boot). The purpose of this is to assess whether the phenomena we find are independent from the filtered class or not. Shirt was chosen, as it is evidently different from ankle boots, whereas, for example, sneakers may not be. MLP-ankle boot: MLP-ankle boot is–as the name suggests–a multilayer perceptron [23] (cf. Fig. 2 in Section 2). That is, all the hidden layers in the neural network are fully connected layers, utilizing ReLU activation and batch normalization. There are two hidden layers with 64 and 128 nodes, respectively, 10 output nodes, and the filtered class is Ankle boot. The reasoning behind the inclusion of MLP-ankle boot is twofold. First, including models with different topology provides additional information whether the results are dependent on certain technologies or not. Second, MLP-ankle boot is a smaller network than the other neural networks. This should give us implications of the effect size that the size of the network has on the phenomena. 4.2. Data collection Data were collected with an experimental set-up utilizing Keras and NumPy.3First, the networks were trained using a training set, from which one class was entirely excluded. Next, co-activation rates for each node in each neural network were calculated using Algorithm 1, introduced in Section 3.1, and the same data that were used for training the network, still excluding one class. Finally, the number of rare coactivations was computed and saved for each input in a separate test data set that included all the classes. Additional details were considered before calculating the co-activation rates, e.g. when should a node be counted as activated. An apparent choice would be when the node outputs a non-zero number, as that is de facto how a ReLU activation function works. However, a node could output a very small number 𝛿 > 0that has no actual effect on the outcome of the computations. It is less obvious if such activations are actually meaningful regarding the outcome. To assess this, we calculate the co-activation rates using three thresholds that are counted as an activation: 0 and two model-specific thresholds, namely, a threshold that is smaller than 90% of that network’s nonzero activations in the training set and one that is smaller than 99% of the non-zero activations. Henceforth, we will address these thresholds as activation thresholds. Furthermore, it is not obvious which output should be counted in the convolutional parts of the CNNs. Activation functions are applied first in the CNNs, after which the strongest activations close to each other are gathered by the pooling layer, while the weakest are filtered out. Thus, there are two consecutive parts that have the outputs of the activation function as their values. We chose to use the outputs of the pooling layer, as they are the ones actually affecting the computations of the following layers. After the co-activation rates for every neural network were calculated, the number of rare co-activations was counted and saved using Algorithm 2. First, for each input in the Fashion-MNIST test set, we mark down which of the three scenarios the input represents: is the input predicted correctly, incorrectly, or is it untrained (cf. Section 3.2). Next, the number of rare co-activations in the neural network 𝑁𝑁 with that input are counted and saved. This way, we obtain three types of data points that, as a whole, represent the three scenarios. Finally, in practice, the data are saved to a .CSV file. On lines 14–23 of Algorithm 2, we attempt to capture the elusive keyword rare. As we do not know how the rarity behaves in various networks, it is entirely possible that, for example, the rare co-activations 3https://numpy.org/. Array 15 (2022) 100201 6 L. Myllyaho et al. Algorithm 2 CountCo-activations(𝑖𝑛𝑝𝑢𝑡𝑠,𝑁𝑁,𝑟𝑎𝑡𝑒𝑠) inputs: A set of inputs and their corresponding outputs for which the number of rare co-activations in NN are counted NN: Neural network in which the co-activations are monitored rates: Co-activation rates for NN 1: rareCoActivations = [][]: an array to store the number of rare coactivations for each input, along with information on whether the input was predicted correctly, incorrectly, or if it belongs to the untrained class 2: for all 𝑖in 𝑖𝑛𝑝𝑢𝑡𝑠 do 3: if 𝑖belongs to the untrained class then 4: rareCoActivations[i][0] = ’untrained’ 5: else if 𝑖predicted correctly by 𝑁𝑁 then 6: rareCoActivations[i][0] = ’correct’ 7: else 8: rareCoActivations[i][0] = ’incorrect’ 9: end if 10: for all node 𝑛in 𝑁𝑁 do 11: if 𝑛activates with 𝑖then 12: for all node 𝑚in 𝑁𝑁 do 13: if 𝑚activates with 𝑖then 14: if 𝑟𝑎𝑡𝑒𝑠[𝑛][𝑚]<0.05 then 15: rareCoActivations[𝑖][1] += 1 16: if 𝑟𝑎𝑡𝑒𝑠[𝑛][𝑚]<0.01 then 17: rareCoActivations[𝑖][2] += 1 18: if 𝑟𝑎𝑡𝑒𝑠[𝑛][𝑚]<0.001 then 19: rareCoActivations[𝑖][3] += 1 20: end if 21: end if 22: end if 23: end if 24: end for 25: end if 26: end for 27: end for 28: return rareCoActivations in larger networks are absolutely rarer than in a smaller one. There is, figuratively speaking, more room for the activation patterns to be mostly or completely segregated, whereas the patterns may have to share a larger portion of their nodes in the smaller networks. Thus, we do not settle for one arbitrary threshold for rarity, but instead introduce a few to gain more information on the rarity in various networks. Henceforth, we address these thresholds as rarity thresholds. 4.3. Data analysis Data analysis is based on statistical tests. To answer RQ1 (do the three scenarios differ in terms of rare co-activations), we assessed whether or not the data points in various groups actually originated from different distributions. That is, we are not only interested in whether our samples are different from each other, but we also want to generalize the results to the entire populations from which the samples originate. Using a statistical test, we can determine how certain we can be that not only the samples are different, but the populations behind them as well. Only after this do descriptive statistics, such as the mean, minimum, and maximum, hold strong relevance when comparing the groups. Once the difference is set by tests designed to do just that, these descriptive statistics reveal the nature of the difference. We use the Kruskal–Wallis test [15] to determine that populations are, in fact, different. The Kruskal–Wallis test is an extension of the Mann–Whitney U test for samples that have more than two groups. As such, it is a non-parametric test that does not presume that samples are normally distributed. The outcome 𝑝of the Kruskal–Wallis test should Table 2 Results of Kruskal–Wallis tests for CNN-ankle boot. Activation threshold Rarity threshold Kruskal–Wallis (p) 0<5% 0.0 <1% 0.0 <0.1% 0.0 0.0156* <5% 0.0 <1% 0.0 <0.1% 0.0 0.112** <5% 0.0 <1% 0.0 <0.1% 0.0 *Activation threshold <99% of activations. **Activation threshold <90% of activations. be interpreted so, that with 1 − 𝑝% of certainty, at least two of the populations from which the samples originate from are different. We use the common 𝑝 < 0.05 for the significance level. Thus, when the test suggests that, with more than 95% certainty, at least two groups come from different distributions, we accept that this is actually the case. To assess which groups are different when the Kruskal–Wallis test is significant, we use Dunn’s test [24] with Bonferroni correction. Once the Kruskal–Wallis test finds a significant difference between the groups and Dunn’s test has identified which groups are different, we compare the descriptive statics of those groups. This way, we acquire knowledge on the nature of the difference: Are rare co-activations more common in certain scenarios? Is there a lot of overlap? Based on these statistics, along with information on which groups actually differ from each other, we assess the potential usefulness in the context of fault tolerance. As descriptive statistics we use mean, median, maximum, and minimum, and – when feasible – cross-tabulation. All statistics were gathered using SPSS.4 5. Results In this section, we examine our results obtained using the experimental set-up. The results are presented for every neural network in their own subsection. In turn, for every network, the results are presented for each activation threshold and each rarity threshold, starting from the lowest one. (see Section 4.2 for more details). Only statistically significant results are presented in detail. 5.1. CNN-ankle boot As presented in Table 2, at least two groups in CNN-ankle boot are statistically different (𝑝 < 0.05) from each other for each threshold. Thus, we can make meaningful interpretations about the rare coactivations between the groups with each threshold. Now, we examine the pairwise comparisons of groups for each activation threshold and rarity threshold. Activation threshold 0: For activation threshold 0, rare co-activations are more common in incorrectly predicted and untrained inputs than in correctly predicted ones. The average numbers of occurrences are higher and the differences are statistically significant. Every pairwise comparison between groups is statistically significant (𝑝 < 0.05) (Table 3). Thus, we can confidently say that in CNN-ankle boot, co-activations that occurred between nodes with a coactivation rate of less than 5%, 1%, or 0.1% in the training set, manifest differently when the prediction is correct, incorrect, or with unknown inputs. Next, we present the descriptive statistics of the groups to compare how the groups differentiate. The comparison is made for each rarity threshold, as they all hold significance. 4https://www.ibm.com/analytics/spss-statistics-software. Array 15 (2022) 100201 7 L. Myllyaho et al. Table 3 Results of pairwise comparisons between groups for CNN-ankle boot with activation threshold 0. Activation threshold Rarity threshold Dunn-Bonferroni (p) 0<5% correct–incorrect 0.0 correct–untrained 0.0 incorrect–untrained 0.0 <1% correct–incorrect 0.0 correct–untrained 0.0 incorrect–untrained 0.0 <0.1% correct–incorrect 0.001 correct–untrained 0.0 incorrect–untrained 0.0 Table 4 Descriptive statistics for different groups in CNN-ankle boot with activation threshold 0. Correct Incorrect Untrained N 8265 735 1000 Rarity <5% Mean 39026.32 40490.21 115802.51 Median 9380 13811 85318 Max 739477 688498 494998 Min 3 25 524 Rarity <1% Mean 2088.31 2813.61 9445.02 Median 16 40 985.5 Max 202251 212147 96923 Min 0 0 0 Rarity <0.1% Mean 61.34 128.01 366.59 Median 0 0 0 Max 30022 39403 19564 Min 0 0 0 Considering the mean and median (Table 4), the number of rare coactivations are – on average – slightly more common when the model prediction is incorrect and much more common when the input is from the class that was filtered out of the training set. The relative difference in mean even rises when lowering the rarity threshold, despite the number decreasing and the median in every scenario falling down to 0. The result is similar when comparing the minimum number. However, the highest number of rare co-activations occurred when the model was correct. This applies for the highest rarity threshold, but the number remains relatively high with the lower thresholds as well, even if the highest maximum number is in the incorrectly predicted ones. This suggests that even correct outputs have outliers with large numbers of rare co-activations. Activation threshold 0.0156: Next, we raise the activation threshold to 0.0156. Rare co-activations are more common in incorrectly predicted and untrained inputs than in correctly predicted ones. The average numbers of occurrences are higher and the differences are statistically significant. The chosen threshold is smaller than 99% of all non-zero activations that occurred in CNN-ankle boot in the training set. As we can see from Table 5, every pairwise comparison suggests a difference in distribution (𝑝 < 0.05). Below, we present the descriptive statistics for every rarity threshold. The descriptive statistics remain somewhat consistent despite the raise in activation threshold (Table 6). Rare co-activations in incorrect predictions are slightly more common on average and at minimum, and much more common in the untrained class. Despite this, the maximum number of occurrences in correctly predicted inputs is larger than in other scenarios with the highest rarity threshold and remains in line with the other scenarios with the lower thresholds as well. Median and minimum numbers fall down to 0 in all three scenarios when the rarity threshold is lowered. Table 5 Results of pairwise comparisons between groups for CNN-ankle boot with activation threshold 0.0156. Activation threshold Rarity threshold Dunn-Bonferroni (p) 0.0156* <5% correct–incorrect 0.0 correct–untrained 0.0 incorrect–untrained 0.0 <1% correct–incorrect 0.0 correct–untrained 0.0 incorrect–untrained 0.0 <0.1% correct–incorrect 0.001 correct–untrained 0.0 incorrect–untrained 0.0 *Activation threshold <99% of activations. Table 6 Descriptive statistics for different groups in CNN-ankle boot with activation threshold 0.0156. Correct Incorrect Untrained Rarity <5% Mean 39735.17 40869.48 117150.13 Median 9629 13646 86450.5 Max 744755 692816 507542 Min 2 21 671 Rarity <1% Mean 2141.24 2899.36 9744.11 Median 736 1646 15452.5 Max 389505 407002 214447 Min 0 0 12 Rarity <0.1% Mean 62.64 132.6 360.68 Median 0 0 0 Max 31614 38045 19756 Min 0 0 0 Table 7 Results of pairwise comparisons between groups for CNN-ankle boot with activation threshold 0.112. Activation threshold Rarity threshold Dunn-Bonferroni (p) 0.112** <5% correct–incorrect 0.001 correct–untrained 0.0 incorrect–untrained 0.0 <1% correct–incorrect 0.0 correct–untrained 0.0 incorrect–untrained 0.0 <0.1% correct–incorrect 0.0 correct–untrained 0.0 incorrect–untrained 0.0 **Activation threshold <90% of activations. Activation threshold 0.112: Next, we raise the activation threshold to 0.112, which is smaller than 90% of non-zero activations occurring in the neural network with the training set. Rare co-activations are more common in incorrectly predicted and untrained inputs than in correctly predicted ones. The average numbers of occurrences are higher and the differences are statistically significant. The pairwise comparisons of the scenarios (Table 7) are statistically significant (𝑝 < 0.05) with every rarity threshold. Below, we present the descriptive statistics for every rarity threshold. Based on the mean and median, rare co-activations are slightly more common in incorrectly predicted inputs for activation threshold 0.112 and much more common in the non-trained inputs (Table 8). Also, the minimum number of occurrences was highest in the untrained inputs and second highest in the incorrectly predicted inputs. However, the maximum number of occurrences was highest in the correctly predicted inputs in all but one rarity threshold. Overall, this follows the basic narrative of the lower activation thresholds. Array 15 (2022) 100201 8 L. Myllyaho et al. Table 8 Descriptive statistics for different groups in CNN-ankle boot with activation threshold 0.112. Correct Incorrect Untrained Rarity <5% Mean 45173.06 46055.6 126369.11 Median 11978 16338 91225.5 Max 793519 770479 575951 Min 25 108 1269 Rarity <1% Mean 2450.87 3126.73 10423.61 Median 33 70 1695.5 Max 223204 240410 92044 Min 0 0 0 Rarity <0.1% Mean 74.47 179.18 359.5 Median 0 0 0 Max 37735 36194 20938 Min 0 0 0 Table 9 Results of the Kruskal–Wallis tests for CNN-ankle boot9. Activation threshold Rarity threshold Kruskal–Wallis (p) 0<5% 0.0 <1% 0.0 <0.1% 0.0 0.015* <5% 0.0 <1% 0.0 <0.1% 0.0 0.114** <5% 0.0 <1% 0.0 <0.1% 0.0 *Activation threshold <99% of activations. **Activation threshold <90% of activations. Table 10 Results of pairwise comparisons between groups for CNN-ankle boot9 with activation threshold 0. Activation threshold Rarity threshold Dunn-Bonferroni (p) 0<5% correct–incorrect 0.002 correct–untrained 0.0 incorrect–untrained 0.0 <1% correct–incorrect 0.0 correct–untrained 0.0 incorrect–untrained 0.0 <0.1% correct–incorrect 0.012 correct–untrained 0.0 incorrect–untrained 0.0 5.2. CNN-ankle boot9 In this subsection, we go through the results for the model CNNankle boot9 in a similar manner. This model is otherwise similar to and similarly trained as CNN-ankle boot, but does not have an output node for the class that was filtered out of the training set. See Section 4.1 for more details. As we can see from Table 9, for each activation threshold and rarity threshold, at least two groups representing the three scenarios are statistically different (𝑝 < 0.05) from each other. Thus, we can make meaningful interpretations about the rare co-activations between the groups with each threshold. Next, we present the pairwise comparisons and descriptive statistics of the groups for each activation and rarity threshold. Activation threshold 0: For activation threshold 0, rare co-activations are more common in incorrectly predicted and untrained inputs than in correctly predicted ones. The average numbers of occurrences are higher and the differences are statistically significant. Table 11 Descriptive statistics for different groups in CNN-ankle boot9 with activation threshold 0. Correct Incorrect Untrained N 8256 744 1000 Rarity <5% Mean 26220.5 28798.58 77398.41 Median 4546.5 7165 49562.5 Max 566458 391499 384479 Min 2 11 334 Rarity <1% Mean 1672.11 2151.78 7923.46 Median 6 15 608.5 Max 179581 139666 82259 Min 0 0 0 Rarity <0.1% Mean 63.46 111.02 152.59 Median 0 0 0 Max 46778 27016 13429 Min 0 0 0 Table 12 Results of pairwise comparisons between groups for CNN-ankle boot9 with activation threshold 0.015. Activation threshold Rarity threshold Dunn-Bonferroni (p) 0.015* <5% correct–incorrect 0.009 correct–untrained 0.0 incorrect–untrained 0.0 <1% correct–incorrect 0.0 correct–untrained 0.0 incorrect–untrained 0.0 <0.1% correct–incorrect 0.055 correct–untrained 0.0 incorrect–untrained 0.0 *Activation threshold <99% of activations. From Table 10, we can see that each pairwise comparison is statistically significant (𝑝 < 0.05) with activation threshold 0. This suggests that the distribution of each group is different from one another, and comparisons between the groups can be made. Below, we present the descriptive statistics for each rarity threshold. The descriptive statistics for rarity threshold seems to follow the trend in the previous model (Table 11). On average, the untrained inputs have a much larger number of rare co-activations than the other two scenarios, and rare co-activations in incorrectly predicted ones are slightly more numerous than in the correctly predicted ones. The same goes for the minimum number of occurrences, before it falls down to 0 in all scenarios. However, the maximum number of occurrences is slightly different than with the previous network. Unlike in the previous neural network, where correctly and incorrectly predicted inputs were quite close to each other with almost every threshold, here, the correctly predicted inputs have a much larger maximum number of occurrences than either of the other two scenarios. Again, this provides more evidence that rare co-activations may occur in high numbers in some cases, even if the network’s prediction is correct. Activation threshold 0.015: For activation threshold 0.015, rare co-activations are more common in incorrectly predicted and untrained inputs than in correctly predicted ones. The average numbers of occurrences are higher and the differences are statistically significant for except one. The pairwise comparison of scenarios in CNN-ankle boot9 with the raised activation threshold of 0.015 can be found in Table 12. The activation threshold is smaller than 99% of the non-zero activations in CNN-ankle boot9 with the training set. The most noticeable differences to the previous results is that with the rarity threshold <0.1%, the differences between correctly and incorrectly predicted inputs do Array 15 (2022) 100201 15 L. Myllyaho et al. CRediT authorship contribution statement Lalli Myllyaho: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Visualization. Jukka K. Nurminen: Conceptualization, Resources, Writing – review & editing, Supervision, Project administration, Funding acquisition. Tommi Mikkonen: Conceptualization, Writing – review & editing, Supervision, Project administration, Funding acquisition. Declaration of competing interest No author associated with this paper has disclosed any potential or pertinent conflicts which may be perceived to have impending conflict with this work. For full disclosure statements refer to https://doi.org/ 10.1016/j.array.2022.100201. Acknowledgements This work was funded by local authorities (‘‘Business Finland’’) under grant agreement ITEA-2019-18022-IVVES of ITEA3 programme and grant agreement ITEA-2020-20219-IML4E of ITEA4 programme. We acknowledge the help of Antti Klemetti, Dennis Muiruri, and Juha Mylläri in implementing the experimental set-up, the help of Mikko Raatikainen and Tomi Männistö in revising the manuscript, and thank CSC – IT Center for Science, Finland, for computational resources. References [1] Myllyaho L, Raatikainen M, Männistö T, Nurminen JK, Mikkonen T. On misbehaviour and fault tolerance in machine learning systems. J Syst Softw 2022;183:111096. [2] Abiodun OI, Jantan A, Omolara AE, Dada KV, Mohamed NA, Arshad H. State-of-the-art in artificial neural network applications: A survey. Heliyon 2018;4(11):e00938. [3] Pei K, Cao Y, Yang J, Jana S. Deepxplore: Automated whitebox testing of deep learning systems. In: Proceedings of the 26th symposium on operating systems principles. 2017, p. 1–18. [4] Tian Y, Pei K, Jana S, Ray B. Deeptest: Automated testing of deep-neuralnetwork-driven autonomous cars. In: Proceedings of the 40th international conference on software engineering. 2018, p. 303–14. [5] Xie X, Ma L, Juefei-Xu F, Xue M, Chen H, Liu Y, et al. Deephunter: A coverageguided fuzz testing framework for deep neural networks. In: Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis. 2019, p. 146–57. [6] Ma L, Juefei-Xu F, Zhang F, Sun J, Xue M, Li B, et al. Deepgauge: Multigranularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering. 2018, p. 120–31. [7] IEEE standard dictionary of measures of the software aspects of dependability. IEEE Std 982.1-2005 (Revision of IEEE Std 982.1-1988), 2006, p. 1–41. [8] Avizienis A, Laprie J-C, Randell B, Landwehr C. Basic concepts and taxonomy of dependable and secure computing. IEEE Trans Dependable Secure Comput 2004;1(1):11–33. [9] Ramanathan A, Pullum LL, Hussain F, Chakrabarty D, Jha SK. Integrating symbolic and statistical methods for testing intelligent systems: Applications to machine learning and computer vision. In: 2016 Design, automation & test in Europe conference & exhibition. IEEE; 2016, p. 786–91. [10] Tsymbal A. The problem of concept drift: Definitions and related work. Comput Sci Dep, Trinity College Dublin 2004;106(2):58. [11] Wang S-C. Artificial neural network. In: Interdisciplinary computing in Java programming. Springer; 2003, p. 81–100. [12] Sharma S, Sharma S, Athaiya A. Activation functions in neural networks. Int J Eng Appl Sci Technol 2020;4(12):310–6. [13] Cheng C-H, Nührenberg G, Yasuoka H. Runtime monitoring neuron activation patterns. In: 2019 Design, automation & test in Europe conference & exhibition. IEEE; 2019, p. 300–3. [14] Denzin NK. Sociological methods: A sourcebook. Routledge; 2017. [15] Kruskal WH, Wallis WA. Use of ranks in one-criterion variance analysis. J Amer Statist Assoc 1952;47(260):583–621. [16] Xiao H, Rasul K, Vollgraf R. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. 2017, arXiv preprint arXiv:1708. 07747. [17] Ackerman S, Farchi E, Raz O, Zalmanovici M, Dube P. Detection of data drift and outliers affecting machine learning model performance over time. In: JSM proceedings. American Statistical Association; 2020, p. 144–60. [18] Kayed M, Anter A, Mohamed H. Classification of garments from fashion MNIST dataset using CNN LeNet-5 architecture. In: 2020 International conference on innovative trends in communication and computer engineering. IEEE; 2020, p. 238–43. [19] Gobert C, Reutzel EW, Petrich J, Nassar AR, Phoha S. Application of supervised machine learning for defect detection during metallic powder bed fusion additive manufacturing using high resolution imaging. Addit Manuf 2018;21:517–28. [20] Albawi S, Mohammed TA, Al-Zawi S. Understanding of a convolutional neural network. In: 2017 International conference on engineering and technology. Ieee; 2017, p. 1–6. [21] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning. PMLR; 2015, p. 448–56. [22] Scherer D, Müller A, Behnke S. Evaluation of pooling operations in convolutional architectures for object recognition. In: International conference on artificial neural networks. Springer; 2010, p. 92–101. [23] Jain AK, Mao J, Mohiuddin KM. Artificial neural networks: A tutorial. Computer 1996;29(3):31–44. [24] Dunn OJ. Multiple comparisons using rank sums. Technometrics 1964;6(3):241– 52. [25] Shadish WR, Cook TD, Campbell DT, et al. Experimental and quasi-experimental designs for generalized causal inference/William R. Shedish, Thomas D. Cook, Donald T. Campbell. Boston: Houghton Mifflin; 2002. [26] De Clercq D, Jalota D, Shang R, Ni K, Zhang Z, Khan A, et al. Machine learning powered software for accurate prediction of biogas production: A case study on industrial-scale Chinese production data. J Cleaner Prod 2019;218:390–9.