Quantifying Lower Reliability Bounds of Deep Neural Networks

Quantifying Lower Reliability Bounds of Deep Neural Networks Max Scheerer, Marius Take, Jonas Klamroth FZI Research Center for Information Technology Karlsruhe, Germany {scheerer,take,klamroth}@fzi.de Abstract—Deep Neural Networks (DNNs) continue to permeate various domains, including mission-critical systems. Their unverifiable nature, however, makes them inherently unreliable such that reliability quantification becomes essential to determine whether a DNN is suitable for operation. In this paper, we approach this problem by quantifying lower reliability bounds of DNNs. Our approach builds upon the widely known Conformal Predicition framework for determining reliability and discusses a sampling procedure for approximating the lower bound of reliability, leveraging properties of the data manifold. In our evaluation, we demonstrate the plausibility of the approximated reliability bounds and also show that incorrect predictions correlate with low reliability. The main benefits of our approach are that it enables the evaluation and comparison of DNNs in terms of their reliability on the one hand, and provides a way to incorporate them in model-based reliability analyses at the system level on the other. Index Terms—Deep Neural Network, Reliability, Quantification, Conformal Prediction, Feature Space Analysis I. INTRODUCTION Deep Neural Networks (DNNs) or Deep Learning in general is the most promising subfield of Artificial Intelligence as recent advances in large language models (based on DNNs) have shown. Although these advances are remarkable, they distract from problems that have not yet been fully addressed, namely the provision of quality assurances. The mere fact that even relatively simple DNNs (e.g., object detection) cannot be fully verified in terms of correctness, makes them inherently uncertain [1]. However, the problem of providing no assurances is a critical one because DNNs often make incorrect predictions or exhibit unreliable behavior but shall operate in safety-critical systems. For example, Tian et al. [2] enumerate three incidents in autonomous driving caused by faulty predictions of DNNs. Consequently, various works focused on uncertainty quantification that encompasses different subfields such as Bayesian methods [3], conformal prediction [4] or ensemble techniques [5]. In uncertainty quantification, the uncertainty of each prediction is determined in order to decide whether it can be trusted. Other methods (e.g., [6]), focus on confidence calibration (i.e., the prediction of probability estimates approximating the true likelihood) to determine an appropriate confidence associated with a prediction. What they all have in common is that they are only applicable at the instance level; that is, uncertainty is quantified for a single prediction given an instance or input x(e.g., an image). However, in terms of safety argumentation or assurance cases, one rather prefers to associate a DNN with a single quantity reflecting its overall reliability. In addition, reliability analyses of software systems containing DNN components are typically difficult to realize because their high degree of complexity does not allow their behavior to be modeled. In this paper, we contribute to this branch of research with an approach for quantifying lower reliability bounds of DNNs. To this end, we introduce a definition of the reliability of DNNs from a probabilistic perspective. Afterward, we discuss how lower reliability bounds are determined based on the framework of Conformal Prediction. Conformal prediction is a well-studied approach to quantify the uncertainty in the prediction of a DNN while providing statistical guarantees. Our approach requires only an initial dataset of labeled examples, but can otherwise determine lower reliability bounds based on unlabeled datasets; we leverage this property by sampling data points from the data manifold of the DNN and approximate lower reliability bounds until convergence. Thus, we circumvent the problem of having datasets of limited size leading to unrepresentative reliability estimations. We evaluated our approach by considering the widely known datasets MNIST, CIFAR10, HAM10000 and Fashion MNIST. The results indicated that our approach not only provides plausible results when approximating the reliability of DNNs of different quality but also that incorrect predictions correlate with lower reliability (and vice versa). To the best of our knowledge, there is no approach in the literature that quantifies lower reliability bounds of DNNs and thus forms the central contribution of this work. From a practical point of view, software or data engineers benefit from our approach in several ways. First, our approach enables them to assess and compare distinct and trained DNNs in terms of reliability. Secondly, our reliability bounds can be used to formulate assurance cases for safety-critical systems, i.e., by determining a tolerable lower bound that DNNs must satisfy. From a scientific view, our approach not only provides findings in terms of reliability assessment of DNNs but also forms a starting point for inclusion in system-level reliability analyses (e.g., Markov analysis) or verification (probabilistic model checking) where DNN components are represented as probabilistic black-boxes. For example, we argue that approaches such as [7], [8] could directly benefit from our approach by including the reliability bounds in their analysis. II. PRELIMINARIES In this section, we briefly describe the common notations and basic concepts on which this work is based. A. Notations We write P r(a)as shorthand notation of Pr(X=a)which describes the probability of observing event aof a random variable X. On the contrary, P(X)defines a probability distribution defined over random variable X. We abstract a DNN by the function f:X → Y (more details follow in the next section). Throughout this paper, we refer to the event f(x) = y∗, in which y∗simply describes the correct label or true output associated with input x. That is to say, f(x) = y∗describes the event that fproduces the correct output y∗for input x. Because we are mainly interested in the overall success of a DNN, we relate a DNN fwith a binary random variable Xfhaving the possible events Success and Fail. Based on Xf, we formulate the success probability Pr(Xf=Success)(e.g., Pr(Xf=Success) = 0.9 means that fgenerates 90% correct predictions). The success probability is one of the central concepts of this paper; we, therefore, use the notation Success(f) = Pr(Xf=Success) to signify its importance. B. Deep Neural Networks and the Feature Space DNNs are functions fmapping an input x∈ X of the input space X(e.g., pixel or image space) to an output y=f(x)∈ Yof the output space Y. As we discuss later, we focus in this paper on classification; that is to say, the output space refers to a set of classes, i.e., Y:= {y1, . . . , ym}, m ∈N. Internally, a DNN is composed of multiple layers generating the deep structure of f. Each layer lforms a designated function f(l)such that fis viewed as a composition of Lfunctions (or layers): f(x)=(f(L)◦f(L−1)◦· · ·◦f(1))(x). The first layer f(1) is commonly described as the input layer; the intermediate layers f(2) to f(L−1) are denoted as hidden layers. Finally, the last layer f(L)refers to the prediction layer. Regarding classification, the prediction layer is commonly represented by a softmax function which normalizes the output vector to a probability distribution defined over Y, i.e., P(Y | x). The layers f(1) to f(L−1) are responsible for dimensionality reduction and feature extraction. More specifically, ϕ:X → Rndescribes the feature extractor of fwhich maps an input to the real vector space ϕ(x)=(f(L−1) ◦ · · · ◦ f(1))(x)or, more accurately, the Feature Space (also referred to as Latent Space), in which the dimension nis much smaller than the dimension of the input space, i.e., n≪dim(X). During training, ϕis learned to extract meaningful features from the input xso that an optimal classification can be performed in the prediction layer. C. Conformal Prediction In the following, we briefly describe the Conformal Prediction framework on which our reliability quantification is based. We refer to [4] for a detailed discussion on conformal prediction. Conformal prediction allows generating Conformal Prediction Sets Cof a DNN containing a set of labels for any given input x(i.e., C(x)⊆ Y), in which the true label y∗is included with a guaranteed probability of 1−εand where εis a user-defined error level. Intuitively, the prediction sets provide a notion of uncertainty in the sense that large sets generated for an input xindicate high uncertainty and vice versa. For example, if we compute a prediction set for an image of the MNIST dataset and observe (say) C(x) := {1,3,6,8,9}of possible solutions, the prediction should be considered with care. Remarkably, the prediction sets are valid without making distributionalor model assumptions. Generally, conformal prediction can be applied to any prediction model and is not limited to classification. In the case of classification, however, a conformal prediction procedure generates conformal sets by considering a heuristic notion of uncertainty (e.g., confidences of the softmax output of a trained DNN) and a calibration set Dcal := {(x1, y1),...,(xn, yn)} (i.e., a holdout set that the DNN has not observed during training). Based on the calibration dataset, the conformal prediction procedure turns the heuristic notion of uncertainty into a rigorous one, such that for any new data pair (xn+1, yn+1) (generated by the same distribution as the calibration dataset), the following inequality is guaranteed [4]: 1−ε≤Pr(yn+1 ∈ C(xn+1)) (1) In other words, the true label yn+1 is guaranteed to be in set C(x)with a probability of at least 1−ε. This inequality is also denoted as marginal coverage property. In this paper, however, we rely on the conditional coverage property: 1−ε≤Pr(yn+1 ∈ C(xn+1)|xn+1)(2) Unfortunately, the conditional coverage property is stronger than the marginal coverage property and can generally not be guaranteed by conformal procedures. Nonetheless, there exist conformal procedures that provide good approximations of the conditional coverage property such as in [9]. III. APPROACH Based on the conformal prediction framework, we quantify lower reliability bounds of DNNs which we discuss below. A. Quantifying the Reliability of Deep Neural Networks We define the reliability of a DNN by considering its success probability Success(f)(recall from section II-A). That is to say, we use the terms reliability and success probability synonymously. For example, we consider the reliability of a DNN to be 0.9 if Success(f)=0.9. The success probability is calculated by taking into account the number of inputs xof the input space Xfor which fpredicts the correct outcome y∗, i.e., f(x) = y∗: Success(f) = Zx∈X Pr(f(x) = y∗|x)dP(x) =Zx∈X 1 {f(x) = y∗}dP(x) (3) (A similar definition can be found in [10].) Hereby, 1 {b} represents the indicator function which evaluates to 1 if bis true and 0 otherwise. The probability Pr(f(x) = y∗|x) can be rewritten to 1 {f(x) = y∗}, if the true outcome y∗is known for any input x, e.g., labeled datasets D:= {(x1, y1),...,(xn, yn)}. In most cases, however, the true label y∗is only known for a subset of the input space X. Consequently, calculating the reliability (or the success probability) of fw.r.t. (3) is not feasible. In the following sections, we examine how we approach this problem by determining lower bounds on reliability. B. Determining the Lower Reliability Bound Calculating the reliability of a DNN w.r.t. equation (3) is not feasible in many cases because we can only acquire the true labels for a subset of the input space X. The input-label pairs (x, y)are provided by a dataset Dwhich is split into subsets for training, validation, testing, etc. For any dataset Deval containing fresh input-label examples a DNN fhas not seen before, we can evaluate the success probability w.r.t. equation (3) as follows: Success(f)≈X (x,y)∈Deval 1 {f(x) = y} · Pr(x)(4) Note, however, that (4) provides only an approximation of Success(f)which strongly depends on the number of examples provided by Deval. Again, we face practical limitations because in most settings (such as in computer vision) we do not have such a high number of examples. We deal with the implications of equations (3) and (4) by determining a lower reliability bound λof a DNN fsuch that λ≤Success(f)which is derived exclusively from examples of the input space X(i.e., without knowing their true label y∗). Therefore, consider the following theorem: Theorem 1. Let C(x)be the conformal prediction set of a DNN ffor a given input x∈ X and ε∈[0,1] be an error level, then the following inequality holds: Pr(f(x) = y∗|x)≥Pr(f(x) = y∗|y∗∈ C(x)) ·(1 −ε) The proof is provided in the appendix of this paper. The theorem states that the probability of a DNN f, making a correct prediction for input x, factorizes into two products. The second factor refers to 1−εwhich (if we carefully review the proof of Theorem 1) describes the conditional coverage property (2) of a conformal procedure (see section II-C). The first factor refers to the probability Pr(f(x) = y∗|y∗∈ C(x)), which conditions the event f(x) = y∗on the event y∗∈ C(x). In other words, it describes the probability that fsuccessfully predicts the correct label of an input xif we already know that the true label y∗is, in fact, in the conformal prediction set C(x). For the probability Pr(f(x) = y∗|y∗∈ C(x)), we can make an interesting observation: Observation 1. For any DNN fthe probability Pr(f(x) = y∗|y∗∈ C(x)) evaluates as follows: Pr(f(x) = y∗|y∗∈ C(x)) = 1,|C(x)|= 1 ?, otherwise In simple terms, observation 1 states that whenever the size of C(x)for any input xis equal to 1, the probability Pr(f(x) = y∗|y∗∈ C(x)) must evaluate to 1 as well. Recall that conformal prediction sets C(x)with a high number of labels indicate high uncertainty and vice versa. Having a singleton prediction set combined with the knowledge that y∗∈ C(x)guarantees that the prediction f(x)for input xis correct. On the contrary, if we observe non-singleton prediction sets (i.e., |C(x)|>1), we cannot make any statements about the probability. Intuitively, one could equate the probability to 1/|C(x)|; however, there is no guarantee for this intuition. Coming back to Theorem 1, observation 1 implies that, for any input x,Pr(f(x) = y∗|x)must have the lower bound (1 −ε)iff the corresponding prediction set Cε(x)contains a single element: |Cε(x)|= 1 ⇔Pr(f(x) = y∗|x)≥(1 −ε)(5) Note that Cε(x)refers to a conformal prediction procedure that computes the prediction set of an input xw.r.t. error level ε. This only works for inputs that generate a singleton prediction set. Note, however, that this lower bound can be controlled by the error level ε. Let us assume, for example, that we have a fixed error level ε= 0.01 and want to evaluate the lower reliability bound for a given input x. If the prediction set satisfies the condition |Cε(x)|= 1, we can assign 0.99 to xas a lower bound. If the condition is not satisfied, we can gradually adjust the error level to (say) 0.98 and check whether the condition is met. This is repeated until |Cε(x)|= 1 is true for a certain error level such that 1−εis returned. Based on the iterative adjustment of the error level, we can construct an algorithm that approximates for a dataset D (and w.r.t. equation (4)) the lower reliability bound. This is illustrated in algorithm 1. Note that δdescribes the step size Algorithm 1: Lower reliability bound approximation Input: Dataset D, DNN f, Step size δ Output: Lower reliability bound λ≈Success(f) 1Λ←empty() // i.e., ∀(x, ·)∈ D : Λ[x] = ∅ 2ε←0 3while D =∅do 4ε←ε+δ 5foreach (x, ·)∈ D do 6if |Cε(x)|= 1 then 7Λ[x]=1−ε 8Remove (x, ·)from D 9end 10 end 11 end 12 return 1 |Λ|PxΛ[x] between error levels, e.g., δ= 0.01 generates the error levels {0.01,0.02,...,0.99}that are iteratively used to calculate prediction sets Cε(x). In the end, the algorithm approximates the lower reliability bound w.r.t. equations (3) and (4) by assuming that each example of Dis equally likely and by replacing Pr(f(x) = y∗|x)with its corresponding success level (1 −ε)(saved in Λ[x]) as discussed in this section. It is important to note that conformal prediction enables us to derive lower reliability bounds of a DNN without the need for a labeled dataset. This simplifies the process of acquiring sufficiently large datasets to approximate Success(f). However, for input spaces such as the pixel space, it is still difficult to acquire input examples large enough to sufficiently approximate the success probability. In the next section, we explain how we achieve good approximations of Success(f) by sampling from the feature space of f. C. Approximating the Lower Reliability Bound So far, we have presented how to determine lower reliability bounds based on an unlabeled dataset D. Because our approach does not require labeled examples, a natural way to generate more data would be to sample from the input space. However, this is not feasible for a lot of applications, exhibiting large and complex input space. Instead, we sample from the feature space of a DNN f. Recall from section II-B that a DNN fis structured into various layers f(x)=(f(L)◦f(L−1) ◦ · · · ◦ f(1))(x), in which the first L−1layers are responsible for dimensionality reduction and feature extraction (recall ϕ).The feature extractor ϕ:X → Rn projects the inputs into the feature space, which is usually of much smaller dimension than the input space X. However, although n≪dim(X), it still seems untractable to sample from the vector space Rn. We approach this problem by leveraging properties of the manifold hypothesis [11], [12] which states that high-dimensional real-world data concentrates on low-dimensional manifolds embedded in the highdimensional space. In face recognition, for example, input images containing faces occupy only a small part of the pixel space and can be reduced to a feature vector of much smaller dimensions. These features are concentrated on contiguous regions in the low-dimensional data manifold of the feature space. This means that we only need to generate samples in very small regions of Rn, lying on the data manifold, which drastically reduces the number of samples required. (Note that for conformal prediction procedures, it makes no difference if the prediction sets are determined from the input space or the feature space as they are calculated from the output of the prediction layer.) Therefore, we fit a Gaussian Mixture Model (GMM) to identify and approximate the contiguous regions in the feature space, lying on the data manifold. For classification problems, it seems obvious to estimate the data manifold by a GMM, because the feature extractor ϕhas been trained to distinguish inputs such that each class occupies a contiguous region in the feature space. By fitting a GMM to approximate the data manifold, we also ensure that we sample meaningful vectors φ describing the features of an image of the operational domain (e.g., all images with human faces). The GMM is used as a generative model to sample new vectors φfrom the feature space, which are used to calculate Success(f)(according to algorithm 1) until convergence. This sampling process is depicted on Figure 1. The sampling process relies on an initial dataset Dinit which is projected into the feature space w.r.t. ϕ. Based on the projections, a GMM is fitted where the number of Gaussians equals the number of classes. The GMM is used to iteratively sample meaningful vectors of the feature space; i.e., those that lie on the data manifold, representing images of the operational domain. A fixed number of data points are sampled in each iteration and appended to the already generated vectors. After each iteration, the generated set of features is passed to algorithm 1 to estimate the success probability. This is repeated until a convergence criterion is reached, e.g., if the difference of Success(f)in iteration i and iteration i+ 1 falls below a predefined threshold. IV. EVALUATION In this section, we present and discuss the evaluation of our approach. Our code and results are publicly available [13]. A. Evaluation Goals, Questions and Metrics We organize the evaluation according to the Goal-QuestionMetric (GQM) approach [14]. The GQM approach structures the evaluation into overarching evaluation goals. Each goal is linked to evaluation questions, the answers to which are intended to allow conclusions to be drawn about the achievement of the goal. Finally, each question is linked with a set of metrics to provide the answers. We focus on the following evaluation goals: (G1) Investigate the plausibility of Success(f). (G2) Analyse the correlation between Success(f)and (in)correct predictions. Goal (G1) is concerned with the plausibility of our approach. To be more specific about what we consider to be plausible, we formulate two evaluation questions: (Q1) Are our lower reliability bounds sound (especially compared to other metrics)? (Q2) Does Success(f)reflect the quality of distinct DNNs? The first question (i.e., (Q1)) investigates whether our approach produces sound approximations of lower reliability bounds compared to other metrics. Let us assume that we can order a set of DNNs according to their qualities (e.g., accuracy), question (Q2) asks whether we observe the very same ordering when approximating Success(f)for each DNN. We aim to answer both questions by the following metrics: (M1) Success probability comparison: Comparison of the approximated lower reliability bounds λ≈Success(f)and λ∗calculated w.r.t. (4) and a labeled dataset D. (M2) Rank preservation: Examine whether for a given total ordering of DNNs f1<· · · < fNthe same holds in terms of Success(f), i.e., f1<· · · < fN⇒Success(f1)<· · · < Success(fN). Regarding evaluation goal (G2), we expect that incorrect predictions correlate with low reliability (or success probabilities) and vice versa. Therefore, we consider the following questions: (i) (ii) (iii) (iv) Fig. 1: Overview of the sampling process. First (i), the prediction layer is removed from DNN fsuch that the inputs of a dataset Dare projected into the feature space (ii). Afterward, a GMM is fitted to estimate the data manifold and to sample meaningful vectors from the feature space (iii). Ultimately, the sampled vectors are evaluated in terms of algorithm 1 (iv); this procedure is repeated until the success probability Success(f)is sufficiently approximated. (Q3) Is there a significant deviation when Success(f)is computed based on inputs that lead to correctand for inputs that lead to incorrect predictions? (Q4) Does the magnitude of output deviations correlate with low reliability (and vice versa) of individual inputs? By output deviation, we mean the position of the true label y∗in the softmax output sorted by their probabilities in descending order. That is, the farther y∗is from the first position of the ordered softmax output, the worse the prediction. Consequently, an increasing position is expected to correlate with low reliability at the instance level which is determined by Pr(f(x) = y∗|x)(recall from (3)). We consider the following metrics for the questions: (M3) Comparison with (in)correct predictions: Given two datasets D+:= {(x, y)|f(x) = y)}and D−:= {(x, y)| f(x)=y)}, we separately compute Success(f)w.r.t. algorithm 1 for D+and D−and compare the results. (M4) Correlation analysis: Correlation of the magnitude of the output deviations to P r(f(x) = y∗|x)of individual inputs by conducting Pearson correlation analysis. B. Methodology and Setup For the evaluation, we consider four datasets, namely MNIST, Fashion MNIST, HAM10000 and CIFAR10. We have split each dataset as follows: •Dtrain: Representing the training data. •Dgaus: For estimating the GMMs (using expectation maximization). •Deval: The evaluation dataset includes the data examples used as the initial dataset for approximating the lower reliability bounds (recall from algorithm 1). •Dcal and Dtun: Reflects the calibration and tuning datasets required for the conformal prediction procedure. As conformal prediction procedure, we used Regularized Adaptive Prediction Sets (RAPS) approach [9]. For each dataset, we train three DNNs sharing the same network architecture but differ in the amount of training data used for training. We organize Dtrain into three datasets: the first contains all training examples, the second only 50%, and the last only 10% of the training examples (for MNIST and Fashion MNIST, we used only 1%). By doing so, we ensure to have models of different quality (i.e., accuracy, recall, specificity, etc.) that we leverage to evaluate the plausibility of our approach. We denote these models fds[best],fds[avg] and fds[worst]to account for their differences in quality where ds is a placeholder of the dataset on which fwas trained, e.g., fmnist[best]for the best DNN for MNIST. Moreover, the models exhibit the following partial order: fds[worst]≤fds[avg]≤fds[best]. C. Results In the following, we present the results obtained by each metric. The results are depicted on Table I. 1) Success probability comparison (M1): We compared the approximations of the lower reliability bounds (denoted as λ) for each DNN with the success probability calculated conventionally by (4) and Deval (denoted as λ∗). Ideally, we would observe λ≤λ∗; however, because λ∗is evaluated on a limited number of examples, we expect them to exhibit a certain degree of similarity. Looking at Table I, we can see that for the MNIST and Fashion MNIST dataset, that λand λ∗are fairly similar. While for fmnist[best]and fmnist[avg]the condition λ≤λ∗is even satisfied, the difference λ−λ∗is small for the other DNNs. We argue that for these models, we expect to observe λ≤λ∗as the number of examples in Deval grows. The DNNs trained on CIFAR10 and HAM10000 show higher deviations, so one may doubt that λ≤λ∗is observed with an increasing number of examples. We attribute these deviations to certain peculiarities of CIFAR10 and HAM10000; we discuss this in section IV-D. 2) Rank preservation (M2): As discussed in IV-B, we trained for each dataset three DNNs sharing the same network architecture but with different amounts of training data. By doing so, we ensure at least partial order among the models (e.g., fmnist[worst]≤fmnist[avg]≤fmnist[best]) and expect the lower reliability bounds to occur in the same order. TABLE I: Overview of the metric results. λdescribes the lower success probability determined by our approach; λ∗describes the success probability evaluated by (4) and dataset Deval. If a metric is annotated by ✓or ✗, this means that only correct predictions (✓) or incorrect predictions (✗) were taken into account. A bar above a metric indicates that the values are averaged, e.g., Success (✓) means: average success probability of correct predictions. Model Metric (M1) and (M2) Metric (M3) Metric (M4) λ λ∗MSE Success Success (✓) Success (✗) Position (✓) Position (✗) Pearson corr. fmnist[best]0.9666 0.9908 0.0014 0.9885 0.9892 0.9088 0 1.4 -0.4033(***) fmnist[avg]0.9722 0.9875 0.0019 0.9881 0.9889 0.9269 0 1.4 -0.4451(***) fmnist[worst]0.9023 0.9089 0.0136 0.9458 0.9558 0.8461 0 1.8 -0.3912(***) fcifar[best]0.8391 0.7804 0.0316 0.8760 0.9095 0.7568 0 1.9 -0.4530(***) fcifar[avg]0.7554 0.6774 0.0443 0.8068 0.8576 0.7001 0 2.1 -0.4139(***) fcifar[worst]0.3582 0.2557 0.0890 0.3637 0.4145 0.3462 0 3.7 -0.3790(***) ffashion[best]0.9374 0.9150 0.0125 0.9617 0.9700 0.8720 0 1.4 -0.4396(***) ffashion[avg]0.9191 0.8982 0.0145 0.9553 0.9651 0.8690 0 1.4 -0.4706(***) ffashion[worst]0.8451 0.7738 0.0316 0.8824 0.9104 0.7864 0 1.7 -0.3997(***) fham[best]0.9316 0.7717 0.0532 0.9189 0.9393 0.8497 0 1.7 -0.3611(***) fham[avg]0.9013 0.7306 0.0615 0.9012 0.9223 0.8438 0 1.8 -0.3035(***) fham[worst]0.8430 0.6651 0.0758 0.8380 0.8701 0.7742 0 2.0 -0.2914(***) *** With p-value <0.001 indicating high statistical significance In Table I, we can observe for each of the (dataset) grouped models that the rank of the orders is preserved by the approximated lower reliability bounds. More specifically, when comparing the second column (the approximated lower bounds) with the third (the success probabilities evaluated w.r.t. (4)) and fourth column (the Mean-Squared-Error (MSE)), it can be seen that the very same order is preserved. (Note that the smaller the MSE, the better the DNN.) 3) Comparison with (in)correct predictions (M3): For each DNN, we split Deval into correct and incorrect predictions (i.e., D+and D−). Afterward, we calculated average success probability Success(f)w.r.t. algorithm 1 for Deval,D+and D−separately. Looking at Table I (columns 5-7 or Metric (M3)), it can be seen that the Success(f)for correct and incorrect predictions significantly deviates for each DNN. 4) Correlation analysis (M4): Finally, we analyzed the correlation between output deviations and low reliability at the instance level, i.e., Pr(f(x) = y∗|x). We expect that large output deviations correlate with low reliability and vice versa. Visually speaking, this means that in a two-dimensional coordinate system where the x-axis corresponds to the output deviations and the y-axis to Pr(f(x) = y∗|x), we expect to observe high reliability on the upper left corner and low reliability on the lower right corner. We conducted a Pearson correlation analysis where the Pearson correlation coefficient must be negative (indicating a negative correlation) to confirm our expectation. In Table I (last column), the correlation coefficient for all DNNs is negative and shows a moderate correlation between our reliability estimates and the output deviations with an average of -0.39. We added also the average output deviation for (in)correct predictions (see columns 8-9). D. Discussion Coming back to our original evaluation goals and questions, we now aggregate the results of the metrics to discuss the evaluation questions and goals. In terms of question (Q1), we considered metrics (M1) and (M2). While the results of metric (M2) were in line with our expectations (all ranks have been preserved by our approach), we observed irregularities for metric (M1) (higher differences in λand λ∗for DNNs trained on the CIFAR10 and HAM10000). Although other DNNs also exhibited differences, they were rather small and can be attributed to the limited size of Deval used for calculating λ∗. For the CIFAR10 dataset, we argue that the deviations are also a result of having a limited dataset Deval. More specifically, while MNIST and Fashion MNIST contain grayscale images with fairly simple backgrounds, simple geometrical shapes, and slight variations in texture and appearance, this is not the case for CIFAR10 which includes colored images with complex backgrounds, wider ranges, and different geometrical shapes and appearances which not only complicates the classification task but also requires more examples compared to MNIST and Fashion MNIST when calculating λ∗. Thus, we argue that with a larger evaluation dataset, λand λ∗would have been closer to each other (note that the evaluation datasets of MNIST, Fashion MNIST and CIFAR10 are identical in size, i.e., |Deval|= 8000). With HAM10000, the evaluation set was not only smaller (i.e. |Deval|= 1500) but also unbalanced, as one class dominated all others by making up a large portion of the dataset (approx. 67%). This not only has a significant impact on the DNN itself but also affects our approach. More specifically, when estimating the GMM, the Gaussian of the dominant class is also dominant in the GMM. During sampling, this leads to a larger number of samples generated for the dominant class, for which the DNN is well trained, leading to higher reliability. In addition, due to the overall small set size of the HAM10000 dataset, we used a subset of the training data for estimating the GMM (i.e., Dgaus ⊂ Dtrain) potentially distorting the true geometric shape of the data manifold. Therefore, we believe that the deviation of λand λ∗results from the combination of the factors mentioned. Looking at the results of MNIST and Fashion MNIST (where not too large evaluation sets are required and which are balanced), it can be seen that the differences between our approximated bounds λand the success probabilities λ∗are small. Considering the results of the metrics (M1) (including the discussion above) and (M2), we conclude that our approximated lower reliability bounds are sound, provided that the evaluation sets have a certain level of quality (e.g., are sufficiently large and balanced). The results of metric (M2) are in line with our expectations such that we can positively answer question (Q2) (the approximated bounds λ≈Success(f)reflect the qualities of distinct DNNs). Thus, we conclude that our approach is plausible in the sense of goal (G1) and questions (Q1) and (Q2). Evaluation question (Q3) asks whether our approach produces a significant deviation in reliability when considering correct and incorrect predictions. If we review the results produced by metrics (M3) and (M4), we can observe such a deviation. However, we have noticed that our approach sometimes assigns high reliability to inputs x, even though they lead to incorrect predictions, and vice versa. We expected a more significant deviation in such cases. We attribute this to DNNs exhibiting very high confidence and accuracy (or vice versa), and to the conformal prediction procedure, which we expect to be highly adaptive and sufficiently approximate conditional coverage. However, because we aim to approximate lower reliability bounds holistically, we argue that this effect averaged out with the number of sampled data points. In terms of evaluation question (Q4), our conclusion is also positive in that low reliability for inputs xcorrelates with high output deviation and vice versa. Coming back to goal (G2), we conclude that our success probabilities correlate with (in)correct predictions. E. Limitations Because we approximate the lower reliability bounds by sampling in the feature space, we can still run into the curse of dimensionality problem. However, since we only generate samples in very small regions of the feature space (those, lying on the data manifold), we consider the problem to be minor. In fact, we did not encounter such problems during the evaluation (all reliability approximations converged quite stable). In algorithm 1, we averaged over the individual reliability estimates (the last line) which implies the assumption that the examples of Dare equally likely. When considering the sampling procedure, this assumption is not quite accurate (because the GMMs are likely to generate samples in higher concentrated regions of the distribution). However, the evaluation results do not indicate that the assumption is too restrictive (which would have been evident in the plausibility results). V. RELATED WORK To the best of our knowledge, no approach determines the reliability (or bounds) of DNNs holistically. The work closest to us refers to Dong et al. [10], introducing a metric to determine the probability of misclassification per random input based on operation profile information and robustness evidence. However, our approach determines lower reliability bounds holistically (i.e., per DNN and not per instance) based on conformal prediction. There are various approaches for analyzing the reliability of a DNN when numerical or hardware defects are observed, e.g. [15], [16]. In contrast, this work does not consider hardware/numerical defects. Uncertainty quantification of DNNs is closely related to our work. In uncertainty quantification, various approaches aim to quantify the predictive uncertainty of DNNs which is a product of the aleatoric and epistemic uncertainty [1]. In conformal prediction, for example, the uncertainty of a prediction is quantified by the size of the corresponding prediction sets. Bayesian methods provide a natural uncertainty measure because the learned model is directly conditioned on the model parameters and thus returns the uncertainty associated with each prediction simultaneously, e.g., [17]. Approaches such as [18] estimate the uncertainty by performing multiple predictions of different trained models or, as in [6], consider the calibration of the softmax output. There are multiple approaches aiming to quantify uncertainty; however, the main difference to our approach is that uncertainty quantification is applied at the instance level, i.e., prediction-wise. Also related to this work are approaches that analyze the data manifold to detect possibly incorrect predictions. The most prominent family of approaches in this area refers to out-of-distribution detection [19], aiming to detect inputs x (observed at runtime) that are unlikely to be generated by the original data distribution and thus prone to generate wrong predictions. Others apply data manifold analyses for network generalization prediction tasks [20] or test generation [21]. In contrast, we are concerned with approximating lower reliability bounds; that is, we identify local (and low dimensional) areas of the data manifold and sample meaningful features. Also in the field of verification, there are approaches related to our work. For example, [22], [23] present approaches to verify robustness properties of neural networks. The verification problem, however, is NP-complete [22] and can only be applied to networks with small model capacity and differs from our work in that we provide no formal guarantees and do not focus on robustness properties. VI. CONCLUSION In this paper, we present an approach to quantify and approximate lower reliability bounds of DNNs. Our approach approximates the lower reliability bounds based on the conformal prediction framework and a sampling procedure, leveraging properties of the data manifold. We demonstrated the plausibility of our approach and showed the correlation of low reliability with high output deviations (and vice versa). In future work, we plan to generalize our approach to regression models and large language models. Moreover, we aim to investigate models other than GMMs (e.g., decision trees) to approximate the data manifold and to overcome implicit assumptions of GMMs. ACKNOWLEDGMENT This work has been performed in the HAL4SDV project, which receives funding within the Chips Joint Undertaking (Chips JU) — the Public-Private Partnership for research, development and innovation under Horizon Europe — and National Authorities under grant agreement no. 101139789. REFERENCES [1] E. H¨ ullermeier and W. Waegeman, “Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods,” Machine learning, vol. 110, no. 3, pp. 457–506, 2021. [2] Y. Tian, K. Pei, S. Jana, and B. Ray, “Deeptest: Automated testing of deep-neural-network-driven autonomous cars,” in Proceedings of the 40th international conference on software engineering, 2018, pp. 303– 314. [3] A. Foong, D. Burt, Y. Li, and R. Turner, “On the expressiveness of approximate inference in bayesian neural networks,” Advances in Neural Information Processing Systems, vol. 33, pp. 15 897–15 908, 2020. [4] A. N. Angelopoulos, S. Bates et al., “Conformal prediction: A gentle introduction,” Foundations and Trends® in Machine Learning, vol. 16, no. 4, pp. 494–591, 2023. [5] M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya et al., “A review of uncertainty quantification in deep learning: Techniques, applications and challenges,” Information fusion, vol. 76, pp. 243–297, 2021. [6] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 2017, pp. 1321–1330. [7] M. Scheerer and R. Reussner, “Reliability analysis of architectural safeguards for ai-enabled systems,” in 2023 27th International Conference on Engineering of Complex Computer Systems (ICECCS). IEEE, 2023, pp. 61–70. [8] E. Asaadi, E. Denney, and G. Pai, “Quantifying assurance in learningenabled systems,” in International Conference on Computer Safety, Reliability, and Security. Springer, 2020, pp. 270–286. [9] A. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan, “Uncertainty sets for image classifiers using conformal prediction,” arXiv preprint arXiv:2009.14193, 2020. [10] Y. Dong, W. Huang, V. Bharti, V. Cox, A. Banks, S. Wang, X. Zhao, S. Schewe, and X. Huang, “Reliability assessment and safety arguments for machine learning components in system assurance,” ACM Transactions on Embedded Computing Systems, vol. 22, no. 3, pp. 1–48, 2023. [11] L. Cayton et al.,Algorithms for manifold learning. eScholarship, University of California, 2008. [12] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. MIT press, 2016. [13] M. Scheerer, M. Take, and J. Klamroth. [Online]. Available: https://github.com/max-sch/ReliabilityDeepNeuralNetwork [14] V. R. B. G. Caldiera and H. D. Rombach, “The goal question metric approach,” Encyclopedia of software engineering, pp. 528–532, 1994. [15] L. Li, Y. Zhang, L. Ren, Y. Xiong, and T. Xie, “Reliability assurance for deep neural network architectures against numerical defects,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). Los Alamitos, CA, USA: IEEE Computer Society, may 2023, pp. 1827–1839. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICSE48619.2023.00156 [16] C. De Sio, S. Azimi, and L. Sterpone, “An emulation platform for evaluating the reliability of deep neural networks,” in 2020 IEEE International Symposium on Defect and Fault Tolerance in VLSI and Nanotechnology Systems (DFT). IEEE, 2020, pp. 1–4. [17] B. Phan, S. Khan, R. Salay, and K. Czarnecki, “Bayesian uncertainty quantification with synthetic data,” in Computer Safety, Reliability, and Security: SAFECOMP 2019 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku, Finland, September 10, 2019, Proceedings 38. Springer, 2019, pp. 378–390. [18] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in Neural Information Processing Systems, 2016. [19] J. Yang, K. Zhou, Y. Li, and Z. Liu, “Generalized out-of-distribution detection: A survey,” arXiv preprint arXiv:2110.11334, 2021. [20] M. O’Brien, B. Wolfinger, J. Bukowski, M. Unberath, A. Pezeshk, and G. Hager, “Mapping dnn embedding manifolds for network generalization prediction,” in 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2023, pp. 6513–6522. [21] T. Byun and S. Rayadurgam, “Manifold for machine learning assurance,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results, 2020, pp. 97– 100. [22] G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer, “Reluplex: An efficient smt solver for verifying deep neural networks,” in International conference on computer aided verification. Springer, 2017, pp. 97–117. [23] G. Singh, T. Gehr, M. P¨ uschel, and M. Vechev, “An abstract domain for certifying neural networks,” Proceedings of the ACM on Programming Languages, vol. 3, no. POPL, pp. 1–30, 2019. APPENDIX PROOF OF THEOREM 1 Proof: Recall the principle of marginal distributions which allows, for any two (or a collection of) random variables Xand Y, summing over marginal variables of the joint distribution: P(X) = PyP(X, Y =y). We apply this principle to Pr(f(x) = y∗|x)by expanding it with a binary random variable that evaluates the events whether the true label y∗is part of the prediction set of the DNN ffor an input xor not, i.e., y∗∈ C(x)or y∗/∈ C(x): Pr(f(x) = y∗|x) = Pr(f(x) = y∗, y∗∈ C(x)|x) +Pr(f(x) = y∗, y∗/∈ C(x)|x) In the next step, we expand the probabilities by applying the following rule: P r(x, y) = Pr(x|y)·P r(y): =Pr(f(x) = y∗|y∗∈ C(x), x)·Pr(y∗∈ C(x)|x) +Pr(f(x) = y∗|y∗/∈ C(x), x) | {z } =0 ·Pr(y∗/∈ C(x)|x) A closer look at Pr(f(x) = y∗|y∗/∈ C(x), x)reveals that the probability must be zero because it is conditioned on the event y∗/∈ C(x). In words, if we know that y∗is not part of the prediction set, then there is no possibility that fpredicts y∗as it selects the label yexhibiting the highest probability of the softmax layer. Moreover, the event f(x) = y∗is conditionally independent of the event of observing input x given the event y∗∈ C(x); that is, the knowledge of xprovides no more information regarding the probability of f(x) = y∗, if event y∗∈ C(x)have already been observed. Consequently, Pr(f(x) = y∗|x)simplifies as follows: Pr(f(x) = y∗|x) = Pr(f(x) = y∗|y∗∈ C(x)) ·Pr(y∗∈ C(x)|x) Ultimately, based on the conditional coverage property (see inequality 2), the following lower bound results: Pr(f(x) = y∗|x)≥Pr(f(x) = y∗|y∗∈ C(x)) ·(1 −ε)