Conformal predictions for probabilistically robust scalable machine learning classification

Vol.:(0123456789) Machine Learning (2024) 113:6645–6661 https://doi.org/10.1007/s10994-024-06571-6 1 3 Conformal predictions forprobabilistically robust scalable machine learning classification AlbertoCarlevaro1,4· TeodoroAlamo3· FabrizioDabbene2· MaurizioMongelli1 Received: 30 December 2023 / Revised: 23 February 2024 / Accepted: 16 May 2024 / Published online: 9 July 2024 © The Author(s) 2024 Abstract Conformal predictions make it possible to define reliable and robust learning algorithms. But they are essentially a method for evaluating whether an algorithm is good enough to be used in practice. To define a reliable learning framework for classification from the very beginning of its design, the concept of scalable classifier was introduced to generalize the concept of classical classifier by linking it to statistical order theory and probabilistic learning theory. In this paper, we analyze the similarities between scalable classifiers and conformal predictions by introducing a new definition of a score function and defining a special set of input variables, the conformal safety set, which can identify patterns in the input space that satisfy the error coverage guarantee, i.e., that the probability of observing the wrong (possibly unsafe) label for points belonging to this set is bounded by a predefined 𝜀 error level. We demonstrate the practical implications of this framework through an application in cybersecurity for identifying DNS tunneling attacks. Our work contributes to the development of probabilistically robust and reliable machine learning models. Keywords Conformal predictions· Scalable classifiers· Confidence bounds· Robust AI Editors: Henrik Boström, Eyke Hüllermeier, Ulf Johansson, Khuong An Nguyen,Aaditya Ramdas. * Alberto Carlevaro alberto.carle[email protected].it Teodoro Alamo [email protected] Fabrizio Dabbene [email protected] Maurizio Mongelli [email protected].it 1 Istituto di Elettronica e di Ingegneria dell’Informazione e delle Telecomunicazioni, CNR, Corso Ferdinando Maria Perrone, 24, 16152Genoa, Italy 2 Istituto di Elettronica e di Ingegneria dell’Informazione e delle Telecomunicazioni, CNR, Corso Duca degli Abruzzi, 24, 10129Turin, Italy 3 Departamento de Ingenierìa de Sistemas y Automàtica, Universidad de Sevilla, Escuela Superior de Ingenieros, Camino de los Descubrimientos, 41092Seville, Spain 4 Funded Research Department, Aitek SpA, Via della Crocetta 15, 16122Genoa, Italy 6646 Machine Learning (2024) 113:6645–6661 1 3 1 Introduction 1.1 Context Conformal predictions (CPs) (Shafer and Vovk 2008) are gaining increasing importance in machine learning (ML) since they validate algorithms in terms of confidence of the prediction. Although it is a fairly recent field of study, there has been an astonishing production of scholarly papers, from the definition of new score functions to different methodologies for constructing conformal sets and, of course, a wide variety of applications. In fact, the ferment of scientific research in this field is so active that even the father of this theory, V. Vovk,1, continues to actively contribute to the improvement of its knowledge, as in the case of Vovk etal. (2017) where he and his colleagues investigate the concept of validity under nonparametric hypotheses or the innovative introduction of Venn predictors as in Vovk etal. (2022). We refer the reader to the surveys (Angelopoulos and Bates 2023; Fontana etal. 2023; Toccaceli 2022) that largely cover all recent publications and discussions on uncertainty quantification (UQ) through CP for machine learning models. Under canonical CP theory, the definition of a score function is very peculiar to either the classifier or the application at hand. For example, Forreryd etal. (2018) defines a special conformity measure (corresponding to a score function), based on the residual between the calibration points and the classification hyperplane of a SVM model. Other example, always SVM-based, can be found in Forreryd etal. (2018), Shafer and Vovk (2008) and Balasubramanian etal. (2009), where different definitions of score function (or conformity/ non-conformity measure) are given. One of the strengths of our approach, as will become clear later, is the unique definition of such a score function, which, given any classifier, allows the conformal prediction framework to be applied in the most natural way. The work in Narteni etal. (2023) defines a score function for rule-based models. The softmax function is used as score function in most image classification problems as in Angelopoulos etal. (2020); Park etal. (2019); Andéol etal. (2024), and many other examples can be provided (see, e.g., the above cited surveys). As a matter of the fact, those definitions come after the setting of the classifier and do not outline a common methodology. 1.2 Contribution By focusing on binary classification, our goal is to introduce to the CP community a way to link ML classifiers with a natural definition of score function that embeds the conformal guarantee by construction. We exploit the concept of scalable classifiers f𝜽( x ,𝜌) (Sect.2.1) introduced in Carlevaro etal. (2023) to develop a new class of score functions that rely on the geometry of the problem and that are naturally built from the classifier itself, by inheriting its properties (Sect.3.1). This allows CP theory to derive the relationship between the input space and the conformity guarantee explicitly. By introducing the new concept of conformal safety region, we provide an analytical form of the specific subsets of the input space in which 1 V. Vovk. is the author of the groundbreaking book Algorithmic Learning in a Random World Vovk etal. (2005), which is the foundation of the theory of conformal prediction. His scientific production is still at the top of research in the field, and a constantly updated list of papers on CP by Vovk and his colleagues can be found here http:// www. alrw. net/. 6647 Machine Learning (2024) 113:6645–6661 1 3 marginal coverage guarantees on prediction (Sect. 3.2) can be ensured. Controlling the misclassification rate (either false positives or false negatives) naturally follows from eliciting the following quantities: the confidence level given by the conformal framework, the binary output y∈ {+1, −1} , the confidence error 𝜀∈(0, 1) , as well as the new notion of conformal safety set S𝜀 that satisfies In short, the paper defines a methodology in which the optimal shape of a classifier is derived, where the optimality criterion is embedded in the classifier by the conformal guarantee. The proposed methodology thus places itself in the recent and as yet unexplored field of set-value classification (Chzhen etal. 2021), a broad theory that studies predictors that have both good prediction properties and specific performance requirements, two points that underlie the proposed research. The remainder of the article is organized by providing a brief recall of the concepts of scalable classifiers and conformal prediction and then delving into the details of the definition of scalable score function and conformal safety region. The whole procedure is then validated on an application use case related to cyber-security for identifying DNS tunneling attacks (Sect.4). 2 Background: scalable classifiers andconformal prediction The background of the theory we would like to propose in this research refers to a new interpretation of classical classification algorithms, scalable classifiers, and another rather new theory on trustworthy AI, called conformal prediction. Both of these techniques belong to the field of reliable AI, searching for the definition of models, procedures or bounds that can make a learning algorithm probabilistically robust and reliable. 2.1 Scalable classifiers Given an input space X⊆ R d , d∈N+ , and an output space Y= {−1, +1} , scalable classifiers (SCs) were introduced in Carlevaro etal. (2023) as a family of (binary) classifiers parameterized by a scale factor 𝜌∈R where the function f𝜽∶X×R ⟶ R is the so-called classifier predictor and the notation with subscript 𝜽 refers to the fact that the classifier also depends on a set of hyperparameters 𝜽 =[𝜽1,⋯,𝜽n 𝜽 ] ⊤ to be set in the model (e.g. different choices of kernel, regularization parameters, etc.). To give a meaningful interpretation of this classifier, we refer to the class +1 as a “safe” situation we want to target and the other class with −1 as an “unsafe” situation. Some examples might be differentiating between a patient’s condition in developing or not developing a certain disease (Lenatti etal. 2022), or understanding what input parameters lead an autonomous car to a collision or non-collision (Carlevaro etal. 2022), among many other applications. SCs rely on the main assumption that for every x∈X , f𝜽( x ,𝜌) is continuous and monotonically increasing in 𝜌 , and that lim 𝜌 → −∞ f 𝜽 ( x ,𝜌)<0<lim 𝜌 → ∞f 𝜽 ( x ,𝜌) , [Carlevaro etal. (2023) Pr{y=−1 and x ∈S𝜀}≤𝜀. (1) 𝜙 𝜽(x,𝜌)≐ { +1 if f𝜽(x,𝜌)< 0, −1 otherwise. 6648 Machine Learning (2024) 113:6645–6661 1 3 Assumption 1]. These assumptions imply that, there exists a unique solution 𝜌 ( x ) to the equation The proof of this claim is available in [Carlevaro etal. (2023) Property 2]. In words, a scalable classifier is a classifier that satisfies some crucial properties: i) given x , there is always a value of 𝜌 , denoted 𝜌 ( x ) , that establishes the border between the two classes, ii) the increase of 𝜌 forces the classifier to predict the −1 class and iii) the target +1 class of a given feature vector x is maintained for a decrease of 𝜌 . Moreover, [Carlevaro etal. (2023) Property 3] shows how any standard binary classifier can be made scalable by simply including the scaling parameter 𝜌 in an additive way with the classifier predictor. That is, given the function  f∶ X⟶ R and its corresponding classifier  𝜙( x ) then the function f 𝜽( x , 𝜌 )= f ( x )+𝜌 provides the scalable classifier 𝜙𝜽( x ,𝜌) Thus, examples of classifiers that can be rendered scalable are support vector machine (SVM), support vector data description (SVDD), logistic regression (LR) but also artificial neural networks. More in detail, given a learning set containing observed feature points and corresponding labels, zi = ( x i ,y i) , and assuming that 𝜑∶X ⟶ V represents a feature map (where V is an inner product space) that allows to exploit kernels, some examples of scalable classifier predictors are: • SVM: f 𝜽 (x,𝜌)=w ⊤ 𝜑(x)−b+ 𝜌 , • SVDD: f𝜽(x,𝜌)=‖𝜑(x)−w‖2−R2+𝜌 , • LR: f 𝜽(x , 𝜌)= 1 2 − 1 1+e( w⊤𝜑(x)−b ) +𝜌 , where the classifier elements w,b and R can be obtained as solution of proper optimization problems. The interested reader can refer to [Carlevaro etal. (2023) SectionIIc] for a more in depth discussion. Different values of the parameter 𝜌 correspond to different classifiers that can be considered as the level sets of the classifier predictor with respect to 𝜌 . In particular, since we are interested in predicting the class +1 which, we recall, encodes a safety condition, we introduce that is the set of points x∈X predicted as safe by the classifier with the specific choice of 𝜌 , i.e. the safety region of the classifier f𝜽 for given 𝜌 . It is easy to see that these sets are decreasingly nested with respect to 𝜌 , i.e. 2.2 Conformal prediction Conformal Prediction is a relatively recent framework developed in the late nineties by V. Vovk. We refer the reader to the surveys (Angelopoulos and Bates 2023; Shafer and Vovk 2008; Fontana etal. 2023) for a gentle introduction to this methodology. CP is mainly an (2) f𝜽( x ,𝜌)=0. Z 𝓁≐ {( x i ,y i)}n i=1 ⊆X×{−1, +1 } (3) S(𝜌)={ x ∈X∶f𝜽( x ,𝜌)<0}, 𝜌1>𝜌 2 ⟹ S(𝜌1)⊂S(𝜌2). 6649 Machine Learning (2024) 113:6645–6661 1 3 a-posteriori verification of the designed classifier, and in practice returns a measure of its “conformity” to the calibration data. We consider the particular implementation of CP discussed in Angelopoulos and Bates (2023), relative to the so-called “inductive" CP: in this setting, starting from a given predictor and a calibration set, CP allows to construct a new predictor with given probabilistic guarantees. To this end, the first key step is the definition of a score function s∶X×Y⟶R . Given a point x∈X and a candidate label y∈ {−1, 1} , the score function returns a score s( x ,y). Larger scores encode worse agreement between point x and the candidate label y . Then, assume to have available a second set of nc observations, usually referred to as calibration set, defined as follows that are pairs of points x with their corresponding true label y. We assume that the observations xi∈Xc come from the same distribution Pr of the observations in the test set Zts = {(x i ,y i )} n ts i=1 =X ts ×Y ts ⊆X×Y . Additionally, CP requires that the data are exchangeable, which is a weaker assumption than that of i.i.d.. Exchangeability means that the joint distribution of the data z1,z2,…,zn is unchanged under permutations: Then, given a user-chosen confidence rate (1−𝜀)∈(0, 1) , a conformal set C𝜀( x ) is defined as the set of candidate labels whose score function is lower than the (⌈(nc+1)(1−𝜀)⌉∕nc) -quantile, denoted as s𝜀 , computed on the s 1 ,…,s n c calibration scores. That is, to every point x , CP associates a set of “plausible labels" The usefulness of the conformal set is that, according with Vovk etal. (1999), C𝜀( x ) possesses the so-called marginal conformal coverage guarantee property, that is, given any (unseen before) observation ( x ,y) , the following holds In other words, the true label y belongs with high probability – at least ( 1−𝜀 ) – to the conformal set. 3 Notion ofscore function forscalable classifiers andconformal safety sets In this section we introduce two concepts: i) a definition of score function for scalable classifiers (see Definition 1) and ii) the notion of conformal safety region (see Definition 2). 3.1 Natural definition ofscore function forscalable classifiers In this paragraph, we show how scalable classifiers allow for a natural definition of the score function, based on their own classifier predictor. (4) Zc ≐ { (x i ,y i ) }n c i=1 =X c ×Y c ⊆X×Y , ( z 1, z 2,…, z n)∼( z 𝜎(1), z 𝜎(2),…, z 𝜎(n)), for all permutations 𝜎. C𝜀( x )={y∈ {−1, 1}∶s( x ,y) ≤ s𝜀}. (5) Pr { y∈C 𝜀 ( x) } ≥1−𝜀 . 6650 Machine Learning (2024) 113:6645–6661 1 3 Definition 1 [Score Function for Scalable Classifier] Given a scalable classifier 𝜙𝜃( x ,𝜌) with classifier predictor f𝜃( x ,𝜌) , given a point x and an associated candidate label y , the score function associated to the scalable classifier is defined as with 𝜌 ( x ) such that f𝜽( x ,𝜌 ( x )) = 0 . We notice that, since f𝜽 is a SC predictor, the existence and uniqueness of such 𝜌 ( x ) is guaranteed (Sect.2.1) and consequently s is well defined. In practice, the score function evaluates how much it is necessary to vary the original classification boundary f𝜽( x ,0) such that the point x falls on the classification boundary of the new classifier f𝜽( x ,𝜌 ( x )) , starting from class y . Alternatively, it is possible to think of the score function as a measure of the “difficulty” of making the classifier predict a certain class: very large values for 𝜌 ( x ) imply that it is difficult to render f𝜽( x ,𝜌 ( x )) positive, or equivalently that the class −1 is not conformal (thus, when y=−1 , the score function is 𝜌 (x)=−y 𝜌 (x) ). Very negative values of 𝜌 ( x ) imply that it is difficult to render the output equal to +1 , thus the score function is in this case − 𝜌 (x)=−y 𝜌 ( x ) . Example 1 Scalable SVDD is the most straightforward example of correctly understanding such a definition for score function. In this case the score function takes this form This represents exactly the quantity that needs to be removed ( y=+1 for the point inside the sphere, ‖ w −𝜑( x )‖2−R2<0 ) or added ( y=−1 for the point outside the sphere, R 2 −‖ w − 𝜑 ( x )‖2 > 0 ) to the radius such that x falls on the boundary of the classifier. For example, consider two classes of points, “safe” ( +1 , in blue in the following figure) and “unsafe” ( −1 , in red in the following figure) sampled from two two-dimensional Gaussian distributions with respectively means and covariance matrices where I is the identity matrix. We trained a linear SVDD classifier (Fig.1a) and plotted the respectively score function (Fig. 1b). Exactly the behavior described above can be observed: the score function associates values according to the geometry provided by the classifier. In this case, points belonging to the boundary of the circumference have score function values of 0 (dashed green line) and negative or positive depending on whether the point is inside or outside the circumference. It is worth noting that the classifier can be interpreted as a level set of the score function, and this interpretation is crucial as will become clear in the following. 3.2 Conformal safety regions Classical CPs define subsets of the output space that satisfy the probabilistic marginal coverage constraint, but it is equally important to understand the relationship between the input space and the conformal sets. In other words, it would be meaningful to define s( x ,y)=−y𝜌 ( x ) s (x,y)=−y � R2− ‖ w−𝜑(x) ‖2�. 𝜇 S= [ −1 −1 ] ,𝜎S=1 2 I;𝜇U= [ +1 +1 ] ,𝜎U=1 2I 6651 Machine Learning (2024) 113:6645–6661 1 3 regions in the input space classified on the basis of the conformal set of their samples to identify for which inputs the classifier is most reliable in making a certain prediction. For example, one should be interested in finding the region of classification uncertainty ( C𝜀( x )={−1, +1} ) or the region in which the conformal classifier predicts a specific label ( C𝜀( x )={+1} or C𝜀( x )={−1} ) or in which it has no guess at all ( C𝜀( x )=∅ ). In particular, since the goal is to find the input values that bring the classification to a “safe” situation (i.e., in our notation, y=+1 ) with a certain level of confidence, we introduce the concept of conformal safety region. Definition 2 [Conformal Safety Region] Consider a calibration set Zc = {(x i ,y i )} n c i=1 from the same data distribution of the test set Zts . Given a level of error 𝜀∈(0, 1) , a score function s∶X×Y ⟶ R , and its corresponding (⌈(nc+1)(1−𝜀)⌉∕nc) -quantile s𝜀 computed on the calibration set, the conformal safety region (CSR) of level 𝜀 is defined as follows In words, a conformal safety region (CSR) is the subset of the input space where the conformal set is composed by only safe labels, C𝜀( x )={+1} , which can be inferred directly from the definition. Note that the above definition is independent on the choice of the score function s. What we will prove in the next is that using the score function defined for SCs (Definition 1) it is possible to give an analytical form to Σ𝜀 . Example 2 Consider the same configuration as in Example 1 but with covariance matrices 𝜎S=𝜎U=I and with a probability to sample an outlier for each class pO=0.1 . Consider the LR classifier and its corresponding score function (6) Σ𝜀={ x ∈X∶s( x ,+1) ≤ s𝜀,s( x ,−1)>s𝜀}. s( x ,y)=−y(b− w ⊤𝜑( x )), Fig. 1 Relationship between the SVDD classifier and the corresponding score function: the absolute value of the score function assigns to a sample its distance to the circumference boundary. The color bar on the right helps to understand the behavior of the score function: darker colors indicate regions with less conformity with the target class, warmer the opposite. The zero value of the score function is obtained exactly on the boundary 6652 Machine Learning (2024) 113:6645–6661 1 3 which is the same of the SVM since the solution of the equation f𝜽( x ,𝜌)=0 is for both 𝜌 ( x )=b− w ⊤𝜑( x ) . We trained on a training set composed by 3000 samples the LR classifier with cubic polynomial kernel (Fig.2a) and then we computed the score values on a calibration set of 5000 samples. We computed the quantiles varying 𝜀 (0.05,0.1 and 0.5) and we plotted (on a test set of 10000 samples) the scatter of the points according to the conformal set. Green points belong to the CSR Σ𝜀 and it is easily understandable that the smaller 𝜀 is, the smaller Σ𝜀 . This behavior is in line with CP theory, since small values of 𝜀 mean that the conformal prediction must be very precise, and this is achievable only if the classifier itself is “very confident” of assigning the true label to a sample. Also, it should be noted that the smaller 𝜀 is, the larger the region of uncertainty for the conformal prediction ( C𝜀( x )={−1, +1} , in yellow in Figs.2a, 2b). Again, since for small 𝜀 high levels of marginal coverage must be satisfied, conformal prediction tends to give both labels to a point when it is uncertain. Contrarily, for high values of 𝜀 (Fig.2c) the conformal sets for uncertain points tend to be empty (in black) because the score is too high and no output meets the specifications to belong to C𝜖 . Finally, it is worth noting that the regions into which the points scatter have a well-defined shape: as introduced in Example 1 and as will become clear in the next section, these regions correspond to level sets of the score function. Fig. 2 Scatter-plots of the conformal set varying 𝜀 for cubic LR. Green and red points correspond to singleton conformal set ( C𝜀( x )={+1} and C𝜀( x )={−1} respectively) yellow points to double predictions ( C𝜀( x )={+1, −1} ) and black points to empty prediction ( C𝜀( x )=∅ ) 6653 Machine Learning (2024) 113:6645–6661 1 3 3.3 Analytical form ofconformal safety regions forscalable classifiers The definition of score we gave for SCs in Definition 1 identifies a particular value of the scalable parameter, which is the one corresponding to the quantile s𝜀 , that we can define formally as To this value, we can associate a level set S(𝜌𝜀) defined as in (3), i.e. the 𝜌𝜀 -safe set We can prove that non-trivial relationships link S𝜀 to the CSR Σ𝜀 . But before, let us split Σ𝜀 in two contribution: where and The relationship between S𝜀 and Σ𝜀 is explored in the following results that provide as final and major contribution the fact that S𝜀⊆Σ𝜀 . Proposition 1 Proof ◻ Corollary 1 Proof Trivial, from ◻ (7) 𝜌𝜀 ≐|s𝜀|. (8) S𝜀={ x ∈X∶ f 𝜽( x,𝜌 𝜀) <0 }. (9) Σ𝜀 =Σ a 𝜀 ∪Σ b 𝜀, (10) Σa 𝜀={ x ∈X∶ s ( x, + 1 ) <s 𝜀 ,s ( x, − 1 ) >s 𝜀}, (11) Σb 𝜀 ={x∈X∶s(x,+1)=s 𝜀 ,s(x,−1)>s 𝜀 } . S𝜀=Σ a 𝜀⊆Σ𝜀. S𝜀 =Σ 𝜀 only ifΣ b 𝜀=∅. Σ𝜀 =Σ a 𝜀 ∪Σ b 𝜀 =S 𝜀 ∪Σ b 𝜀 . x∈S 𝜀⟺ f 𝜽 ( x ,|s 𝜀 |)<0, ⟺f𝜽(x,|s𝜀|)<f𝜽(x,𝜌 (x)), ⟺|s𝜀|< 𝜌 (x), ⟺−s𝜀< 𝜌 (x)and s𝜀< 𝜌 (x), ⟺−s𝜀<−s(x,+1)and s𝜀<s(x,−1) , ⟺s(x,+1)<s𝜀and s(x,−1)>s𝜀, ⟺x∈Σ a 𝜀 ⊆Σ 𝜀 . 6660 Machine Learning (2024) 113:6645–6661 1 3 Acknowledgements The authors would like to thank Anastasios Angelopulos of University of California, Berkeley and Sara Narteni of CNR-IEIIT for their thoughtful suggestions about Conformal Prediction. Author contributions Alberto Carlevaro: Methodology, Validation, Investigation, Software, Data curation, Writing. Teodoro Alamo: Methodology, Validation, Investigation, Writing, Supervision. Fabrizio Dabbene: Methodology, Validation, Investigation, Writing, Supervision. Maurizio Mongelli: Methodology, Validation, Investigation, Data curation, Writing, Supervision. Funding Open access funding provided by Consiglio Nazionale Delle Ricerche (CNR) within the CRUICARE Agreement. This work was supported in part by REXASI-PRO H-EU project, call HORIZON-CL4- 2021-HUMAN-01-01, Grant agreement ID: 101070028. The work was also supported by Future Artificial Intelligence Research (FAIR) project, Italian Recovery and Resilience Plan (PNRR), Spoke 3 - Resilient AI. Moreover T. Alamo acknowledges support from grant PID2022-142946NA-I00 funded by MCIN/AEI/ 10.13039/501100011033 and by ERDF, A way of making Europe. Data availability The datasets generated and analysed during the current study are not publicly available due to potentially sensitive data from the CNR-IEIIT’s internal network but are available from the corresponding author on reasonable request. Data availability The authors prefer not to make the codes available yet. It can be requested from the corresponding author upon reasonable request. Declarations Conflict of interest There are no either Conflict of interest or Conflict of interest to declare. Ethics approval Not Applicable. Consent to participate Not Applicable. Consent for publication Not Applicable. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. References Abellán, J., Klir, G. J., & Moral, S. (2006). Disaggregated total uncertainty measure for Credal sets. International Journal of General Systems, 35(1), 29–44. Andéol, L., Fel, T., De Grancey, F., & Mossina, L. (2024). Conformal prediction for trustworthy detection of railway signals. AI and Ethics, 4, 1–5. Angelopoulos, A. N., & Bates, S. (2023). Conformal prediction: A gentle introduction. Foundations and trends®in machine learning, 16(4), 494–591. https:// doi. org/ 10. 1561/ 22000 00101 Angelopoulos, A.N., Bates, S., Jordan, M. & Malik, J. (2020) Uncertainty sets for image classifiers using conformal prediction. In: International conference on learning representations Aiello, M., Mongelli, M., & Papaleo, G. (2015). DNS tunneling detection through statistical fingerprints of protocol messages and machine learning. International Journal of Communication Systems, 28(14), 1987–2002. Balasubramanian, V.N., Gouripeddi, R., Panchanathan, S., Vermillion, J., Bhaskaran, A. & Siegel, R. (2009) Support vector machine based conformal predictors for risk of complications following a coronary drug eluting stent procedure. In: 2009 36th annual computers in cardiology conference (CinC), (pp. 5–8) . IEEE 6661 Machine Learning (2024) 113:6645–6661 1 3 Carlevaro, A., Alamo, T., Dabbene, F. & Mongelli, M. (2023) Probabilistic safety regions via finite families of scalable classifiers arXiv: 2309. 04627 [stat.ML] Carlevaro, A., Lenatti, M., Paglialonga, A., & Mongelli, M. (2022). Counterfactual building and evaluation via explainable support vector data description. IEEE Access, 10, 60849–60861. https:// doi. org/ 10. 1109/ ACCESS. 2022. 31800 26 Carlevaro, A., & Mongelli, M. (2021). A new SVDD approach to reliable and eXplainable AI. IEEE Intell Syst. https:// doi. org/ 10. 1109/ MIS. 2021. 31236 69 Chzhen, E., Denis, C., Hebiri, M. & Lorieul, T. (2021) Set-valued classification–overview via a unified framework. arXiv: 2102. 12318 [stat.ML] Fontana, M., Zeni, G., & Vantini, S. (2023). Conformal prediction: A unified review of theory and new challenges. Bernoulli, 29(1), 1–23. https:// doi. org/ 10. 3150/ 21- BEJ14 47 Forreryd, A., Norinder, U., Lindberg, T., & Lindstedt, M. (2018). Predicting skin sensitizers with confidence–Using conformal prediction to determine applicability domain of gard. Toxicology in Vitro, 48, 179–187. https:// doi. org/ 10. 1016/j. tiv. 2018. 01. 021 Hüllermeier, E., & Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110, 457–506. https:// doi. org/ 10. 1007/ s10994- 021- 05946-3 Lenatti, M., Carlevaro, A., Guergachi, A., Keshavjee, K., Mongelli, M., & Paglialonga, A. (2022). A novel method to derive personalized minimum viable recommendations for type 2 diabetes prevention based on counterfactual explanations. PLOS ONE, 17(11), 1–24. https:// doi. org/ 10. 1371/ journ al. pone. 02728 25 Narteni, S., Carlevaro, A., Dabbene, F., Muselli, M. & Mongelli, M. (2023) Confiderai: Conformal interpretable-by-design score function for explainable and reliable artificial intelligence. In: Conformal and probabilistic prediction with applications, (pp. 485–487). Park, S., Bastani, O., Matni, N. & Lee, I. (2019) Pac confidence sets for deep neural networks via calibrated prediction. In: International conference on learning representations. Park, S., Dobriban, E., Lee, I., & Bastani, O. (2022). PAC prediction sets for meta-learning. Advances in Neural Information Processing Systems, 35, 37920–37931. Sale, Y., Bengs, V., Caprio, M. & Hüllermeier, E. (2023) Second-order uncertainty quantification: A distance-based approach arXiv: 2312. 00995 [cs.LG] Sale, Y., Caprio, M. & Hüllermeier, E. (2023) Is the volume of a credal set a good measure for epistemic uncertainty? arXiv: 2306. 09586 [cs.LG] Shafer, G., & Vovk, V. (2008). A tutorial on conformal prediction, 9, 371–421. http:// jmlr. org/ papers/ v9/ shafe r08a. html. Toccaceli, P. (2022). Introduction to conformal predictors. Pattern Recognition, 124, 108507. https:// doi. org/ 10. 1016/j. patcog. 2021. 108507 Vaccari, I., Carlevaro, A., Narteni, S., Cambiaso, E., & Mongelli, M. (2022). eXplainable and reliable against adversarial machine learning in data analytics. IEEE Access, 10, 83949–83970. https:// doi. org/ 10. 1109/ ACCESS. 2022. 31972 99 Valiant, L. (2013). Probably approximately correct: Nature’s algorithms for learning and prospering in a complex world. USA: Basic Books Inc. Vovk, V. (2012) Conditional validity of inductive conformal predictors. In: Hoi, S.C.H., Buntine, W. (eds.) Proceedings of the Asian conference on machine learning. Proceedings of machine learning research, vol. 25, pp. 475–490. PMLR, Singapore Management University, Singapore . https:// proce edings. mlr. press/ v25/ vovk12. html Vovk, V., Gammerman, A. & Saunders, C. (1999) Machine-learning applications of algorithmic randomness. In: Proceedings of the sixteenth international conference on machine learning. ICML ’99, pp. 444–453. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic learning in a random world. Berlin, Heidelberg: Springer. Vovk, V., Gammerman, A., & Shafer, G. (2022). Probabilistic classification: Venn predictors (pp. 157– 179). Cham: Springer. https:// doi. org/ 10. 1007/ 978-3- 031- 06649-8_6 Vovk, V., Shen, J., Manokhin, V. & Xie, M. (2017) Nonparametric predictive distributions based on conformal prediction. In: Conformal and probabilistic prediction and applications, pp. 82–102 Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.