Improving the Results in Credit Scoring by Increasing Diversity in Ensembles of Classifiers

Received 19 May 2023, accepted 5 June 2023, date of publication 8 June 2023, date of current version 15 June 2023. Digital Object Identifier 10.1109/ACCESS.2023.3284137 Improving the Results in Credit Scoring by Increasing Diversity in Ensembles of Classifiers SERAFÍN MORAL-GARCÍA AND JOAQUÍN ABELLÁN Department of Computer Science and Artificial Intelligence, University of Granada, 18071 Granada, Spain Corresponding author: Serafín Moral-García ([email protected].es) This work was supported in part by UGR-Fondo Europeo de Desarrollo Regional (FEDER) Funds under Project A-TIC-344-UGR20; and in part by the ‘‘FEDER/Junta de Andalucía-Consejería de Transformación Económica, Industria, Conocimiento y Universidades’’ under Project P20_00159. ABSTRACT The ensembles of classifiers are techniques that have obtained excellent results in the credit scoring domain. It is known that Decision Trees (DTs) are suitable for ensembles because they encourage diversity, the key point for the success of an ensemble scheme. Ensembles of DTs have obtained good performance in a wide range of areas, including credit scoring. Some works have highlighted that DTs that employ imprecise probability models, called Credal Decision Trees (CDTs), improve the results of ensembles in credit scoring. The performance of CDT is strongly influenced by a hyperparameter. In fact, it was shown that different values of the hyperparameter might yield different models. Hence, the diversity in ensemble schemes can be increased by randomly selecting the value of the hyperparameter in each CDT, instead of fixing one. In this work, it is shown that increasing the diversity of the ensembles that use CDT by varying the value of the hyperparameter in each base classifier improves the results in credit scoring. Thereby, the use of CDT randomly selecting the value of the hyperparameter would suppose notable economic benefits for banks and financial institutions. Few gains in accuracy might imply huge gains in economic benefits. INDEX TERMS Credit scoring, ensemble schemes, diversity, base classifiers, random credal decision tree. I. INTRODUCTION Credit scoring is one of the main tools to analyze credit risk. It consists of a set of methods that allow classifying credit applicants into two classes: good or bad. These techniques help to make decisions about granting credit to an applicant. In consequence, credit scoring systems are very useful for banks and financial institutions. Actually, any slight improvement would produce great benefits in terms of faster decisions, risk reduction, and less cost in credit analysis [1]. So far, many approaches have been applied to credit scoring. They can be divided into statistical methods and Artificial Intelligent techniques [2]. The methods belonging to the first category assume previous knowledge about the data. These prior assumptions are not always realistic. In contrast, Artificial Intelligence techniques do not need prior knowledge about the data to extract information directly from them. Artificial Intelligence approaches, especially from the Data The associate editor coordinating the review of this manuscript and approving it for publication was Jad Nasreddine . Mining field, yield better results than statistical methods in credit scoring [3], [4], [5]. Within Data Mining, many individual algorithms have been applied to credit scoring, such as Support Vector Machines (SVM) [6], [7], [8], Decision Trees (DT) [9], [10], [11], K-nearest neighbors (KNN) [12], Neural Networks (NN) [13], [14], [15], [16] and Bayesian Networks (BN) [17], [18], [19]. Nonetheless, the data mining techniques that have achieved the best results in credit scoring are the ensembles of classifiers [20], [21], [22], [23], [24]. They consider multiple individual classifiers and, then, the pieces of information obtained by them are combined to make a final prediction. Experimental studies have been carried out to determine which base classifiers are the most suitable for ensembles in credit scoring [21], [25]. NNs have obtained satisfactory results as individual classifiers for credit scoring [14], but they perform worse than other individual learners in ensemble schemes for credit scoring datasets [25], [26]. The reason is that ensembles do not have to use very accurate and complex individual learners. As pointed out by Breiman [27], the key VOLUME 11, 2023 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ 58451 S. Moral-García, J. Abellán: Improving the Results in Credit Scoring by Increasing Diversity issue for the success of an ensemble scheme is that the base classifiers are not only accurate but also diverse or unstable. Several experimental studies have been carried out to compare the performance of several individual classifiers in the main ensemble schemes for credit scoring [21], [28]. These studies have shown that the base classifiers that obtain the best results are the unstable classifiers, not necessarily the most accurate as individual learners. Decision trees (DTs) are individual classifiers that are built fast and are easy to interpret. DTs are well-known to be very unstable since, with these classifiers, little variations in the training sets might give rise to considerable variations in the learned models. Thereby, DTs are very suitable individual learners to employ in ensembles because they enhance diversity for the combination of classifiers [27]. DTs based on imprecise probabilities, called Credal Decision Trees (CDTs), were proposed in [29]. They use uncertainty measures on credal sets (closed and convex sets of probability distributions) in the tree-building process. CDTs have obtained good results, especially with class noise in the data [30], [31], [32]. Furthermore, in [33], it was highlighted that CDTs also have very good performance when they are used in Bagging schemes. In addition, CDTs have shown to be the individual learners that provide the best results in ensembles of classifiers for credit scoring [28], [34]. CDTs have an important hyperparameter that strongly influences their performance [31], [35]. As shown in [36], different values of the hyperparameter may lead to different trees when they are built with the same training set. Therefore, when CDT is the base classifier of an ensemble scheme, diversity can be increased by varying the value of the hyperparameter in each tree. Considering the previous points, in this work, we propose the use of a base classifier in ensemble schemes for credit scoring datasets called Random Credal Decision Tree (RCDT). It consists of a version of the CDT algorithm that randomly chooses the value of the hyperparameter before building the tree within a range of possible values. Hence, the value might be different for each base classifier of the ensemble in such a way that the diversity of the ensembles is increased. An experimental analysis is carried out in this work with several credit scoring datasets and the main ensemble schemes that have been employed in credit scoring in the literature. As the individual classifiers, we use RCDT, as well as the ones that achieved the best results in ensembles in an experimental study for credit scoring carried out in [28]. This analysis shows that, as expected, RCDT is the individual classifier that obtains the best results in ensembles for credit scoring datasets. This paper is arranged as follows: In Section II, the main ensemble schemes used for credit scoring are detailed. Section III describes Credal Decision Trees. Section IV introduces the Random Credal Decision Tree algorithm. Our experimental study is detailed in Section V. Section VI concludes this paper. II. ENSEMBLES OF CLASSIFIERS In many areas of science, it is very common to combine multiple opinions to make a decision. In this way, that decision is probably better than the one made from a single opinion. This idea has also been applied in Machine Learning via ensemble schemes. They consider multiple classifiers and, for classifying an instance, the predictions made by the individual learners are normally combined via a scheme vote. In the literature, the combination of information obtained from different classifiers has supposed improvement in classification over the use of single classifiers. Indeed, ‘‘with ensembles of classifiers, the predictions tend to be more accurate, and the robustness is increased’’ [37]. The following ensemble approaches, very known in the data mining area, have been employed: •Bagging [27]: ‘‘This method creates Msamples S1,...,Smrandomly drawn from the original training set with replacement. The size of the samples coincides with the size of the original set. Hence, in each sample Si, 1 ≤i≤M, some instances might appear more than once, whereas others may not appear. A classifier Ci is trained from each sample Si,∀i=1,...,M. Since each classifier is built with a different training set, the learners are often different from each other. To classify an instance, the predictions made by the classifiers are combined via a majority vote’’. •Boosting [38]: ‘‘It also creates Msamples randomly choosen with replacement. However, unlike Bagging, the resampling is directed to obtain the most informative data for each consecutive learner. When an instance is required to be classified, the predictions of the individual classifiers, weighted by their accuracy, are combined’’. The most known Boosting method is Adaboost [39]. In this algorithm, ‘‘the successive samples are obtained by re-weighting the training instances. Initially, the same weight is assigned to all instances. In each iteration, these weights are adjusted depending on the classification errors made by the resulting base learners in such a way that instances erroneously classified are more probable to appear in the next sample’’ [39]. •Random Subspace [40]: This approach considers M classifiers built with Fattributes randomly selected from the original attribute set, that is, each classifier is learned with the same instances as the original training set but using only Ffeatures. Experiments have revealed that ‘‘the standard value of Fthat yields good results is equal to half of the total number of attributes’’ [41]. In order to classify an instance, the predictions are combined via the majority vote. •DECORATE [42] (Diverse Ensemble Creation by Oppositional Relabelling of Artificial Training Examples): ‘‘It generates an ensemble by learning a new classifier in each iteration. The first base classifier is built with the original training set. The remaining learners are built with an artificial training set resulting from the union of the original set and artificial instances, 58452 VOLUME 11, 2023 S. Moral-García, J. Abellán: Improving the Results in Credit Scoring by Increasing Diversity which are generated from the data distribution and obtained by probabilistically estimating the value of each attribute [42]. The labels of the artificial instances are selected so that they maximally differ from the current predictions. Hence, diversity is increased. For maintaining training accuracy, the classifier is added to the ensemble scheme if, and only if, its incorporation does not decrease the performance of the ensemble’’. •Rotation Forest [43]: ‘‘This ensemble approach separates the predictive variables into Fnon-overlapping subsets equally sized. As in bagging, bootstrap sampling is carried out. Then, a principal component analysis (PCA) is run separately in each subset, and a new set of variables is extracted by integrating all principal components. Thus, Kaxis rotations happen to obtain the new attributes used to build the base classifier. Each base learner of the ensemble is built from the bootstrap sample in a rotate feature space’’. A. DIVERSITY IN ENSEMBLES In ensemble schemes, there is no much gain combining very similar classifiers. Consequently, ‘‘to the ensemble schemes are successful, the base classifiers must be not only accurate, but few variations in training data have to give rise to considerable changes in the model, which is known as instability or diversity’’ [44]. For the previous reason, Decision Trees (DTs) are frequently used in ensemble schemes; they are very simple and unstable models, which enhance the diversity of the ensembles of classifiers. Experimental analyses carried out in [28] and [34] have shown that the base classifiers than perform better in ensemble schemes for credit scoring are not the ones that achieve the best results as individual learners, but DTs. It is because DTs are accurate and diverse classifiers and, thus, very suitable for ensembles. Other ways for increasing diversity in the set of base classifiers for an ensemble method are: •Randomly selecting samples from the original training set with replacement to build each base classifier. Examples of ensemble methods that use this procedure are Bagging and Rotation Forest. Some areas of the search space of the problem may not be studied by the base classifiers because these areas are hidden by the more frequent samples. With the random selection of instances for each base classifier, some base classifiers can study these zones with less frequent samples and, in this way, highlight some interesting characteristics for improving the accuracy and robustness of the ensemble method. •Randomly selecting attributes from the original set of variables to build each base classifier. Examples of ensemble methods with this property are Random Subspace and Random Forest. Some attributes might not be used by the base classifiers because they are hidden by other more important attributes according to the ranking of the variables established by the base classifier. However, these hidden attributes can provide interesting information for the ensemble classifier. Hence, the random selection of attributes can give an opportunity to the hidden attributes for providing their knowledge to the ensemble method. Thereby, this method can improve its accuracy and generalization. III. CREDAL DECISION TREES Decision Trees (DTs) have been widely used in the classification task since the publication of the ID3 algorithm [45]. A few years later, Quilan also proposed the C4.5 classifier [46]. Since then, C4.5 has been a standard classification algorithm. Furthermore, DTs have been commonly applied to many areas, such as biology, medicine, and astronomy. In a DT structure, each node corresponds to an attribute or feature. Each branch between a node and its child is associated with a possible value of that feature, and each leaf or terminal node is labeled with a class value. Once the tree has been built, to classify a new instance, ‘‘a path from the root node to a leaf is followed using its attribute values. The predicted class value is the one labeled to the terminal node’’ [41]. The tree-building process is determined by the following points: 1) The criterion utilized to choose the feature to insert in a node, i.e, the split criterion. 2) The criterion to stop ramifying. 3) The criterion to assign a class label to the terminal nodes. 4) Optionally, a DT can use a final post-pruning process to simplify the tree structure and reduce the over-fitting. A terminal node is labeled with the most frequent class value in it. Regarding point 2, the branching is stopped when there is no information gain for any attribute. Hence, the main difference among the different DTs is the split criterion. Let Cbe the class variable and {c1,...,ck}its possible values. Let Ddenote the dataset associated with a node. Let Xbe an attribute that takes values in {x1,...,xt}. Classical DTs use precise probabilities in the split criterion. Specifically, the basis of the split criterion in them is the Shannon Entropy [47] of the class variable, defined as: HD(C)= k X i=1 −p(ci) log p(ci).(1) The average entropy that derives from the feature Xis determined by: HD(C|X)= t X j=1 PD(X=xj)HDj(C|X=xj),(2) PD(X=xj) being the probability that X=xjin Dand Djthe set of instances of Dsuch that X=xj,∀j=1,2,...,t. VOLUME 11, 2023 58453 S. Moral-García, J. Abellán: Improving the Results in Credit Scoring by Increasing Diversity The split criterion used in many classical DTs, called InfoGain, is defined as follows: IG(C,X)D=HD(C)−HD(C|X).(3) In these trees, the attribute that provides the maximum value of IG is selected for branching. Credal Decision Trees (CDTs) were proposed by Abellán and Moral [29]. In the split criterion, they utilize uncertainty measures on credal sets, i.e, closed and convex sets of probability distributions. Specifically, CDTs employ the Imprecise Dirichlet Model (IDM) [48], a formal imprecise probability model that uses probability intervals. The IDM estimates that, ∀j= 1,2,...,k, the probability that C=cj, denoted by p(cj), is within the following interval: p(cj)∈ncj N+s, ncj+s N+s,(4) where Nis the number of instances in the dataset, ncjis the number of instances in the dataset that verify that C= cj,∀j=1,...,k, and s is a given hyperparameter. It is easy to check that IDM intervals are wider as the svalue is higher. In [48], a definitive recommendation for the svalue was not given, although two values were suggested: s=1 and s=2, and it was recommended s=1. These probability intervals lead to the following credal set on C[49]: KD(C)=   p|p(cj)∈ncj N+s, ncj+s N+s, ∀j=1,...,k, k X j=1 p(cj)=1   .(5) Uncertainty measures can be applied on this credal set. The maximum entropy is the one used in the split criterion of CDTs. It is determined by: H∗(KD(C)) =max nHD(p)|p∈KD(C)o(6) The maximum entropy is a well-established uncertainty measure because it verifies the required properties. It is consistent with the principle of maximum uncertainty [50], which states that ‘‘the probability distribution that attains the maximum entropy, compatible with the available restrictions, should be chosen’’ [51]. The procedure to obtain H∗was exposed in [49]. It reaches its lowest computational cost when s≤1. The split criterion utilized in CDTs, called the Imprecise Info-Gain (IIG) [29], is similar to IG. Nevertheless, the former is based on the maximum entropy in KD(C), whereas IG uses the Shannon entropy. Formally, IIG is defined as: IIGD(C,X)=H∗(KD(C)) −H∗(KD(C|X)) (7) where the value of H∗(KD(C|X) is obtained similarly to HD(C|X) in IG. Algorithm 1summarizes the building procedure of a CDT. Algorithm 1 Building Procedure of a CDT Procedure Build_CDT(Node N, set of features F) if F= ∅ then Exit end 1. Let Dbe the dataset corresponding to N if |D|<minimum number of instances then Exit end 2. max_entropy =maxX∈FIIGD(C,X) if max_entropy ≤0then Exit end else 3. Let X′∈Fbe the attribute that reaches max_entropy 4. Assign X′to N 5. F′=F\x′ for xipossible value of X′do 6. Add a node Nichild of N 7. Build_CDT(Ni,F′) end end A. CLASSICAL DTs VS CREDAL DECISION TREES When s=0, the credal set determined by Equation (5) only contains the probability distribution corresponding to relative frequencies, and, consequently, CDTs and classical DTs are identical. If s>0, then IDM intervals are wider as the size of the set (N) is larger. Thus, at the upper levels of the tree, where Nis often very large, IIG and IG tend to obtain similar values and, therefore, classical DTs and CDTs may behave similarly. Nonetheless, at the lower levels of the tree, where N is usually small, the IDM credal set probably contains many probability distributions far different from the one used to compute the Shannon entropy. Hence, at the lower levels of the tree, CDT and classical DTs might have notably different behavior since IIG and IG may provide very different values. Unlike IG, the value of IIG can be negative for an attribute [26]. In consequence, CDTs avoid choosing attributes that worsen the information about the class variable. Thereby, CDTs might stop branching the tree before classical DTs, which means that CDTs usually overfit less the data than classical DTs. As pointed out in [35], CDTs are less sensitive to noise than DTs based on precise probabilities. Moreover, in [28], [34], it was highlighted that CDTs perform much better than classical DTs when they are employed in ensembles for credit scoring datasets. IV. RANDOM CREDAL DECISION TREE The Random Credal Decision Tree algorithm (RCDT) randomly chooses a value for the shyperparameter before 58454 VOLUME 11, 2023 S. Moral-García, J. Abellán: Improving the Results in Credit Scoring by Increasing Diversity building the tree between a given range of possible values. Recall that this value is used in the calculation of the split criterion of the algorithm. Then, the tree is built in the same way as CDT. Thus, the building process of an RCDT can be summarized as follows: 1) Select s′a random value of the shyperparameter. 2) Build the tree using s′in the split criterion. For classifying a new instance, the RCDT algorithm carries out the same procedure as CDT. Concerning the range of values for the shyperparameter, in this work, we use {1,1.5,2,2.5,3,3.5}, the one established as appropriated in [36].1 A. JUSTIFICATION OF RCDT In [36] and [49], it was demonstrated that, if s1>s2, then the credal set determined by Equation (5) with s1as the value of the shyperparameter contains the credal set corresponding to s2. In consequence, the maximum entropy value for the class variable does not decrease as the svalue is higher. Also, for an attribute X, if s1>s2, then the maximum entropy of the class variable conditioned on Xis greater for s1than for s2. Given two attributes X1and X2, it is possible that H∗ s1(C|X1)−H∗ s2(C|X1)>H∗ s1(C|X2)−H∗ s2(C| X2), where H∗ siis the maximum entropy on the credal set corresponding to s=si, for i=1,2. Hence, different values of the shyperparameter might yield different split attributes via the IIG criterion. This is highlighted in the following example. Example 1: Suppose that we have two class values, namely c1and c2. Let us assume that, in a certain node, we have 5 instances for which C=c1and 10 instances for which C=c2. Suppose that, in this node, there are three attributes X1,X2, and X3, and that each one of them takes values in xi 1,xi 2, for i=1,2,3. Let us assume the following class frequencies for each attribute value: X1=x1 1→(nc1=4,nc2=10) X1=x1 2→(nc1=1,nc2=0) X2=x2 1→(nc1=4,nc2=4) X2=x2 2→(nc1=1,nc2=6) X3=x3 1→(nc1=3,nc2=9) X3=x3 2→(nc1=2,nc2=1) We have the following maximum entropy values for s=0, s=1 and s=2: H∗(KD(C))s=0=0.9183. H∗(KD(C))s=1=0.9544. H∗(KD(C))s=2=0.9774. The change of the maximum entropy value when spasses from 0 to 1 is notably higher than when spasses from 1 to 2. 1We do not use s=0 because, in such a case, RCDT would be equivalent to classical DTs. We do also not employ very larger values of s since it would suppose a quite high pruning in the trees [26]. Regarding the conditioned entropies, we have the following values: H∗(KD(C|X1))s=0=0.8056. H∗(KD(C|X1))s=1=0.9237. H∗(KD(C|X1))s=2=0.9575. H∗(KD(C|X2))s=0=0.8094. H∗(KD(C|X2))s=1=0.9119. H∗(KD(C|X2))s=2=0.9619. H∗(KD(C|X3))s=0=0.8327. H∗(KD(C|X3))s=1=0.9124. H∗(KD(C|X3))s=2=0.9522. Again, the most notable changes are produced when s passes from 0 to 1. The values of IIG for each attribute and for each attribute and value of the shyperparameter are the following ones: IIGs=0(C,X1)=H∗(KD(C))s=0−H∗(KD(C|X1))s=0 =0.1127. IIGs=0(C,X2)=H∗(KD(C))s=0−H∗(KD(C|X2))s=0 =0.1089. IIGs=0(C,X3)=H∗(KD(C))s=0−H∗(KD(C|X3))s=0 =0.0856. IIGs=1(C,X1)=H∗(KD(C))s=1−H∗(KD(C|X1))s=1 =0.0307. IIGs=1(C,X2)=H∗(KD(C))s=1−H∗(KD(C|X2))s=1 =0.0425. IIGs=1(C,X3)=H∗(KD(C))s=1−H∗(KD(C|X3))s=1 =0.0420. IIGs=2(C,X1)=H∗(KD(C))s=2−H∗(KD(C|X1))s=2 =0.0199. IIGs=2(C,X2)=H∗(KD(C))s=2−H∗(KD(C|X2))s=2 =0.0155. IIGs=2(C,X3)=H∗(KD(C))s=2−H∗(KD(C|X3))s=2 =0.0252. In this way, when s=0, the selected feature is X1. However, the X2attribute is chosen when s=1, whereas, for s=2, the feature X3is selected. Consequently, in this case, the choice of the svalue decisively influences the split attribute. The above example shows that the value of the shyperparameter might be crucial for the split attribute in a node. Hence, different svalues may lead to different tree structures. In this way, in an ensemble, the diversity is increased by randomly choosing the value of the shyperparameter in each CDT. For this reason, in ensemble schemes, it is more suitable to employ RCDT as the base classifier than CDT with the same svalue for all trees. This is validated in Section Vwith exhaustive experimentation. VOLUME 11, 2023 58455 S. Moral-García, J. Abellán: Improving the Results in Credit Scoring by Increasing Diversity TABLE 1. Datasets description. V. EXPERIMENTS A. EXPERIMENTAL SETUP We have based on the experimental study about the performance of individual classifiers in ensembles for credit scoring datasets carried out in [28], considering the same experimental setting. In this experimental analysis, six credit scoring datasets have been employed. Table 1allows us to observe the main characteristics of each dataset: number of instances, number of attributes, and percentages of instances labeled as good/positive and bad/negative. The datasets Australian,German, and Japanese can be found in the UCI Machine Learning repository [52]. The Iranian dataset [53] ‘‘comes from the corporate client of the work of a small private bank in Iran’’. The Polish dataset derives from the research carried out in [54] about companies of bankruptcy forecast. The UCSD dataset ‘‘is a reduced version of a very large dataset employed in the 2007 Data Mining Contest of the University of California, San Diego’’ [41]. In our experimental analysis, five ensemble schemes have been used: Adaboost, Bagging, Random Subspace, DECORATE, and Rotation Forest. They were described in Section II. Regarding the base classifiers, we have used RCDT and the ones that achieved the best results in [28]: LOGR, C4.5, and CDT. We aim to analyze the performance of the base classifiers in each ensemble for the credit scoring datasets considered in this research. For our experimentation, we have utilized the Weka software [55]. The implementations provided in this software for the ensemble schemes have been employed, as well as the implementations for LOG-R, C4.5, and CDT. We have added the structures and methods required for using the RCDT algorithm. Consistently with the experimental analysis carried out in [28], for CDT, we have fixed the svalue to 1, and the post-pruning process has not been considered. In both CDT and RCDT, the missing values and continuous attributes have been treated as in the C4.5 algorithm [26]. Remark that, for RCDT, the range of values {1,1.5,2,2.5,3,3.5}has been used, the one established as the most suitable in [36]. The rest of the parameters for the algorithms have been the ones given by default in Weka. As in the experimental analyses carried out in [21] and [28], for each pair ensemble/classifier and dataset, a 5-fold cross-validation procedure has been repeated 50 times. Regarding the evaluation of the performance, remark that Accuracy is not the most suitable evaluation measure for credit scoring since, in this field, a false positive may have worse consequences than a false negative. For this reason, as in [28], we have not only used Accuracy, but we have also considered the area under the ROC curve (AUC). It is a wellestablished evaluation measure in the literature for binary classification problems where the error costs are different. Following the recommendations given in [56], for statistical comparisons in Accuracy and AUC, we have used the following statistical tests to compare the results obtained by more than two classifiers, with a level of significance of α=0.1: •Friedman test [57]: ‘‘It is a non-parametric test that separately ranks the algorithms for each dataset (the best-performing algorithm is assigned to rank 1, the second-best to rank 2, and so on). The null hypothesis of the Friedman test is that all algorithms perform equivalently’’. •If the null hypothesis of the Friedman test is rejected, then all algorithms are compared to each other via the Holm test [58]. B. RESULTS AND DISCUSSION In order to analyze the results, we principally consider Friedman’s ranks for Accuracy and AUC. In the literature, it has been considered an important reference when it is required to compare the performance of several algorithms [56]. Apart from the order in the performance, we also want to know the cases in which the differences are statistically significant. In some cases, the differences among the base classifiers are hardly notable. Nevertheless, as said previously, small gains in performance might imply important gains in economical benefits, which is the main goal. 1) ACCURACY RESULTS Table 2shows the average accuracy results obtained by each ensemble with each base classifier for each dataset. It also allows us to see Friedman’s ranks of the base classifiers in each ensemble scheme. For each ensemble, the best result is marked with bold fonts and the second-best with italic fonts. Table 3presents a summary of Friedman’s ranks obtained by each base classifier in Accuracy. Specifically, it shows the Friedman ranks of the base classifiers in each one of the ensemble schemes, as well as the average Friedman rank for each base classifier. Taking into account Tables 2and 3, we express the following comments about the Accuracy results for each base classifier: 1) LogR: •It is the worst base classifier concerning Friedman’s ranks in all ensembles, except for DECORATE, where it achieves the second-best results. •This base classifier obtains the worst average Friedman rank. •It obtains 9 of the best performances out of 30 by ensemble/dataset. For only 1 pair 58456 VOLUME 11, 2023 S. Moral-García, J. Abellán: Improving the Results in Credit Scoring by Increasing Diversity TABLE 2. Average accuracy results obtained by each base classifier in each dataset grouped by ensemble. TABLE 3. Results of the Friedman’s ranks about accuracy of each base classifier. ensemble/dataset, it achieves the second-best performance. •German and Japanese are the datasets in which Log-R performs better. For these datasets, it obtains the best results for four ensembles. 2) C4.5: •It obtains the second-worst average Friedman rank, after LogR. •C4.5 occupies the second position regarding Friedman’s ranks, tied with CDT, in Adaboost and Bagging. It obtains the worst Friedman rank in DECORATE, and it achieves the second-best performance in Rotation Forest. •This base classifier has 4 wins by ensemble/dataset. For 8 combinations ensemble/dataset, C4.5 occupies the second position. •Iranian is the dataset for which C4.5 performs better. Indeed, all its wins occur in this dataset. 3) CDT: •It obtains the second-best average Friedman rank. •In 3 of the 5 ensemble schemes considered, CDT achieves the second-best Friedman rank (tied with C4.5 in Adaboost and Bagging). It is not the worst base classifier in any ensemble. •This base classifier does not obtain the best result for any combination ensemble/dataset. Nevertheless, for 13 pairs ensemble/dataset, it obtains the second-best performance. •CDT achieves the best results in the UCSD dataset. 4) RandomCDT: •It achieves the best Accuracy results by far; it obtains the best Friedman rank for all ensembles. TABLE 4. Summary of the results of the Holm tests corresponding to Accuracy. In each column, the base classifier significantly outperforms the ones in the row in the ensembles indicated in the cell. Furthermore, it obtains a much lower average Friedman rank than other base classifiers. •For 18 pairs out of 30 ensemble-dataset, RandomCDT achieves the best performance, and, for other 7 pairs, it obtains the second-best results. •The most notable differences among RandomCDT and the rest of the base classifiers can be found in Rotation Forest and Random Subspace. In these two ensemble schemes, RandomCDT obtains the best results. •Even though in DECORATE and Adaboost RandomCDT performs better than the remaining base classifiers, in these two ensemble schemes, RandomCDT achieves the worst results. •It is convenient to remark that, for the Polish and UCSD datasets, RandomCDT always achieves the best performance. For Australian, it also obtains the best results in all ensembles, except for DECORATE, in which it obtains the second-best performance. Table 4summarizes the results of the Holm tests corresponding to Accuracy for the cases in which there are statistically significant differences via the Friedman test. As can be observed, there are only statistically significant differences in Random Subspace, where RandomCDT outperforms LogR. C4.5 and CDT obtain statistically equivalent results to all base classifiers for all the ensemble schemes considered here. 2) AUC RESULTS Table 5presents the average AUC results obtained by each base classifier in each ensemble scheme for each dataset. It also shows the Friedman rank of each base classifier in each ensemble scheme. Similar to Table 2, the best results are noted in bold and the second-best results in italic. Table 6 summarizes the Friedman’s ranks results for AUC. Specifically, it shows the Friedman rank obtained by each base classifier in each ensemble scheme, as well as the average Friedman rank for each base classifier. From Tables 5and 6, the following issues can be observed about each one of the base classifiers for the AUC measure: 1) LogR: •This base classifier occupies the third position in the average Friedman rank. •LogR has the worst performance with Adaboost, whereas it achieves the best results with VOLUME 11, 2023 58457 S. Moral-García, J. Abellán: Improving the Results in Credit Scoring by Increasing Diversity TABLE 5. Average AUC results obtained by each base classifier in each dataset grouped by ensemble. TABLE 6. Results of the Friedman’s ranks in AUC for each base classifier. DECORATE. LogR is the second-best in Bagging, Random Subspace, and Rotation Forest. •It achieves 11 of the best results out of 30 by ensemble-dataset. For 4 combinations ensemble/dataset, it obtains the second-best result. •LogR obtains very good results for the Australian,German, and Japanese datasets, except in Adaboost. 2) C4.5: •This base classifier obtains the worst results, except in AdaBoost, where it occupies the second position. •For 3 pairs out of 30 ensemble-dataset, it obtains the best results, and, for 4 pairs, it achieves the second-best performance. •There is no dataset for which C4.5 obtains good results. 3) CDT: •It achieves the second-lowest average Friedman rank. •CDT occupies the third position in all ensembles regarding Friedman’s ranks, except in Adaboost, where it achieves the best results. •It is the best method for 2 combinations ensemble/dataset, and, for 13 combinations, it obtains the second-best performance. •UCSD is the dataset for which CDT achieves the best results. 4) RandomCDT: •It is the clear winner base classifier for the AUC measure; it obtains the lowest average Friedman rank. TABLE 7. Summary of the results of the Holm tests corresponding to AUC. In each column, the base classifier significantly outperforms the ones in the row in the ensembles indicated in the cell. •RandomCDT achieves the best Friedman rank in 3 of 5 ensembles, the second position for DECORATE, and the third one for Adaboost. However, it is easy to check that, in general, Adaboost is the ensemble that obtains the worst AUC results. •For 14 pairs ensemble/dataset out of 30, it obtains the best performance, and, for 9 pairs, it achieves the second-best results. •This base classifier is always the best for the Polish dataset. It also obtains very good results for Iranian and UCSD, except with Adaboost. Table 7allows us to see a summary of the results of the Holm tests associated with AUC when there are statistically significant differences according to the Friedman test. Taking into account Table 7, we can describe the following cases of statistical differences for each base classifier: •LogR: It performs significantly worse than C4.5 and CDT in Adaboost, whereas it obtains significantly better results than C4.5 in DECORATE. •C4.5: The results obtained by this base classifier are significantly better than the ones obtained by LogR in Adaboost. Nonetheless, in DECORATE, it performs significantly worse than LogR and RandomCDT. It is also significantly outperformed by RandomCDT in Random Subspace. •CDT: It does not perform significantly worse than any base classifier for any ensemble, and it obtains significantly better results than LogR in Adaboost. •RandomCDT: It performs significantly better than C4.5 in DECORATE and Random Subspace. As CDT, RandomCDT does not perform worse than any base classifier for any ensemble scheme. A financial expert is interested in optimizing the predictive model for a credit scoring dataset. For this reason, we analyze, for each credit scoring dataset considered here, for which combination ensemble/base classifier the best AUC results are attained. •Australian: The best result in this dataset is obtained by Random Subspace with RCDT as the base classifier, and the second-best one by Bagging with LogR. •German: Rotation Forest with LogR as the base classifier gets the best result in this dataset. The secondbest result in German is also achieved by Rotation Forest using RCDT. 58458 VOLUME 11, 2023 S. Moral-García, J. Abellán: Improving the Results in Credit Scoring by Increasing Diversity FIGURE 1. Average Friedman rank for each base classifier in Accuracy and AUC. •Japanese: In this dataset, the best result is obtained by Rotation Forest using LogR and the second-best one by Bagging also using LogR. •Iranian,Polish and UCSD: The best results in these datasets are got by Rotation Forest with RCDT as the base classifier, and the second-best ones by Bagging also employing RCDT. 3) SUMMARY OF THE RESULTS The following issues summarize the results obtained in this experimental study: •Figure 1shows the average Friedman ranks obtained by the base classifiers. It is easy to observe that RandomCDT is the base classifier that achieves, in general, the best results in both Accuracy and AUC. •RandomCDT obtains the best Friedman ranks for almost all ensembles, and it is the base classifier that achieves the best results by ensemble/dataset. The only ensemble where it does not obtain good results in AUC is Adaboost. Nonetheless, Adaboost is the ensemble scheme that performs worst in AUC. •Therefore, we can state that RandomCDT is the winner base classifier among the ones considered in this work. The reason is that, as we argued in Section IV-A, the RandomCDT algorithm is quite suitable for ensembles because it encourages diversity, which is the most important issue in ensembles. •The ensemble schemes where RandomCDT achieves the best results are Random Subspace and Rotation Forest. •LogR obtains the worst results in Accuracy, whereas it achieves the second-best results in AUC, although, with Adaboost, it obtains the worst performance for this measure. •The C4.5 algorithm obtains the third position in Accuracy and the worst results in AUC. We can state that it is the worst base classifier among the ones considered in this research in terms of Friedman ranks and wins ensemble-dataset. In this way, DTs based on classical Probability theory are not the most appropriate to employ in ensembles for credit scoring. •CDT achieves the second-best results in Accuracy and AUC concerning Friedman ranks and wins ensembledataset. It achieves the best results for very few combinations ensemble-dataset, but CDT obtains the second-best performance for a considerable number of pairs ensemble-dataset. •Concerning the combinations of ensemble/dataset that lead to the best results in each dataset, it must be remarked that Rotation Forest and Bagging are the ensemble schemes that achieve the best performance in almost all datasets. The base classifier that gets the best results in these ensembles is RCDT, followed by LogR. •From a financial point of view, the important result here might be the performance of the final procedure (ensemble +base classifier) on each dataset. We can conclude that, for each dataset, our proposal, RCDT join with a type of ensemble is always the best one or the second best in results on the AUC measure. It is the first one in that measure for 4 of those 6 datasets, and the second one on the other 2 but with results very close to the best ones. For example, for the Japanese dataset, the best procedure is Rotation Forest +LogR with 0.9349 value, and the second one is Random Subspace+RCDT with 0.9344 value. VI. CONCLUSION AND FUTURE WORK Improving credit scoring methods is an essential issue for banks and financial institutions. Slight improvements may produce great benefits. Ensembles of classifiers are the techniques that have achieved the best results in credit scoring. In this work, we have completed previous experimental studies about the use of base classifiers in ensemble schemes for credit scoring datasets. Specifically, we have considered a CDT, an algorithm that has obtained the best results in ensembles for credit scoring and strongly depends on a hyperparameter. This algorithm takes into account the lack of precision of the information obtained from data. The proposed version, called Random Credal Decision Tree (RCDT), randomly chooses the value of the hyperparameter, among a range established as suitable in a previous study, before building the tree. As pointed out previously, different values of the hyperparameter might give rise to different trees when they are built with the same training set. In this way, RCDT encourages more diversity than CDT when both methods are used in ensembles. Remark that ‘‘the key point for the success of an ensemble scheme is that the base classifiers are not only accurate but also diverse and unstable’’ [44]. Thus, the RCDT method is suitable to be used as the base classifier in ensemble schemes. An exhaustive experimental analysis has been carried out in this research with several credit scoring datasets. In such an analysis, the main ensemble schemes considered in the literature for credit scoring have been utilized. As the base classifiers, we have considered RCDT and the ones that achieved the best results in ensembles for credit scoring in previous experimental studies: LogR, C4.5, and CDT. It has VOLUME 11, 2023 58459