Enhancing multi-class classification in FARC-HD fuzzy classifier: On the synergy between n-dimensional overlap functions and decomposition strategies

1 Enhancing multi-class classification in FARC-HD fuzzy classifier: On the synergy between n-dimensional overlap functions and decomposition strategies Mikel Elkano, Mikel Galar, Jose Sanz, Alberto Fern´ andez, Edurne Barrenechea, Francisco Herrera, Member, IEEE, and Humberto Bustince, Member, IEEE Abstract—There are many real-world classification problems involving multiple classes, e.g., in bioinformatics, computer vision or medicine. These problems are generally more difficult than their binary counterparts. In this scenario, decomposition strategies usually improve the performance of classifiers. Hence, in this paper we aim to improve the behaviour of FARC-HD fuzzy classifier in multi-class classification problems using decomposition strategies, and more specifically One-vs-One (OVO) and One-vs-All (OVA) strategies. However, when these strategies are applied on FARC-HD a problem emerges due to the low confidence values provided by the fuzzy reasoning method. This undesirable condition comes from the application of the product t-norm when computing the matching and association degrees, obtaining low values, which are also dependent on the number of antecedents of the fuzzy rules. As a result, robust aggregation strategies in OVO such as the weighted voting obtain poor results with this fuzzy classifier. In order to solve these problems, we propose to adapt the inference system of FARC-HD replacing the product t-norm with overlap functions. To do so, we define n-dimensional overlap functions. The usage of these new functions allows one to obtain more adequate outputs from the base classifiers for the subsequent aggregation in OVO and OVA schemes. Furthermore, we propose a new aggregation strategy for OVO to deal with the problem of the weighted voting derived from the inappropriate confidences provided by FARC-HD for this aggregation method. The quality of our new approach is analyzed using twenty datasets and the conclusions are supported by a proper statistical analysis. In order to check the usefulness of our proposal, we carry out a comparison against some of the state-of-the-art fuzzy classifiers. Experimental results show the competitiveness of our method. Index Terms—Multi-classification, one-vs-one, fuzzy rule-based classification systems, aggregations, overlaps. This work was supported in part by the Spanish Ministry of Science and Technology under projects TIN2011-28488, TIN-2012-33856 and TIN-201340765-P and the Andalusian Research Plan P10-TIC-6858 and P11-TIC-7765. Mikel Elkano, Mikel Galar, Jose Sanz, Edurne Barrenechea, and Humberto Bustince are with the Departamento de Autom´ atica y Computaci´ on, Universidad P´ ublica de Navarra, Navarra, 31006 Spain (e-mail: [email protected], {mikel.galar, joseantonio.sanz, edurne.barrenechea, bustince}@unavarra.es) Alberto Fern´ andez is with the Department of Computer Science, University of Ja´ en, Ja´ en, 23071 Spain (e-mail: [email protected]). Francisco Herrera is with the Department of Computer Science and Artificial Intelligence, CITIC-UGR (Research Center on Information and Communications Technology), University of Granada, Granada, 18071 Spain and with the Faculty of Computing and Information Technology - North Jeddah, King Abdulaziz University, Jeddah, 21589 Saudi Arabia (e-mail: [email protected].es) I. INTRODUCTION FUZZY Rule-Based Classification Systems (FRBCSs) are well-known and widely used tools in the field of pattern recognition and classification problems. They provide an interpretable model by using linguistic labels in the antecedents of their rules [1]. FRBCSs have been applied in multiple realworld problems, including domotics [2], anomaly intrusion detection [3], image processing [4], and medical problems [5], among others. In classification, two types of problems can be differentiated depending on the number of classes that compose the output of the problem: binary (two classes) and multi-class problems (more than two classes). Usually, it is more difficult to build a classifier in the latter case due to the overlapping among the examples of the different classes of the problem, which makes the definition of decision boundaries more complex. Even so, multi-class problems are present in several applications domains such as fingerprints recognition [6], handwritten digits [7], microarrays [8] or cardiovascular disease classification [5]. A commonly used solution to deal with multi-class classification problems is to use decomposition techniques [9], [10], which try to divide the original multi-class problem into easier to solve binary classification problems, which are faced by independent binary classifiers named base classifiers. Different decomposition strategies have been proposed in the specialized literature [10]. Two of the most well-known and used ones are One-vs-One (OVO) and One-vs-All (OVA) [9], which can be included within the wider Error Correcting Output Codes (ECOC) framework [11]. OVO scheme divides the original problem into as many sub-problems as possible pairs of classes, whereas in OVA the division results in as many sub-problems as classes in the original one. In both strategies each binary problem is addressed by an independent base classifier. When classifying a new example, the outputs of all the base classifiers are combined to make the final decision (aggregation phase). In this paper we aim to improve the performance of FARCHD (Fuzzy Association Rule-based Classification model for High-Dimensional problems) [12] in multi-class problems using decomposition strategies. Previous works have shown that although the base classifier can be capable of solving multiclass problems, such as FARC-HD, these strategies usually 2 work better than addressing the problem directly [9], [13]– [16]. However, two main difficulties emerge when carrying out the proposed hybridization: 1) The aggregation of the classifiers in OVA and OVO schemes directly depends on the confidences provided by the base classifiers. In the case of FARC-HD, due to the usage of additive combination [17] as fuzzy reasoning method, we consider as confidence the sum of the association degrees obtained for each class, which are computed by multiplying the matching degrees (of the example with the antecedents of the rules using the product t-norm to model the conjunction) and the rule weight. For this reason, when combining FARC-HD and decomposition strategies the confidences obtained when carrying out the inference process of FARC-HD are not suitable for the subsequent aggregation. From our point of view, this is due to the usage of the product in the inference of FARC-HD, which produces small confidences with low variations for each pair of classes and penalizes the rules with the largest number of antecedents. 2) Aggregation strategies that usually have a robust and accurate performance in OVO, such as the weighted voting (WV) [18], [9], [19], do not obtain good results when using FARC-HD as base classifier (while others do not present this problem). Our hypothesis is that the confidence estimation of the non-predicted class provided by FARC-HD distorts the combination in OVO when using this aggregation strategy. In order to address the former problem, we propose to adapt the inference process of FARC-HD in such a way that the confidences obtained allow decomposition strategies to produce more accurate aggregations and consequently, can lead to improve the classification in OVO and OVA models. To do so, we will use overlap functions [20], which satisfy similar properties to those of the product, in the inference of FARCHD. These functions allow us to obtain values with a higher variation than those provided by the product, in such a way that the confidences used in OVO and OVA (stored in the score-matrix or vector) are better modeled, and hence greater knowledge is acquired for the posterior aggregation. Since overlaps functions are originally defined for 2dimensional problems, in this paper we introduce the concept of n-dimensional overlap functions to be able to compute the overlap among ninput values. More specifically, according to the problem related to the number of antecedents, we propose the usage of n-dimensional overlap functions that do not decrease the results as the number of input values (n) increases. In addition, aiming at facing the latter problem, we propose an alternative to the usage of the WV in the aggregation phase of OVO strategy. To do so, we propose an aggregation strategy named WinWV, which follows the idea of the WV, in which we do not consider the confidences obtained by non-predicted classes, since its usage is not appropriate for the classification in the case of FARC-HD (we will show that OVA and other OVO aggregations are not affected by this problem). In order to assess the quality of the methods, we use twenty numerical datasets from the KEEL dataset repository [21] and we contrast the results obtained using non-parametric statistical tests, as suggested in the specialized literature [22], [23]. In these experiments, we will study the goodness of the usage n-dimensional overlap functions and we will also analyze whether the usage of WinWV allows the performance of the WV to be enhanced. Moreover, we will show the validity of our proposal to improve the performance of FARC-HD in multi-class problems, comparing it against the original FARCHD algorithm and some of the best performing fuzzy methods, i.e., FURIA algorithm [24], IVTURSFARC-HD [25], and PTTD [26]. The rest of this paper is organized as follows. In Section II, we briefly introduce FARC-HD and decomposition strategies and we describe some of the aggregations for OVO that we use in this paper. Section III contains a detailed description of our proposals to use FARC-HD with decomposition strategies and puts forward the definition of the n-dimensional overlap functions. The set-up of the experimental framework is given in Section IV and the analysis of the results obtained is presented in Section V. Finally, Section VI concludes this paper. II. PRELIMINARIES In this section, we first recall some concepts about FRBCSs and we briefly explain the FARC-HD algorithm [12] (Section II-A). Then we describe OVO and OVA decomposition strategies and some of the OVO aggregation methods studied in the literature (Section II-B). Finally, we review the related works in Section II-C. A. Fuzzy Rule-Based Classification Systems and FARC-HD A classification problem consists in learning a mapping function called classifier from a set of training examples, named training set, that allows one to classify previously unknown examples. Let xp= (xp1, . . . , xpn)be the pth example of the training set which is composed of Pexamples, where xpi is the value of the i-th attribute (i= 1,2, . . . , n) of the p-th training example. Each example belongs to a class yp∈C={C1, C2, ..., Cm}, where mis the number of classes of the problem. We find multiple techniques used to cope with classification problems. Among them, FRBCSs are widely used because they provide an interpretable model by means of the use of linguistic labels in their rules [1]. The two main components of FRBCSs are the following ones: 1) Knowledge base: It is composed of both the rule base (RB) and the database, where the rules and the membership functions are stored, respectively. 2) Fuzzy reasoning method: This is the mechanism that classifies examples using the information stored in the knowledge base. In this work we focus on a fuzzy rule learning algorithm known as FARC-HD [12], since it is currently one of the 3 most accurate and interpretable FRBCSs in the literature. This algorithm makes use of the following rule structure: Rule Rj:If x1is Aj1and . . . and xnis Ajn then Class = Cjwith RWj (1) where Rjis the label of the j-th rule, x= (x1, . . . , xn)is an n-dimensional pattern vector that represents the example, Aji is a fuzzy set, Cj∈Cis the class label and RWjis the rule weight, which is computed using the most common specification, i.e., the fuzzy confidence value or certainty factor defined in [27]: RWj=CFj=X xp∈ClassCj µAj(xp) P X p=1 µAj(xp) (2) where µAj(xp)is the matching degree of the example xpwith the antecedent part of the fuzzy rule Rj, which is computed using the Eq. (3), shown further in this subsection. In the case of FARC-HD, linguistic labels are modeled using uniformly distributed triangular membership functions, which form a strong partition (Fig. (1)). 1.0 0.0 0.0 1.0 Fig. 1. Linguistic labels partitioning in FARC-HD. In order to generate the rule base, FARC-HD applies a learning process composed of three steps: 1) Fuzzy association rule extraction for classification: With the aim of obtaining the fuzzy rule base, a search tree [28] is constructed for each class. To do so, the frequent itemsets (an item is a linguistic label) are computed using the support and confidence. Finally, the fuzzy rules are generated from the obtained frequent itemsets. The number of linguistic terms in the antecedents of the rules is limited by the maximum depth of the tree. 2) Candidate rule prescreening: This phase makes use of subgroup discovery to preselect the most interesting rules from the rule base obtained in the previous stage by means of a pattern weighting scheme [29]. The weights of the examples are based on the coverage of the fuzzy rules. 3) Genetic rule selection and lateral tuning: An evolutionary algorithm is used both to perform a lateral tuning of the fuzzy sets [30] and to select the most accurate rules from the rule base generated in the previous steps. Let xp= (xp1, ..., xpn)be a new example to be classified, FARC-HD applies a fuzzy reasoning method called additive combination [17], computed in four steps. 1) Matching degree. In this step the strength of activation of the if -part for all rules in the RB with the pattern xp is computed. µAj(xp) = T(µAj1(xp1), . . . , µAjnj(xpnj)) (3) where µAji (xpi)is the matching degree of the example with the i-th antecedent of the rule Rj,Tis a t-norm (in the case of FARC-HD the product), and njis the number of antecedents of the rule. 2) Association degree. The association degree of the pattern xpwith each rule in the RB is computed. bj(xp) = µAj(xp)·RWj(4) 3) Confidence degree. In this stage the confidence degree for each class is computed. To obtain the confidence degree of a class, the association degrees of the rules of that class, i.e., those whose consequent class is the class we are considering, are summed. confl(xp) = X Rj∈RB;Cj=l bj(xp), l = 1,2, . . . , m (5) 4) Classification. The class that obtain the highest confidence degree is the predicted one. Class =arg max l=1,...,m(confl(xp)) (6) As we can observe in the rule structure and the fuzzy reasoning method, FARC-HD is capable of solving multi-class classification problems directly. However, previous works have shown that decomposition strategies usually work better than addressing the problem directly [9], [10], [13]–[16]. Therefore, we propose to use the decomposition strategies with the aim of improving the performance of FARC-HD when facing multiclass problems. However, as we have stated in the introduction, we will show that there are some issues we need to address when combining FARC-HD and these strategies. B. Decomposition strategies Decomposition strategies [10] divide the original multiclass problem into simpler binary problems that are faced by independent binary classifiers, which are referred as base classifiers. These strategies can be used both with classifiers that are able to face only two-class classification problems and those having an inherent multi-class support. In this paper we consider two of the most used decomposition strategies in the literature: One-Versus-One (OVO) and One-Versus-All (OVA) [9] strategies. 1) One-Versus-One (OVO): OVO decomposition divides a problem of mclasses into m(m−1)/2binary sub-problems (all possible pairs of classes). Each problem is faced by a binary classifier, which is responsible for distinguishing a pair of classes {Ci, Cj}. When classifying a new example, each base classifier will return a pair of confidence degrees rij, rji ∈[0,1] in favour of classes Ci, Cj, respectively (if only a confidence degree is given for the predicted class, the other is usually computed as rji = 1 −rij , being Cithe predicted one). The outputs (confidence degrees) provided by all the base classifiers are stored in the score-matrix Ras 4 follows: R=     −r12 · · · r1m r21 − · · · r2m . . .. . . rm1rm2· · · −      (7) Since each binary sub-problem is faced by an independent classifier, we normalize the score-matrix such that the range of the confidences provided by all classifiers is the same. This normalization is very important when using FARC-HD because it does not return confidences in [0,1] that can be interpreted as probabilities. The score-matrix is normalized as follows. rij =(rij rij +rji if rij 6= 0 or rji 6= 0 0.5if rij =rji = 0 (8) Finally, the outputs of the base classifiers are aggregated and the predicted class is obtained. This aggregation step is a key factor for the classification process [9]. In this paper, we consider four well-known OVO aggregation methods. •Voting strategy (VOTE) [31]. Each base classifier votes for the predicted class and the class having the largest number of votes is given as output: Class =arg max i=1,...,m X 1≤j6=i≤m sij (9) where sij is 1 if rij > rji and 0 otherwise. •Weighted Voting (WV) [18]. Each base classifier votes for both classes based on the confidences obtained for them. The class having the largest value is given as output: Class =arg max i=1,...,m X 1≤j6=i≤m rij (10) •Non-Dominance Criteria (ND) [32]. The score-matrix is considered as a fuzzy preference relation. Then class with the highest non-dominance degree is the predicted one: Class =arg max i=1,...,m 1−max j=1,...,m r0 ji(11) where R0is the strict score-matrix. •Learning valued preference for classification (LVPC) [33], [34]. This aggregation strategy, as in ND, considers the score-matrix as a fuzzy preference relation. In this manner, the original relation is decomposed into three new relations with different meanings: the strict preference, the conflict, and the ignorance. In order to obtain the output class, a decision rule based on voting strategy is proposed: Class =arg max i=1,...,m X 1≤j6=i≤m Pij +1 2Cij +Ni Ni+Nj Iij (12) where Niis the number of examples from the class iin the training data, Cij is the degree of conflict (the degree to which both classes are supported), Iij is the degree of ignorance (the degree to which none of the classes are supported), and Pij and Pji are the strict preference for iand j, respectively. These variables are computed as follows: Cij = min {rij, rji} Pij =rij −Cij Pji =rji −Cij Iij = 1 −max {rij, rji} Owing to the way in which the multi-class problem is divided in OVO scheme, there is an issue inherent to this decomposition method: the non-competent classifiers [19]. The learning process of each base classifier is performed using only the examples belonging to the two classes that this classifiers will classify and consequently, it ignores the examples belonging to other classes. Therefore, the remainder classes are unknown for these classifiers and their outputs are irrelevant when classifying examples of those classes. However, these outputs are aggregated in the same way as the relevant ones, possibly misleading the correct labeling of the example. Although this is an interesting line, it is out of the scope of this paper and we leave it as a future research line. 2) One-Versus-All (OVA): OVA decomposition divides a problem of mclasses into mbinary problems, which are addressed by independent binary classifiers. Each base classifier distinguishes one of the classes from all other classes. The learning process of these classifiers is performed using the whole training data, considering the examples from the single class as positives and the rest of examples as negatives. When classifying a new example, each base classifier will return a confidence degree ri∈[0,1] in favour of the class Ci, which will be stored in a score-vector R: R= (r1, . . . , ri, . . . , rm)(13) As in OVO, we need to normalize the score-vector such that the range of the confidences provided by all classifiers is the same. In order to do so, we also need the score-vector in which the confidences obtained by each classifier for the negative class are stored ( ˆ R). With both vectors, the normalization of the score-vector Ris performed as follows. ri=ri ri+ ˆri (14) Finally, the most commonly used aggregation in OVA consider the usage of the maximum value in the score-vector, and thus the class with the highest confidence will be predicted. C. Related works Decomposition strategies can be included in the broader categories of ensembles and multiple classifier systems (MCSs) [35], [36]. These types of systems aim to improve the classification performance by the combination of several classifiers. In fact, ensembles and MCSs are usually referred to those methods where the base classifiers are able to predict any of the classes of the problem; however, decomposition techniques are also formed of sets of classifiers, but there is a major difference: each base classifier is not able to predict all the classes in the problem (only two of them or two combinations of different classes are predicted). Such an important differ- 5 ence has produced rather different approaches for each type of method. Traditional ensembles aim to produce diverse base classifiers [37], whose differences in their predictions allow them to increase the performance, since they complement each other. Widely used ensemble methods are Bagging [38] and Boosting [39], [40]. These type of methods have been also considered in the fuzzy community, where different approaches have been proposed using fuzzy systems as base classifiers [41]– [44]. In [41], the authors combined the FRBCSs obtained in the pareto front of a multi-objective optimization genetic algorithm. An extension of the classical Random Forests (a variant of bagging) using fuzzy decision trees was presented in [42], [45]. Boosting in the fuzzy context was applied in [46]. The authors of [43], [44], [47] developed a methodology to construct FURIA-based multi-classifiers in a series of works, including all the different phases of an ensemble, from its construction (also bagging-based) to the combination procedure presented in the latter work. All these type of models are usually no longer interpretable, and hence the fuzzy classifiers are used instead of other weak classifiers such as the commonly used decision trees to take advantage of the fuzzy decision boundaries to reach highly accurate models, which may need the usage of thousands of rules [44], even though some authors focused on reducing this number [48]. Moreover, FURIA [24] has been one of the most extended base classifier in this framework, which by itself is not as interpretable as classical FRBCSs [1], since it make use of hyper-rectangles adjusted for each rule instead of using the same linguistic labels in each rule. For this reason, in this paper we only deal with decomposition-based ensembles, which may maintain part of the interpretability of the original models. Otherwise, decomposition strategies have also attracted attention as a way of improving classification in multi-class problems with FRBCSs [26], [32], [34], [49]. In this framework, different base classifiers have been used (Fuzzy Ripper [34], FH-GBML [50] or SLAVE [51]), as well as different combination methods have been proposed such as the NonDominance criterion (ND) in [32] or the Learning Valued Preference for Classification (LVPC) [34], [49] already described in Section II-B. In these papers, the authors considered the score-matrix as a preference relation from which the best alternative should be predicted. In order to do so, the conflict and ignorance were modeled in [49] and thereafter applied in the Fuzzy Ripper algorithm presented in [34]. Similarly but with a different approach to output the class from the score-matrix, the authors in [32] proposed the usage of the ND criterion, showing good results with FH-GBML and SLAVE classifiers. In addition, in [26] the authors presented the TopDown induction of Fuzzy Pattern Trees (PTTD), which made use of OVA approach. However, recent developments with fuzzy classifiers are not only related to ensemble strategies. Taking into account the good properties of FRBCSs, several approaches [24], [25], [52]–[54] have been proposed aimed at improving the tradeoff between accuracy and interpretability [55]. FURIA [24] extended the RIPPER algorithm using fuzzy rules and it provides accurate results. In [52], authors combined a feature selection process using the so-called modulator functions and a fuzzy rule extraction mechanism based on fuzzy clustering. Castro et. al [53] defined a fuzzy classifier using general fuzzy rules and a new mechanism aimed at solving the conflicts among them. In [54], authors defined a new approach to design fuzzy classifiers using k-means clustering and a memetic algorithm to find the optimal values of fuzzy rules and membership functions. In order to improve the interpretability of TSK fuzzy classifiers, the usage of a minimax probability was proposed in [56]. Finally, Sanz et. al [25] provided a framework to improve the performance of FRBCSs using interval-valued fuzzy sets. On account of the different fuzzy methodologies described above, we have considered to include those following a similar philosophy to our proposal in the experimental study. More specifically, we have considered both aggregations defined in the fuzzy context (ND and LVPC) [32], [34], [49], the PTTD [26] method as an OVA based fuzzy system as well as FURIA [24] and IVTURS [25] as state-of-the-art fuzzy classifiers. Finally, we should mention that none of the ensemble/multiclassifier approaches previously enumerated has addressed the problem affecting the inference that we aim to overcome in this paper. III. INTRODUCING N-DIMENSIONAL OVERLAP FUNCTIONS TO ADAPT FARC-HD BEHAVIOUR IN OVA AND OVO In this paper, we propose to combine FARC-HD with OVA and OVO decomposition strategies in order to improve the performance of FARC-HD in multi-class classification problems. However, the confidences provided by FARC-HD are not adequate for them due to the use of the product to compute the association degree, as we will show in the experimental analysis. Thus, the inference process needs to be adapted for the sake of a better synergy between FARC-HD and decomposition schemes. In the remainder of this section, we first describe the way in which we introduce FARC-HD in OVO and OVA models, as well as the problems that we have to address when carrying out this combination (Section III-A). Next, we recall the concept of two-dimensional overlap functions and we present the new definition of n-dimensional overlap functions, as well as their construction method (Section III-B). Then, we describe the modification of the inference of FARC-HD using n-dimensional overlap functions aiming at improving the synergy between FARC-HD and decomposition strategies (Section III-C). Finally, we present a new aggregation strategy for the OVO model named WinWV that solves the problems of the WV with the confidences of FARC-HD (Section III-D). A. Using FARC-HD as base classifier in the OVA and OVO strategies In order to use OVO and OVA strategies with FARC-HD, we need to fill the score-matrix of OVO (Eq. (7) and the scorevector of OVA (Eq. (13)) with the confidences provided by FARC-HD for each class. More specifically, we consider as confidences the confidence degree for each class computed using Eq. (5). Both the matching and the association degrees 6 of the example with the fuzzy rules are computed by Eq. (3) and (4), respectively, using the product t-norm. When low values are aggregated using the product t-norm, the range in which the result can vary is small (the lower the input values are, the smaller the range becomes), which may happen when computing the matching degree of several fuzzy rules. This effect is further accentuated as the number of antecedents of the rules increases, which implies that the association degrees of those rules with more antecedents will be smaller and will have a lower variation. This behaviour implies that the confidences stored in the score-matrix and score-vector will have low variations as well, which is not desirable for the subsequent aggregation performed in OVO and OVA schemes. Consequently, it seems suitable to modify the inference process in such a way that the aggregation of the values involved in the computation of the association degrees is made using functions whose results are in a wider range, maintaining more information for the aggregation process (Section III-C). The previous problem does not affect the behaviour of the original FARC-HD, since the confidences obtained after the inference process are not used beyond classification and hence, this variation does not affect the final result given by the algorithm. However, in decomposition strategies, the confidences provided by FARC-HD are used in the aggregation phase, and hence the predictions are used beyond the decision of the class prediction of the base classifiers. Thus, a low variation in the confidences might have a negative effect in OVO and OVA models, which is especially reflected in the unexpected behaviour of robust aggregation strategies used in OVO, such as the WV, as we will show in the experimental study. For this reason, a new aggregation strategy that solves the problems of the WV is needed (see Section III-D). B. n-dimensional overlap functions The concept of overlap function [20] was introduced in image processing in order to classify those pixels that it was not clear whether they belonged to the object or to the background. This concept has been applied in many image processing problems [57]–[59] and used in [60] to model the indifference in preference relations. However, the application range of these functions has turned out to be much wider, since they allow one to recover many of the characteristics of the t-norms without imposing the associativity. Precisely because the associativity is not demanded, the extension of the concept of overlap function to dimensions higher than two is not direct. Moreover, this extension is necessary in order to use overlap functions in problems in which the associativity is not necessary or even natural and in which t-norms have been used. In this paper, we propose a definition of overlap function in any finite dimension, which particularly allows one to recover the two-dimensional case. Additionally, we present the construction method of overlap functions using rational expressions. We first recall the following definition of two-dimensional overlap functions: Definition 1: [20] A function O: [0,1] ×[0,1] →[0,1] is an overlap function if satisfies the following conditions : 1) O(x, y) = O(y, x)for all x, y ∈[0,1]. 2) O(x, y)=0if and only if x·y= 0. 3) O(x, y)=1if and only if x·y= 1. 4) is increasing. 5) is continuous. Following this concept we define the extension of the previous two-dimensional overlap functions to ndimensions: Definition 2: An n-dimensional function On: [0,1]n→ [0,1] with n≥2is a n-dimensional overlap function if the following properties hold: 1) Onis symmetric. 2) On(x1, . . . , xn)=0if and only if n Q i=1 xi= 0. 3) On(x1, . . . , xn)=1if and only if n Q i=1 xi= 1. 4) Onis increasing. 5) Onis continuous in each of the variables. Example 1: The following functions are examples of ndimensional overlap functions: 1) The minimum is a n-dimensional overlap function which is also a t-norm. On(x1...,xn) = min(x1, . . . , xn)(15) 2) Take p > 0. Then the function On(x1, . . . , xn) = n Y i=1 xi!p (16) is a n-dimensional overlap function. Furthermore, Onis associative if and only if p= 1. a) If p= 1 we recover the product, which is a t-norm as well. On(x1, . . . , xn) = n Y i=1 xi(17) b) If p=1 nwe have the geometric mean: On(x1, x2, . . . , xn) = n v u u t n Y i=1 xi(18) 3) The harmonic mean is a n-dimensional function: On(x1, x2, . . . , xn) =       n 1 x1+. . . +1 xn if xi6= 0,for all i= 1, . . . , n 0otherwise. (19) 4) The function On(x1, . . . , xn) = sin π 2 n Y i=1 xi!α!(20) where α≤1 2n, is another example of n-dimensional overlap function. As we have shown, both the product, which is used in the original FARC-HD, and the minimum, which is another t- 7 norm that is commonly used in FRBCs, are examples of ndimensional overlap functions. Finally, we present a construction method for n-dimensional overlap functions using rational expressions. Theorem 1: The mapping On: [0,1]n→[0,1] is a n-dimensional overlap function if and only if there exist f, g : [0,1]n→[0,1] with On(x1, . . . , xn) = f(x1, . . . , xn) f(x1, . . . , xn) + g(x1, . . . , xn) where 1) fand gare symmetric. 2) fis non-decreasing and gis non-increasing. 3) f(x1, . . . , xn)=0if and only if n Q i=1 xi= 0. 4) g(x1, . . . , xn)=0if and only if n Q i=1 xi= 1. 5) fand gare continuous. Proof. To see the necessity, assume that Onis a ndimensional overlap function. We can define f(x1, . . . , xn) = On(x1, . . . , xn)and g(x1, . . . , xn)=1−f(x1, . . . , xn). Hence, the properties (1)-(5) of the Theorem are direct and also: f(x1, . . . , xn) f(x1, . . . , xn) + g(x1, . . . , xn)=On(x1, . . . , xn) 1 Let us take a look at what happens with the sufficiency. We must see that the function defined in Eq. (1) is in fact a n-dimensional overlap function. The continuity, the symmetry and the monotony are evident. Also: On(x1, . . . , xn) = 0 iff f(x1, . . . , xn) = 0 iff n Y i=1 xi= 0 and On(x1, . . . , xn) = 1 iff f(x1, . . . , xn) =f(x1...,xn) + g(x1, . . . , xn) that is, if and only if g(x1, . . . , xn) = 0 and if and only if n Q i=1 xi= 1. Example 2: The function On(x1, . . . , xn) = n Y i=1 xi! 1 n n Y i=1 xi! 1 n +max 1≤i≤n(1 −xi) is an example of n-dimensional overlap function. C. Modification of the inference process using n-dimesional overlap functions Once the n-dimensional overlap functions and the construction methods have been presented, we show the proposed modification of the inference process of FARC-HD in order to improve the aggregation in OVO and OVA strategies. More specifically, we propose to compute the matching degree and the association degree of the example with the fuzzy rules using the previously defined n-dimensional overlap functions. To do so, we replace the t-norm in the matching degree computation (Eq. (3)) by an overlap function (Eq. (21)): µAj(xp) = On(µAj1(xp1), . . . , µAjnj(xpnj)) (21) We must stress that the matching degree is employed to compute both the support and the confidence used in the first step of the learning algorithm shown in Section II-A. Thus, this modification also affects the learning process of the algorithm as well. Similarly, we substitute the product in the association degree computation (Eq. (4)) by an overlap function, according to Eq. (22). bj(xp) = O(µAj(xp), RWj) =O(On(µAj1(xp1), . . . , µAjnj(xpnj)), RWj)(22) The reason for computing the association degree using an overlap function instead of the product is the same as in the case of the matching degree, that is, the low variation of the association degrees obtained when multiplying the matching degree and the rule weight. In this paper, we have considered five different overlap functions to observe their effect in the rule base and their influence on the accuracy of the model (for the sake of brevity, in the experiments we take the overlap Oin Eq. (22) the same as On). Each overlap function returns lower or higher values than the rest for the same input tuple. According to the values returned by the overlap functions, we can establish an order among them. Thus, we will consider that an overlap function is greater than other one if the values returned by the first function are higher than those returned by the second one for the same arguments. A short description of each function is shown below, sorted in ascending order by the returned value: •Product (PROD): The returned value is the product of input values (Eq. (17)). Indeed, this is the case of the original FARC-HD, and hence we are able to recover the original method using the proposed extension of overlap functions. •Minimum (MIN): Returns the minimum of input values (Eq. (15)). This is a t-norm as well, but unlike the product, the returned value does not decrease when the number of arguments increases. The minimum is commonly used in FRBCs. •Harmonic Mean (HM): The returned value is the harmonic mean of input values if all of them are different than zero and 0 otherwise (Eq. (19)). •Geometric Mean (GM): Returns the geometric mean of input values (Eq. (18)). •Sine (SIN): This an example of an overlap function that returns higher values than means (Eq. (20)). The use of this type of functions is interesting in order to check what happens in these cases. In the experiments carried out in Section V, we take α=1 2n. Among the considered overlap functions, the first one is the product (used in the original FARC-HD). The product is a t-norm that returns values with a lower variation than the 8 other functions and whose returned value decreases as the number of arguments increases. Next, we have the minimum, which is a t-norm as well, but whose returned value does not depend on the number of arguments. Then, we consider the harmonic and the geometric means as representatives of means that return higher values than t-norms [61]. Finally, we have considered a function (SIN) that returns higher values than means. This variety of overlap functions allows us to have a general overview in the experiments (Section V). According to Eq. (19) and (18), both the harmonic and the geometric means return 0 when one of the arguments is 0. This property is satisfied by t-norms as well, and it is very important to conserve the necessary discrimination capability of FARC-HD. In the experimental study, we will show that another desirable property that the overlaps should satisfy in order to work well in our framework is the idempotence, which is satisfied if: On(x, . . . , x) = x(23) Among the previous overlaps, the minimum, the harmonic mean and the geometric mean satisfy this property. In Fig. (2a) and (2b) we can graphically observe the previously mentioned differences in the values returned by each overlap function (we depict overlaps with n= 2 to ease the visualization of their behavior). Fig. (2a) depicts the behavior of the overlap functions when aggregating a value with the value 1, whereas Fig. (2b) shows the returned values when aggregating a value with itself. Looking at Fig. (2a) and (2b), we can see that the proposed n-dimensional overlap functions provide a higher variation than the product when the input arguments are small. However, both figures show that there is a huge difference between the SIN and the rest of the overlap functions. In fact, in Fig. (2b) we can observe that the returned value is greater than the input arguments when aggregating a value with itself, which might not be a desirable behavior in this framework, as it may produce a loss of the discrimination capabilities of FARC-HD. D. Adapting the Weighted Voting to FARC-HD confidence estimation: WinWV In addition to the adaptation of the inference process of the base classifiers to OVO and OVA using overlap functions, we propose a new aggregation method for the OVO strategy named WinWV, which is a modification of the WV that does not achieve the expected results. As we described in Section II-B, each base classifier provides a pair of confidence degrees rij, rji ∈[0,1] in favor of classes Ci, Cj, so that rij is the confidence predicting the class Ciand rji is the confidence predicting the class Cj. Although we have improved the confidences provided by FARC-HD making them more suitable for the OVO strategy, we will show that the WV is still not working as expected. From our point of view, the reason is that the confidence estimation of the non-predicted class distorts the aggregation phase in OVO, as we will show in Section V. Likewise, the LVPC strategy does not work properly with this type of confidences, as it can be observed in [32]. In this case, the confidence for the non-predicted class does not allow one to model the conflict and ignorance degrees properly. Notice that if these terms were not considered the original WV would be recovered. For this reason, we focus on solving the problems of the WV with the confidence estimation of the non-predicted class. To do so, we propose to consider only the confidence of the predicted class, whereas that of the non-predicted class is not taken into account. Therefore, the WinWV aggregation strategy works as follows: Class =arg max i=1,...,m X 1≤j6=i≤m sij (24) where sij is rij if rij > rji and 0 otherwise. Notice that OVA and other combination strategies in OVO such as VOTE and ND (Section II-B) managing the confidence for the nonpredicted class differently need not be modified in order to achieve competitive results. IV. EXPERIMENTAL FRAMEWORK In this section, we present the set-up of the experimental framework used to develop the experiments carried out in Section V. First, we describe the datasets selected for the experimental study (Section IV-A). Next, we show the parameter setup considered for each method (Section IV-B). Finally, we introduce the statistical tests that are necessary to assess whether significant differences exist among the results obtained (Section IV-C). A. Datasets In order to analyze the performance of our proposal, we have considered twenty datasets selected from the KEEL dataset repository [21]. Table I summarizes the features of the selected datasets, showing for each dataset the number of examples (#Ex.), number of attributes (#Atts.), number of numerical (#Num.) and nominal (#Nom.) attributes, and the number of classes (#Class.). To carry out the different experiments we consider a 5-fold stratified cross-validation model, i.e., we randomly split the dataset into five partitions of data, each one containing 20% of the patterns, and we employed a combination of four of them (80%) to train the system and the remaining one to test it. We use three different seeds for the execution of the methods in each partition. In this manner, the result for each dataset is obtained by computing the average of the five partitions using the three seeds in each one. Instead of the commonly used cross-validation and in order to correct the dataset shift, that is, when the training data and the test data do not follow the same distribution [62], [63], we will use a recently published partitioning procedure called Distribution Optimally Balanced Cross Validation [64]. B. State-of-the-art fuzzy classification methods used for comparison In this section we briefly describe the different methods used throughout the experiments and the configuration that we have considered for each one. We have selected three 9 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 O O O O O O O O O O O x On(1, x) Product OMinimum Harmonic Mean Geometric Mean Sine (a) Considering (1, x) as input 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 O O O O O O O O O O O x On(x,x) Product OMinimum Harmonic Mean Geometric Mean Sine (b) Considering (x,x) as input Fig. 2. Values returned by the different overlap functions. TABLE I SUMMARY OF THE FEATURES OF THE DATASETS USED IN THE EXPERIMENTAL STUDY. Id. Dataset #Ex. #Atts. #Num. #Nom. #Class. aut autos 159 25 15 10 6 bal balance 625 4 4 0 3 cle cleveland 297 13 13 0 5 con contraceptive 1473 9 6 3 3 eco ecoli 336 7 7 0 8 gla glass 214 9 9 0 7 hay hayes-roth 132 4 4 0 3 iri iris 150 4 4 0 3 new newthyroid 215 5 5 0 3 pag pageblocks 548 10 10 0 5 pen penbased 1100 16 16 0 10 sat satimage 643 36 36 0 7 seg segment 2310 19 19 0 7 shu shuttle 2175 9 9 0 5 tae tae 151 5 3 2 3 thy thyroid 720 21 21 0 3 veh vehicle 846 18 18 0 4 vow vowel 990 13 13 0 11 win wine 178 13 13 0 3 yea yeast 1484 8 8 0 10 recognized state-of-the-art fuzzy classifiers to compare against our proposal, in addition to the comparison against the original FARC-HD algorithm. •FURIA [24]: This algorithm modifies and extends the RIPPER rule induction algorithm [65]. In particular, FURIA learns fuzzy rules of the form given in Eq. (1) instead of conventional rules, using fuzzy sets with trapezoidal membership functions. Additionally, the model built by FURIA learns unordered rule sets instead of rule lists. The learning process is divided in two stages: 1) Learn a rule set for each class using OVA decomposition. To do so, a modified and extended version of RIPPER is applied, which can be divided into the building and the optimization phase. 2) Extract the fuzzy rules by fuzzifying RIPPER’s rules using a greedy algorithm. When classifying a new example, the class predicted by FURIA is the one with maximal support. If the example is not covered by any rule, a rule generalization (stretching) is carried out and all rules are replaced by their minimal generalizations, which is obtained by deleting all antecedents that are not satisfied by the query. In the case of a tie, a decision in favor of the class with highest frequency is made. •IVTURSFARC-HD [25]: This method uses FARC-HD to accomplish the fuzzy rule learning process and then, it substitutes the original fuzzy sets by interval-valued (IV) fuzzy sets and it modifies the inference process using an IV fuzzy reasoning method. This inference process uses IV restricted equivalence functions to increase the relevance of the rules in which the equivalence of the interval membership degrees of the patterns and the ideal membership degrees is greater. In addition, it combines a tuning of the parameters used in the IV fuzzy reasoning method and rule selection, in order to both decrease the complexity and increase the performance of the system. •PTTD [26]: This method constructs a fuzzy pattern tree for each class (OVA decomposition) whose inner nodes are marked with generalized fuzzy logical operators and whose leaf nodes are associated with linguistic terms on input attributes. The learning algorithm used by PTTD builds the pattern tree in a top-down manner. The configuration of the previous methods and that of our proposal is shown in Table II. C. Performance measure and statistical tests In order to test the performance of the different methods, we have used the most common metric, that is, the accuracy rate. This metric is defined as percentage of correctly classified examples related to the total number of examples. However, accuracy rate may not properly reflect the behaviour of different algorithms in multi-class problems as they do not take into account the classes of the examples in its computation. Therefore, the usage of additional metrics (not opposite, yet 16 [15] J. A. S´ aez, M. Galar, J. Luengo, and F. Herrera, ”Analyzing the presence of noise in multi-class problems: alleviating its influence with the onevs-one decomposition,” Knowledge and Information Systems, vol. 38, no. 1, pp. 179-206, 2014. [16] M. Galar, A. Fern´ andez, E. Barrenechea, and F. Herrera, ”Empowering difficult classes with a similarity-based aggregation in multi-class classification problems,” Information Sciences, vol. 264, pp. 135-157, 2014. [17] O. Cord´ on, M. J. del Jesus, and F. Herrera, ”A proposal on reasoning methods in fuzzy rule-based classification systems,” International Journal of Approximate Reasoning, vol. 20, no. 1, pp. 21-45, 1999. [18] E. H¨ ullermeier and S. Vanderlooy, ”Combining predictions in pairwise classification: An optimal adaptive voting strategy and its relation to weighted voting,” Pattern Recognition, vol. 43, no. 1, pp. 128-142, 2010. [19] M. Galar, A. Fern´ andez, E. Barrenechea, H. Bustince, and F. Herrera, ”Dynamic classifier selection for one-vs-one strategy: Avoiding noncompetent classifiers,” Pattern Recognition, vol. 46, no. 12, pp. 34123424, 2013. [20] H. Bustince, J. Fernandez, R. Mesiar, J. Montero, and R. Orduna, ”Overlap functions,” Nonlinear Analysis: Theory, Methods & Applications, vol. 72, no. 3-4, pp. 1488-1499, 2010. [21] J. Alcal´ a-Fdez, A. Fernandez, J. Luengo, J. Derrac, S. Garc´ ıa, L. S´ anchez, and F. Herrera, ”KEEL data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework,” Journal of Multiple-Valued Logic and Soft Computing, vol. 17:23, pp. 255-287, 2011. [22] S. Garc´ ıa, A. Fern´ andez, J. Luengo, and F. Herrera, ”A study of statistical techniques and performance measures for genetics-based machine learning: Accuracy and interpretability,” Soft Comput., vol. 13, no. 10, p. 959-977, 2009. [23] S. Garc´ ıa, A. Fern´ andez, J. Luengo, and F. Herrera, ”Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power,” Information Sciences, vol. 180, no. 10, pp. 2044-2064, 2010. [24] J. H¨ uhn and E. H¨ ullermeier, ”FURIA: an algorithm for unordered fuzzy rule induction,” Data Mining and Knowledge Discovery, vol. 19, no. 3, pp. 293-319, 2009. [25] J. Sanz, A. Fern´ andez, H. Bustince, and F. Herrera, ”IVTURS: A linguistic fuzzy rule-based classification system based on a new intervalvalued fuzzy reasoning method with tuning and rule selection,” IEEE Transactions on Fuzzy Systems, vol. 21, no. 3, pp. 399-411, 2013. [26] R. Senge and E. H¨ ullermeier, ”Top-down induction of fuzzy pattern trees,” IEEE Transactions on Fuzzy Systems, vol. 19, pp. 241 - 252, 2011. [27] H. Ishibuchi and T. Yamamoto, ”Rule weight specification in fuzzy rule-based classification systems,” IEEE Transactions on Fuzzy Systems, vol. 13, no. 4, pp. 428-435, 2005. [28] R. Agrawal and R. Srikant, ”Fast algorithms for mining association rules,” in International Conference on Very Large Data Bases, Santiago de Chile, Chile, September 1994, p. 487-499. [29] B. Kavsek and N. Lavrac, ”Apriori-sd: Adapting association rule learning to subgroup discovery,” Applied Artificial Intelligence, vol. 20, no. 7, p. 543-583, 2006. [30] R. Alcal´ a, J. Alcal´ a-Fdez, and F. Herrera, ”A proposal for the genetic lateral tuning of linguistic fuzzy systems and its interaction with rule selection,” IEEE Transactions on Fuzzy Systems, vol. 15, no. 4, pp. 616635, 2007. [31] J. Friedman, ”Another approach to polychotomous classification,” Department of Statistics, Stanford University, Tech. Rep., 1996. [Online]. Available: http://www-stat.stanford.edu/∼jhf/ftp/poly.ps.Z [32] A. Fern´ andez, M. Calder´ on, E. Barrenechea, H. Bustince, and F. Herrera, ”Solving mult-class problems with linguistic fuzzy rule based classification systems based on pairwise learning and preference relations,” Fuzzy Sets and Systems, vol. 161, no. 23, pp. 3064-3080, 2010. [33] E. H¨ ullermeier and K. Brinker, ”Learning valued preference structures for solving classification problems,” Fuzzy Sets and Systems, vol. 159, no. 18, pp. 2337-2352, 2008. [34] J. C. Huhn and E. Hullermeier, ”FR3: A fuzzy rule learner for inducing reliable classifiers,” IEEE Transactions on Fuzzy Systems, vol. 17, no. 1, pp. 138-149, 2009. [35] L. I. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms. Wiley-Interscience, 2004. [36] M. Wo´ zniak, M. Gra˜ na, and E. Corchado, ”A survey of multiple classifier systems as hybrid systems,” Information Fusion, vol. 16, pp. 3-17, 2014. [37] L. I. Kuncheva, ”Diversity in multiple classifier systems,” Information Fusion, vol. 6, no. 1, pp. 3-4, 2005. [38] L. Breiman, ”Bagging predictors,” Machine Learning, vol. 24, pp. 123140, 1996. [39] R. E. Schapire, ”The strength of weak learnability,” Machine Learning, vol. 5, pp. 197-227, 1990. [40] Y. Freund and R. E. Schapire, ”A decision-theoretic generalization of on-line learning and an application to boosting,” Journal of Computer and System Sciences, vol. 55, no. 1, pp. 119-139, 1997. [41] H. Ishibuchi and Y. Nojima, ”Fuzzy ensemble design through multiobjective fuzzy rule selection,” in Multi-Objective Machine Learning, ser. Studies in Computational Intelligence, Y. Jin, Ed. Springer Berlin Heidelberg, 2006, vol. 16, pp. 507-530. [42] P. Bonissone, J. M. Cadenas, M. C. Garrido, and R. A. D´ ıaz-Valladares, ”A fuzzy random forest,” International Journal of Approximate Reasoning, vol. 51, no. 7, pp. 729-747, 2010. [43] K. Trawi´ nski, O. Cord´ on, and A. Quirin, ”On designing fuzzy rulebased multiclassification systems by combining FURIA with bagging and feature selection,” International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, vol. 19, no. 4, pp. 589-633, 2011. [44] K. Trawinski, O. Cordon, L. Sanchez, and A. Quirin, ”A genetic fuzzy linguistic combination method for fuzzy rule-based multiclassifiers,” IEEE Transactions on Fuzzy Systems, vol. 21, no. 5, pp. 950-965, 2013. [45] J. M. Cadenas, M. C. Garrido, R. Mart´ ınez, and P. P. Bonissone, ”Extending information processing in a fuzzy random forest ensemble,” Soft Computing, vol. 16, no. 5, pp. 845-861, 2012. [46] A. M. Palacios, L. S´ anchez, and I. Couso, ”Boosting of fuzzy rules with low quality data,” Journal of Multiple-Valued Logic and Soft Computing, vol. 19, no. 5-6, pp. 591-619, 2012. [47] K. Trawi´ nski, O. Cord´ on, and A. Quirin, ”A study on the use of multiobjective genetic algorithms for classifier selection in FURIAbased fuzzy multiclassifiers,” International Journal of Computational Intelligence Systems, vol. 5, no. 2, pp. 231-253, 2012. [48] B. Soua, A. Borgi, and M. Tagina, ”An ensemble method for fuzzy rule-based classification systems,” Knowledge and Information Systems, vol. 36, no. 2, pp. 385-410, 2013. [49] E. H¨ ullermeier and K. Brinker, ”Learning valued preference structures for solving classification problems,” Fuzzy Sets and Systems, vol. 159, no. 18, pp. 2337-2352, 2008. [50] H. Ishibuchi, T. Yamamoto, and T. Nakashima, ”Hybridization of fuzzy GBML approaches for pattern classification problems,” IEEE Transactions on System, Man and Cybernetics B, vol. 35, no. 2, pp. 359-365, 2005. [51] A. Gonz´ alez and R. Perez, ”SLAVE: a genetic learning system based on an iterative approach,” Fuzzy Systems, IEEE Transactions on, vol. 7, no. 2, pp. 176-191, 1999. [52] Y.-C. Chen, N. R. Pal, and I.-F. Chung, ”An integrated mechanism for feature selection and fuzzy rule extraction for classification,” IEEE Transaction on Fuzzy Systems, vol. 20, no. 4, pp. 683-698, 2012. [53] J. Castro-Schez, M. J.M., R. Miguel, and X. Luo, ”Knowledge acquisition based on learning of maximal structure fuzzy rules,” KnowledgeBased Systems, vol. 44, pp. 112-120, 2013. [54] A. Acilar and A. Arslan, ”A novel approach for designing adaptive fuzzy classifiers based on the combination of an artificial immune network and a memetic algorithm,” Information Sciences, vol. 264, pp. 158-181, 2014. [55] M. J. Gacto, R. Alcal´ a, and F. Herrera, ”Interpretability of linguistic fuzzy rule-based systems: An overview of interpretability measures,” Information Sciences, vol. 181, no. 20, pp. 4340-4360, 2011. [56] Z. H. Deng, L. B. Cao, Y. Z. Jiang, and S. T. Wang, ”Minimax probability TSK fuzzy system classifier: A more transparent and highly interpretable classification model,” IEEE Transactions on Fuzzy Systems, 2014. [57] D. Paternain, M. Pagola, J. Fernandez, R. Mesiar, G. Beliakov, and H. Bustince, ”Brain MRI thresholding using incomparability and overlap functions,” in Intelligent Systems Design and Applications (ISDA), 2011 11th International Conference on, 2011, pp. 808-812. [58] A. Jurio, H. Bustince, M. Pagola, A. Pradera, and R. R. Yager, ”Some properties of overlap and grouping functions and their application to image thresholding,” Fuzzy Sets and Systems, vol. 229, pp. 69-90, 2013. [59] D. Paternain, J. Fernandez, H. Bustince, R. Mesiar, and G. Beliakov, ”Construction of image reduction operators using averaging aggregation functions,” Fuzzy Sets and Systems, vol. in press, 2014. [60] H. Bustince, M. Pagola, R. Mesiar, E. Hullermeier, and F. Herrera, ”Grouping, overlap, and generalized bientropic functions for fuzzy modeling of pairwise comparisons,” IEEE Transactions on Fuzzy Systems, vol. 20, no. 3, pp. 405-415, 2012. 17 [61] G. Beliakov, A. Pradera, and T. Calvo, Aggregation Functions: A Guide for Practitioners, ser. Studies in Fuzziness and Soft Computing. Springer Berlin Heidelberg, 2007, vol. 221. [62] J. Moreno-Torres, T. Raeder, R. Alaiz-Rodr´ ıguez, N. Chawla, and F. Herrera, ”A unifying view on dataset shift in classification,” Pattern Recognition, vol. 45, no. 1, pp. 521-530, 2012. [63] V. L´ opez, A. Fern´ andez, and F. Herrera, ”On the importance of the validation technique for classification with imbalanced datasets: Addressing covariate shift when data is skewed,” Information Sciences, vol. 257, pp. 1-13, 2014. [64] J. Moreno-Torres, J. Saez, and F. Herrera, ”Study on the impact of partition-induced dataset shift on k-fold cross-validation,” IEEE Transactions on Neural Networks and Learning Systems, vol. 23, no. 8, pp. 1304-1312, 2012. [65] W. W. Cohen, ”Fast effective rule induction,” presented at the 12th Int. Conf. Mach. Learn., Lake Tahoe, CA, USA, 1995. [66] J. Cohen, ”A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37-46, 1960. [67] F. Wilcoxon, ”Individual comparisons by ranking methods,” Biometrics, vol. 1, no. 6, pp. 80-83, 1945. [68] J. L. Hodges and E. L. Lehmann, ”Ranks methods for combination of independent experiments in analysis of variance,” Ann. Math. Statist., vol. 33, pp. 482-497, 1962. [69] S. Holm, ”A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, pp. 65-70, 1979. Mikel Elkano received the M.Sc. degree in Computer Science from the Public University of Navarre, Spain, in 2014. He was awarded with the best Master’s degree final project in the Artificial Intelligence Summer School 2014, organized by the Spanish Association for Artificial Intelligence. He is currently a Ph.D. student at the Public University of Navarre. His research interests include machine learning, fuzzy systems, ensemble learning, bioinformatics and Big Data. Mikel Galar received the M.Sc. and Ph.D. degrees in Computer Science in 2009 and 2012, both from the Public University of Navarre, Pamplona, Spain. He is currently an assistant professor in the Department of Automatics and Computation at the Public University of Navarre. His research interests are data-mining, classification, multi-classification, ensemble learning, evolutionary algorithms, fuzzy systems and fingerprint recognition. Jos´ e Antonio Sanz received the M.Sc. degree in computer sciences and the Ph.D. degree, both form the Public University of Navarre in 2008 and 2011 respectively. He is currently an Associate Lecturer with the Department of Automatics and Computation, Public University of Navarre. He is the author of 15 published original articles in international journals and is involved in teaching artificial intelligence for students of computer sciences. He received the best paper award in the FLINS 2012 international conference and the Pepe Mill´ a award in 2014. His research interests include fuzzy techniques for classification problems, interval-valued fuzzy sets, genetic fuzzy systems, data mining and medical applications of soft computing techniques. Dr. Sanz is member of the European Society for Fuzzy Logic and Technology. Alberto Fern´ andez received the M.Sc. and Ph.D. degrees in computer science from the University of Granada, Granada, Spain, in 2005 and 2010, respectively. He is currently an Assistant Professor with the Department of Computer Science, University of Ja´ en, Spain. His research interests include data mining, classification in imbalanced domains, fuzzy rule learning, evolutionary algorithms, multiclassification problems with ensembles and decomposition techniques, and Data Science in Big Data applications. Dr. Fern´ andez received the Lofti A. Zadeh Prize of the International Fuzzy Systems Association for the Best paper in 2009-2010 for his work of hierarchical fuzzy rule based classification system with genetic rule selection for imbalanced data-sets, and the 2012 University of Granada Prize for Scientific Excellence Works in the field of Engineering for the paper ”Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining”. Edurne Barrenechea is an Assistant Lecturer at the Department of Automatics and Computation, Public University of Navarre. She received an M.Sc. in Computer Science at the Pais Vasco University in 1990. She worked in a private company (Bombas Itur) as analyst programmer from 1990 to 2001, and then she joined the Public University of Navarre as Associate Lecturer. She obtained the Ph.D. in Computer Science in 2005 on the topic intervalvalued fuzzy sets applied to image processing. Her publications comprise more than 30 papers in international journals and about 15 book chapters. Her research interests are fuzzy techniques for image processing, fuzzy sets theory, interval type-2 fuzzy sets theory and applications, decision making, and medical and industrial applications of soft computing techniques. She is member of the board of the European Society for Fuzzy Logic and Technology (EUSFLAT). Francisco Herrera received his M.Sc. in Mathematics in 1988 and Ph.D. in Mathematics in 1991, both from the University of Granada, Spain. He is currently a Professor in the Department of Computer Science and Artificial Intelligence at the University of Granada. He has been the supervisor of 35 Ph.D. students. He has published more than 290 papers in international journals. He currently acts as Editor in Chief of the international journals ”Information Fusion” (Elsevier) and ”Progress in Artificial Intelligence” (Springer). He acts as editorial board member of a dozen of journals, among others: International Journal of Computational Intelligence Systems, IEEE Transactions on Fuzzy Systems, Information Sciences, Knowledge and Information Systems, Fuzzy Sets and Systems, Applied Intelligence, Knowledge-Based Systems, Memetic Computation, and Swarm and Evolutionary Computation. He received the following honors and awards: ECCAI Fellow 2009, IFSA Fellow 2013, 2010 Spanish National Award on Computer Science ARITMEL to the ”Spanish Engineer on Computer Science”, International Cajastur ”Mamdani” Prize for Soft Computing (Fourth Edition, 2010), IEEE Transactions on Fuzzy System Outstanding 2008 and 2012 Paper Award (bestowed in 2011 and 2015 respectively), 2011 Lotfi A. Zadeh Prize Best paper Award of the International Fuzzy Systems Association, 2013 AEPIA Award to a scientific career in Artificial Intelligence (September 2013). His current research interests include bibliometrics, computing with words in decision making, information fusion, evolutionary algorithms, evolutionary fuzzy systems, biometrics, data preprocessing, data mining, cloud computing and big data. 18 Humberto Bustince received his Bs. C. degree on Physics from the Salamanca University, Spain, in 1983 and his Ph.D. degree in Mathematics from the Public University of Navarre, Pamplona, Spain, in 1994. He has been a teacher at the Public University of Navarre since 1991, and he is currently a Full Professor with the Department of Automatics and Computation. He served as subdirector of the Technical School for Industrial Engineering and Telecommunications from 01/01/2003 to 30/10/2008 and he was involved in the implantation of Computer Science courses at the Public University of Navarre. He is currently involved in teaching artificial intelligence for students of computer sciences. Dr. Bustince has authored more than 120 journal papers (Web of Knowledge), and more than 100 contributions to international conferences. He has also been coauthor of four books on fuzzy theory and extensions of fuzzy sets. He is fellow of the IEEE Computational Intelligence Systems society and Member of the board of the European Society for Fuzzy Logic and Applications (EUSFLAT). He currently acts as Editor in chief of the Mathware&Soft Computing Magazine and of Notes on Intuitionistic Fuzzy Sets. He is also guest editor of the Fuzzy Sets and Systems journal and member of the editorial board of the Journal of Intelligence&Fuzzy Systems, the International Journal of Computational Intelligence Systems and the Axioms Journal. His current research interests include interval-valued fuzzy sets, Atanassov’s intuitionistic fuzzy sets, aggregation functions, implication operators, inclusion measures, image processing, decision making and approximate reasoning.