A study on output normalization in multiclass SVMs

A study on output normalization in multiclass SVMs L. Gonzalez-Abril a, ⇑ , F. Velasco a , C. Angulo b , J.A. Ortega c a Applied Economics I Dept., Seville University, 41018 Seville, Spain b Grup de Recerca en Enginyeria del Coneixement, Universitat Politècnica de Catalunya, 08800 Vilanova i la Geltrú, Spain c Computer Languages and Systems Dept., Seville University, 41012 Seville, Spain article info Article history: Received 11 May 2012 Available online 22 November 2012 Communicated by S. Sarkar Keywords: 1-v-r SVM Convex hull Kernel methods Multiclassification abstract The use of binary support vector machines (SVMs) in multi-classification is addressed in this paper. Margins associated to the bi-classifiers, since they depend on the geometrical disposition of the classes being separated, are, in general, of various magnitudes. In order to overcome this scaling problem, a normalization process should be applied on the SVMs’ outputs. Thus, a new normalization approach is presented based on the convex hulls that contain the classes to be separated. Furthermore, a theoretical study is developed which justifies the proposed approach, and an interpretation is provided. An empirical study is also carried out to compare this normalization with others found in the literature. Ó2012 Elsevier B.V. All rights reserved. 1. Introduction SVMs are learning machines which implement the structural risk minimization inductive principle to obtain good generalization on a limited number of learning patterns (Vapnik, 1998). This theory was developed on the basis of a separable binary classification problem where the optimization criterion is the width of the margin with ‘ 2 -norm 1 between the positive and negative examples. An SVM with a large margin separating two classes has a small VC dimension, which provides good generalization performance, as it has been demonstrated in several applications (Cristianini and Shawe-Taylor, 2000). The extension of binary classification to multi-classification is an on-going research issue (Mayoraz and Alpaydin, 1999; Angulo et al., 2006; Wang et al., 2008). Although some joint SVM methods exist, the binary ad hoc methods of Kone-versus-rest (1-v-r) or K(K1)/2 one-versus-one SVMs for the solution of the multi-class problem still prevail due, in general, to their good performance and manageable optimization. In standard SVM formulation, the output scale is determined such that outputs for the support vectors are ±1. Therefore, a direct comparison of the output of different SVMs working on a multi-classification problem is inadequate because scaling varies for each machine considered. Some kind of normalization is therefore crucial for the comparison of outputs of different SVMs. The usual procedure to circumvent this scaling problem is the direct comparison of the real-valued outputs for each machine. It is argued in this work, however, that the scaling problem cannot be circumvented but must be tackled by considering some kind of normalization. Those available in the literature are analyzed, and a new output normalization method is proposed based on the convex hulls that contain the classes to be separated. More precisely, the proposed normalization is based on the problem of finding the nearest point between reduced convex hulls. Furthermore, the reliability of the standard SVM is taken into account in order to carry out the normalization. The remainder of this paper is arranged as follows: Section 2 presents the standard SVM approach. Section 3puts forward two output-normalization schemes based on different criteria, and the proposed normalization is developed. An experiment is carried out in Section 4in order to show the accuracy rate of several normalization processes. Finally, conclusions are drawn. 2. Standard SVM approach Let Z¼ z i fg n i¼1 ¼ðx 1 ;y 1 Þ;...;ðx n ;y n Þ fg be a training set, with x i 2Xas the input space and y i 2Y¼ h 1 ;h 2 fg ¼þ1;1 fg the output space. Let /:X!Fbe a feature mapping with a dot product denoted by h;i. A linear classifier f w ðxÞ¼hx;wiþbis sought in F, with b2R. Outputs are obtained in the form h w ðxÞ¼signðf w ðxÞÞ. For the standard primal SVM 2-norm formulation (Vapnik, 1998), the optimization problem becomes 0167-8655/$ - see front matter Ó2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.patrec.2012.11.003 ⇑ Corresponding author. E-mail addresses: [email protected] (L. Gonzalez-Abril), [email protected] (F. Velasco), [email protected] (C. Angulo), [email protected] (J.A. Ortega). 1 A generalization is given in (Gonzalez-Abril et al., 2011). Pattern Recognition Letters 34 (2013) 344–348 Contents lists available at SciVerse ScienceDirect Pattern Recognition Letters journal homepage: www.elsevier.com/locate/patrec min w2F;b2R 1 2kwk 2 þCX i n i s:t:y i hx i ;wiþbðÞþn i P1;n i P0;z i 2Z ð1Þ where Cis the regularization term and n i are slack variables. The solution can be written as w¼X i a i y i x i ð2Þ where a i are Lagrange multipliers for the dual problem of (1). Furthermore, X i a i y i ¼0ð3Þ 06 a i 6C;i¼1;...;nð4Þ a i ðy i hx i ;wiþbðÞ1þn i Þ¼0i¼1;...;nð5Þ A vector x i is called a support vector (SV) when a i –0. Term bis calculated a posteriori (Gonzalez-Abril et al., 2008). The classifier can be written as fðxÞ¼X i a i y i hx i ;xiþbð6Þ Note that if a þ i and a  j are multipliers associated to the vectors of Z ðþÞ ¼ðx i ;y i Þ2Z;y i ¼þ1 fg and Z ðÞ ¼ðx j ;y j Þ2Z;y j ¼1  , respectively, then P i a þ i ¼P j a  j >0, since otherwise w¼0. 3. Normalization of the SVM outputs There are several alternatives, further to that which considers the signed output, which tackle the scaling problem when comparing the output of different SVMs. Let us consider the two most used approaches: MS normalization: This normalization, which is called MS (for ‘Mayoraz scheme’), was introduced in (Mayoraz and Alpaydin, 1999). It is obtained by dividing the classifier ffrom (6) by the norm of the solution vector w, that is, the scalar factor is k M ¼ 1 kwk . Hence, the new classifier is f M ðxÞ¼ 1 kwk fðxÞ. This approach presents a geometrical interpretation since the norm of the normal vector of the classifiers f M is one. SRM normalization: An approach, called SRM (static reliability measure), was given in (Liu and Zhang, 2005). It takes into account the value of the objective function of (1).k R ¼expfð 1 2 kwk 2 þCPn i Þ=ðCnÞg is considered as the scalar factor in this approach. Hence, the normalized classifier is rewritten in the form f R ðxÞ¼k R fðxÞ. The main concept behind both the normalization procedures above, MS and SRM, is the same: since the classifier with a smaller kwk, which corresponds to a larger margin, is considered to be more accurate in generalization, then 1 kwk and k R , as reliability measures, should be larger. Nevertheless, there is not theoretical result which provides a good explanation for these scalar factors. A new normalization procedure, denoted by CH (convex hull), is introduced here, and is based on the location of the nearest point between the reduced convex hulls of Z ðþÞ and Z ðÞ (Bennett and Bredensteiner, 2000; Bredensteiner and Bennett, 1999). The geometrical interpretation for this point between convex hulls can be seen in (Bredensteiner and Bennett, 1999). Let us develop this normalization. The solution for (1) can be written González et al., 2006 as w¼kw  ;b¼kb  , for k¼P i a þ i and w  ¼P i u  i x i P j v  j x j , with u  i  and f v  j gas the solution of the dual problem min u;v 1 2X i u i x i X j v j x j          2 s:t:X i u i ¼X j v j ¼1;06u i ; v j 6C k z i 2Z ðþÞ ;z j 2Z ðÞ ð7Þ From (6), the classifier can be written as fðxÞ¼kX i u  i hx i ;xiX j v  j hx j ;xiþb  ! ð8Þ The kvalue is such that the margin between the reduced convex hull of Z ðþÞ ;RZ ðþÞ ¼f P i u i x i ;P i u i ¼1;06u i 6 C k ;z i 2Z ðþÞ g, and Z ðÞ ;RZ ðÞ ¼f P j v j x j ;P i v j ¼1;06 v j 6 C k ;z j 2Z ðÞ g, is non-zero. In this form, the optimization problem (7) is prevented from collapsing into a trivial solution. Note that a large value for kinvolves small reduced convex hulls RZ ðþÞ and RZ ðÞ , and hence the number of support vectors in the solution wwill be high. Since P i a þ i ¼P j a  j ¼k, the kvalue can be interpreted from (7) as the strength that support vectors must attain in order to obtain good accuracy in generalization. A different expression for kcan be obtained from the constraints in the optimization problem (1). From (5),if a i –0, then y i hx i ;wiþbðÞ1þn i ¼0, and by considering (2) and (3), 0¼X i a i ðy i hx i ;wiþbðÞ1þn i Þ ¼X i a i y i x i ;w *+ þbX i a i y i X i a i þX i a i n i ¼hw;wiþ0X i a þ i X j a  j þX i a i n i ¼kwk 2 2kþX i a i n i Therefore, k¼1 2kwk 2 þX i a i n i ! ð9Þ Note that if the problem (1) is separable then, from (9),k¼ 1 2 kwk 2 , since n i ¼0 for all i. Let N SV ðþÞ ¼# a þ i ; a þ i –0  (the number of SVs for the positive class), N SV ðÞ ¼# a  i ; a  i –0  (the number of SVs for the negative class), and N SV ¼N SV ðþÞ þN SV ðÞ (the total number of SVs). From (4), k¼P i a þ i 6CN SV ðþÞ ,k¼P i a  i 6CN SV ðÞ , and P i a i n i 6CP i n i . Hence, lower and upper bounds for the value of kcan be given: 1 2kwk 2 6k6min CN SV ðþÞ ;CN SV ðÞ ;1 2kwk 2 þCX i n i !() Finally, by applying min N SV ðþÞ ;N SV ðÞ no 6 1 2 N SV , a different and more manageable upper bound can be obtained: 1 2kwk 2 6k6C 2N SV Table 1 Characteristics of the selected data sets. Data set Patterns Classes Features Data per classes Iris 150 3 4 All 50 Tae 151 3 5 49, 50, 52 Wine 178 3 13 59, 71, 48 Glass 214 6 9 70, 76, 17, 13, 9, 29 Thyroid 215 3 5 150, 35, 30 Ecoli 330 6 8 141, 77, 52, 35,20,5 Dermat 358 6 34 111, 60, 71, 48, 48, 20 Vowel 990 11 11 All 90 Segment 2310 7 19 All 330 L. Gonzalez-Abril et al. / Pattern Recognition Letters 34 (2013) 344–348 345 The value of Cis given a priori in the problem (1), therefore, as aforementioned, a large value for kresults in a high number of support vectors N SV . Similarly, looking into the lower bound, a small value for kimplies that the margin separating Z ðþÞ and Z ðÞ ; 2 kwk 2 ,is large. Hence, the solution will provide good generalization performance as well as being smooth (small VC-dimension), and therefore its reliability is better than for a sharp solution (kvalue is high). Hence, the proposed approach will consider k CH ¼ 1 k as the scalar factor. The classifier f CH ðxÞ¼k CH fðxÞis considered, that is, from (8), f CH ðxÞ¼X i u  i hx i ;xiX j v  j hx j ;xiþb  Unlike the previously considered normalization procedures, the f CH function has a specific meaning from the solution of a dual problem (7) plus a bias. This normalization agrees with Bayesian arguments advocating that smoother solutions should be given higher-valued weights. The schemes used for multiclassification represent a typical illustration of the problem where the comparison of the output of different SVMs is carried out. Thus, in the following section, a study on the use of the proposed normalization procedure when the 1-v-r SVM scheme is used for multiclassification, is presented since this is the most widely used implementation (Rifkin and Klautau, 2004). 4. Normalization in the 1-v-r SVM scheme Let h 1 ;...;h K fgbe a set of possible labels with KP2 and Z k ¼fðx i ;y i Þ:y i ¼h k g. In the 1-v-r SVM scheme, in a first decomposition phase, Kbinary classifiers are trained to generate functions f j ðxÞ¼hw j ;xiþb j ;16j6K, by separating training vectors Z j with label h j from the rest of the training vectors ZnZ j (j-v-r SVM). In the reconstruction phase, the label distribution generated by the trained machines is considered through a merging scheme. A map H :X! h 1 ;...;h K fg is defined from the set of classifiers F¼f j ðxÞ  K j¼1 such that, given a input vector x, it assigns a label as follows (non-normalization or standard scheme – SS): H SS ðxÞ¼arg max j¼1;...;K f j ðxÞð10Þ It is demonstrated below that by means of experimental results and a statistical study, the use of a normalization process for this multiclassification problem is straightforward and sufficient. Firstly, it will be defined: MS normalization: The classifiers are f M j ðxÞ¼ 1 kw j k f j ðxÞ, and the decision function is H M ðxÞ¼arg max j¼1;...;K f M j ðxÞ. SRM normalization: The classifiers, f R j ðxÞ¼k R j f j ðxÞ, and the decision function, H R ðxÞ¼arg max j¼1;...;K f R j ðxÞ, are considered. CH normalization: The classifiers are f CH j ðxÞ¼k CH j f j ðxÞ, and the decision function is H CH ðxÞ¼arg max j¼1;...;K f CH j ðxÞ. It is worth noting that if K¼2, then (10) and the three normalization approaches provide the same solutionsince there are only twojv-r SVM functions, f 1 ðxÞ(1-v-2 SVM) and f 2 ðxÞ¼f 1 ðxÞ(2-v-1 SVM). Note that f j ðxÞ;f M j ðxÞ;f R j ðxÞand f CH j ðxÞcan be interpreted as the similarity score for the h j class (Crammer and Singer, 2001), and the class of xis therefore assigned to that of the classifiers with the highest score. Hence, the normalization process is very useful when SVMs are used in order to give a score for the different classes. 5. Experimental results and statistical study The comparison between normalization and non-normalization is conducted on nine widely used data sets from UCI Repository. 2 The experiment was carried out by following a similar experimental framework to that used in (Angulo et al., 2006) as suggested in (Hsu and Lin, 2002). Hence, the training data are normalized in order to avoid problems with outliers. The test data is normalized accordingly. The selected data sets are: Iris Plants, Teaching assistant evaluation, Wine Recognition Data, Glass Identification Database, Thyroid Disease, Protein Localization Sites (E. coli), Dermatology, Vowel Recognition Data, and Image Segmentation. A summary of the characteristics of these data sets is shown in Table 1. Performanceforthe1-v-rSVM,intheformofaccuracyrate,isevaluated on models using the linear kernel which is chosen as a baseline for the empirical evaluation, and Cis explored on a one-dimensional grid with the following values: C¼2 3 ;2 2 ;...;2 9 ;2 10 hi . The criteria employed to estimate the generalized accuracy is the N-fold cross-validation on the whole set of training data. 3 This procedure is repeated Rtimes, according to the size of the data set, in order to ensure good statistical behavior. The best cross-validation mean rate for the values of C, its standard deviation, and the value of the optimal parameter Care reported in Table 2 for the SS;MS;SRM and CH schemes. Table 2 Results of the experiment where the best mean accuracy rates, its standard deviation and C-parameter are presented. Data set R,N SS MS SRM CH Iris 50,10 94:85 0:537 ð2 10 Þ91:72 0:722 ð2 2 Þ95:48 0:524 ð2 10 Þ96:96 0:446 ð2 1 Þ Tae 50,10 48:05 1:223 ð2 2 Þ46:75 1:169 ð2 2 Þ48:05 1:223 ð2 2 Þ48:08 1:214 ð2 2 Þ Wine 50,10 97:94 0:322 ð2 1 Þ97:89 0:330 ð2 0 Þ97:93 0:324 ð2 1 Þ97:71 0:354 ð2 1 Þ Glass 30,10 57:86 1:102 ð2 3 Þ54:57 1:140 ð2 2 Þ58:10 1:080 ð2 3 Þ58:18 1:108 ð2 3 Þ Thyroid 30,10 96:11 0:448 ð2 9 Þ95:21 0:467 ð2 3 Þ95:92 0:456 ð2 10 Þ96:19 0:402 ð2 5 Þ E. coli 25,5 85:59 0:485 ð2 1 Þ85:75 0:454 ð2 1 Þ85:70 0:480 ð2 1 Þ85:53 0:461 ð2 1 Þ Dermat 25,5 96:40 0:192 ð2 0 Þ97:10 0:210 ð2 3 Þ96:53 0:192 ð2 0 Þ96:99 0:210 ð2 3 Þ Vowel 10,2 54:81 0:270 ð2 0 Þ44:07 0:300 ð2 0 Þ54:79 0:268 ð2 0 Þ54:58 0:284 ð2 0 Þ Segment ⁄ 89:64 0:174 ð2 3 Þ90:21 0:117 ð2 3 Þ89:73 0:169 ð2 3 Þ91:22 0:115 ð2 3 Þ Table 3 p-Value and sample size min hypothesis tests. Data set SS vs MS MS vs CH SS vs CH m Iris 0.0000 0:0000 0.0000 7500 Tae 0:1090 0:1028 0:9771 7500 Wine 0:8297 0:4029 0:2931 8500 Glass 0:0002 0:0000 0:7180 6300 Thyroid 0:0126 0:0065 0:8168 6300 Ecoli 0:7727 0:6895 0:9118 8250 Dermat 0.0083 0:6574 0.0278 8875 Vowel 0:0000 0:0000 0:0989 9900 Segment 0.0064 0:0000 0.0000 41200 2 Available at http://www.ics.uci.edu/mlearn/MLRepository.html. 3 Except for the Segment data set, where the whole data is randomly partitioned 20 times by stratified sampling into a training set (with 250 instances) and a test set. 346 L. Gonzalez-Abril et al. / Pattern Recognition Letters 34 (2013) 344–348 Some conclusions can be drawn from the experimentation carried out: The accuracy rate attained for the CH normalization is the best in 5 of the 9 data sets (see Table 2). Furthermore, there are no statistically significant differences between the accuracy rates of the CH normalization and the best obtained accuracy rates (see Table 3), when it is not the winner. The accuracy rate attained for the standard approach is improved in 7 of the 9 data sets using some kind of normalization. The MS normalization is the best option in two cases. Nevertheless, its results have been very poor for another two cases (Glass and Vowel). This is due to the fact that only the value of kw j kis taken into account in the scalar factor and, in these cases, the value of CPn i is high due to the errors (low accuracy rate). The computational cost in each normalization is small since the norm of the normal vector (MS), the value of the decision function (SRM) and the sum of the Lagrange multipliers (CH) are usually all calculated in the optimization problem. However, if these values were not calculated directly then the complexity would be OðnÞ. Hypothesis tests have been carried out in order to test whether there are statistically significant differences between the accuracy rates. Note that the SRM approach is omitted since its mean accuracy rates are not the best in any data set. Let X i ¼0 be assigned when the imachine provides an error for the pattern x i , and X i ¼1 otherwise. Hence, X i follows a Bernoulli distribution, X i Bðp i Þ, where p i is the probability that the imachine assigns a correct class for a new input. Given two machines, by testing H 0 :p i ¼p j versus H 1 :p i –p j , the statistic is Z¼ffiffi n pðX i X j Þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X i ð1X i ÞþX j ð1X j Þ pNð0;1Þ, where X i and X j are the mean sample of size nof the X i and X j variables, respectively. The p-value for each hypothesis test is given in Table 3. From Table 3, it can be observed that the normalization procedure has significantly improved the accuracy rate in Iris, Dermatology and Segment data sets with respect to the standard scheme, since all p-values are less than 2.78%. As an important result, the accuracy rate for the CH normalization is significantly better than the accuracy rate for the SS schemes in 3 out of the 9 data sets. Furthermore, there are no statistically significant differences between the accuracy rates between these two approaches in the rest of the data sets. Table 4 Results of the experiment where the best mean accuracy rates, its standard deviation and C-parameter are presented for a polinomial kernel of degree 2. Data set R,N SS MS SRM CH Iris 50,10 97:07 0:398 ð2 0 Þ96:97 0:410 ð2 1 Þ97:07 0:398 ð2 0 Þ97:37 0:394 ð2 1 Þ Tae 50,10 58:71 1:339 ð2 6 Þ59:11 1:385 ð2 6 Þ58:58 1:337 ð2 6 Þ58:84 1:342 ð2 6 Þ Wine 50,10 96:90 0:364 ð2 2 Þ96:82 0:370 ð2 2 Þ96:90 0:364 ð2 2 Þ97:18 0:358 ð2 2 Þ Glass 30,10 68:08 0:909 ð2 3 Þ69:66 0:869 ð2 5 Þ68:85 0:912 ð2 3 Þ70:05 0:942 ð2 4 Þ Thyroid 30,10 97:40 0:290 ð2 0 Þ97:34 0:297 ð2 1 Þ97:40 0:290 ð2 0 Þ97:44 0:300 ð2 0 Þ Ecoli 25,5 84:83 0:476 ð2 2 Þ84:93 0:455 ð2 0 Þ87:75 0:456 ð2 2 Þ84:77 0:438 ð2 1 Þ Dermat 25,5 95:61 0:177 ð2 3 Þ95:47 0:177 ð2 3 Þ95:61 0:177 ð2 3 Þ95:44 0:190 ð2 3 Þ Vowel 10,2 86:72 0:191 ð2 1 Þ86:67 0:218 ð2 1 Þ86:81 0:192 ð2 1 Þ88:89 0:205 ð2 1 Þ Segment 90:79 0:101 ð2 1 Þ91:62 0:105 ð2 0 Þ90:79 0:101 ð2 1 Þ91:97 0:109 ð2 1 Þ Table 5 Results of the experiment where the best mean accuracy rates, its standard deviation and C-parameter are presented for a polynomial kernel of degree 3. Data set R,N SS MS SRM CH Iris 50,10 97:07 0:425 ð2 1 Þ97:20 0:417 ð2 1 Þ97:07 0:425 ð2 1 Þ97:22 0:417 ð2 1 Þ Tae 50,10 56:40 1:216 ð2 1 Þ56:80 1:204 ð2 1 Þ56:71 1:219 ð2 1 Þ56:93 1:204 ð2 1 Þ Wine 50,10 96:47 0:408 ð2 2 Þ96:08 0:433 ð2 2 Þ96:47 0:408 ð2 2 Þ96:08 0:433 ð2 2 Þ Glass 30,10 68:56 0:951 ð2 2 Þ69:81 0:904 ð2 2 Þ68:75 0:951 ð2 2 Þ70:15 0:946 ð2 2 Þ Thyroid 30,10 96:25 0:405 ð2 0 Þ96:22 0:397 ð2 0 Þ96:24 0:405 ð2 0 Þ96:22 0:421 ð2 0 Þ Ecoli 25,5 83:13 0:460 ð2 2 Þ83:09 0:442 ð2 2 Þ83:19 0:448 ð2 2 Þ83:27 0:409 ð2 2 Þ Dermat 25,5 95:04 0:230 ð2 4 Þ94:10 0:250 ð2 4 Þ95:02 0:230 ð2 4 Þ94:31 0:246 ð2 4 Þ Vowel 10,2 89:27 0:210 ð2 0 Þ91:29 0:170 ð2 0 Þ89:27 0:210 ð2 0 Þ91:31 0:179 ð2 0 Þ Segment 90:75 0:121 ð2 0 Þ91:76 0:114 ð2 0 Þ90:76 0:120 ð2 0 Þ92:26 0:104 ð2 0 Þ Table 6 Results of the experiment where the best mean accuracy rates, C-parameter and r-parameter are presented. Data set R,N SS MS SRM CH Iris 50,10 96:27 0:520 ð2 2 ;2 1 Þ96:13 0:512 ð2 2 ;2 1 Þ96:26 0:520 ð2 2 ;2 1 Þ96:31 0:520 ð2 2 ;2 1 Þ Tae 50,10 60:40 1:221ð2 6 ;2 1 Þ60:71 1:223 ð2 6 ;2 1 Þ60:30 1:217 ð2 6 ;2 1 Þ60:63 1:201 ð2 6 ;2 1 Þ Wine 50,10 98:51 0:282ð2 5 ;2 1 Þ98:54 0:271 ð2 5 ;2 1 Þ98:51 0:282 ð2 5 ;2 1 Þ98:55 0:271ð2 5 ;2 1 Þ Glass 30,10 74:61 1:040 ð2 5 ;2 0 Þ74:44 1:028 ð2 4 ;2 0 Þ74:10 1:037 ð2 4 ;2 0 Þ74:53 1:074 ð2 5 ;2 0 Þ Thyroid 30,10 97:27 0:349ð2 5 ;2 1 Þ97:43 0:328 ð2 5 ;2 1 Þ97:27 0:349 ð2 5 ;2 1 Þ97:37 0:360 ð2 5 ;2 1 Þ Ecoli 25,5 86:60 0:410 ð2 3 ;2 1 Þ86:59 0:350 ð2 3 ;2 1 Þ86:56 0:395ð2 3 ;2 1 Þ86:54 0:358 ð2 3 ;2 1 Þ Dermat 25,5 96:65 0:145 ð2 4 ;2 1 Þ96:63 0:144ð2 4 ;2 0 Þ96:59 0:150ð2 4 ;2 1 Þ96:66 0:145 ð2 4 ;2 1 Þ Vowel 10,2 95:24 0:250 ð2 3 ;2 0 Þ94:60:2845 ð2 7 ;2 1 Þ94:60 0:216 ð2 5 ;2 0 Þ95:30 0:220 ð2 7 ;2 2 Þ Segment 90:06 0:176 ð2 3 ;2 2 Þ89:85 0:165 ð2 3 ;2 2 Þ90:07 0:175 ð2 3 ;2 2 Þ90:54 0:146ð2 3 ;2 2 Þ L. Gonzalez-Abril et al. / Pattern Recognition Letters 34 (2013) 344–348 347 As aforementioned, a conclusion is that if the accuracy rate in a data set is small then the MS normalization must not be considered since its scalar factor fails to take the errors into account. The same conclusions obtained from the experimentation with the linear kernel are derived when nonlinear kernels are employed. Thus, three experiments with polynomial kernels (degrees two and three) and the RBF kernel are carried out, similarly to the linear kernel, whose results are provided in Tables 4–6. 6. Conclusions Output normalization is an absolutely necessary procedure for the comparison of outputs from different SVMs, since scaling varies for each machine considered. Note that in the experimentation, the training and test data are normalized in order to prevent problems with outliers. Hence, the argument defended in this paper is that the output must also be normalized. The CH normalization process is a good alternative since It has a rigorous theoretical foundation based on the reliability of the classifier. It follows Bayesian arguments advocating that smoother solutions should be given higher-valued weights. It has been empirically demonstrated that the accuracy rate of this normalization process is significantly better than the accuracy rate of the non-normalization in some data sets. No statistically significant differences between the accuracy rates of these two approaches exist in the rest of the data sets. Acknowledgements This study was partially supported by a grant from the Andalusian Regional Ministry of Economy, Innovation and Science (Simon-TIC-8052). References Angulo, C., Ruiz, F., González, L., Ortega, J.A., 2006. Multi-classification by using triclass SVM. Neural Proc. Lett. 23 (1), 89–101. Bennett, K.P., Bredensteiner, E.J., 2000. Duality and geometry in SVM classifiers. In: Proceedings of the Seventeenth International Conference on Machine Learning. Morgan Kaufmann Publishers Inc., pp. 57–64. Bredensteiner, E.J., Bennett, K.P., 1999. Multicategory classification by support vector machines. Comput. Optim. Appl. 12 (1–3), 53–79. Crammer, K., Singer, Y., 2001. On the algorithmic implementation of multiclass kernel-based vector machines. J. Machine Learn. Res. 2, 265–292. Cristianini, N., Shawe-Taylor, J., 2000. An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. Cambridge University Press. Gonzalez-Abril, L., Angulo, C., Velasco, F., Ortega, J., 2008. A note on the bias in SVMs for multi-classification. IEEE Trans. Neural Networks 19 (4), 723–725. Gonzalez-Abril, L., Velasco, F., Ortega, J., Franco, L., 2011. Support vector machines for classification of input vectors with different metrics. Comput. Math. Appl. 61 (9), 2874–2878. González, L., Angulo, C., Velasco, F., Català, A., 2006. Dual unification of bi-class support vector machine formulations. Pattern Recognition 39 (7), 1325– 1332. Hsu, C., Lin, C., 2002. A comparison of methods for multiclass support vector machine. IEEE Trans. Neural Networks 13 (2), 415–425. Liu, Y., Zhang, Y.F., 2005. One-against-all multi-class SVM classification using reliability measures. In: Proc. of the IJCNN ’05. Mayoraz, E., Alpaydin, E., 1999. Support vector machines for multi-class classification. In: IWANN, vol. 2, pp. 833–842. Rifkin, R., Klautau, A., 2004. In defense of one-vs-all classification. J. Machine Learn. Res. 5, 101–141. Vapnik, V., 1998. Statistical Learning Theory. John Wiley & Sons Inc.. Wang, L., Xue, P., Chan, K., 2008. Two criteria for model selection in multiclass support vector machines. IEEE Trans. Systems Man Cybernet. Part B 38 (6), 1432–1448. 348 L. Gonzalez-Abril et al. / Pattern Recognition Letters 34 (2013) 344–348