Influence Analysis on Discriminant Coordinates

Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=lssp20 Communications in Statistics - Simulation and Computation ISSN: 0361-0918 (Print) 1532-4141 (Online) Journal homepage: https://www.tandfonline.com/loi/lssp20 Influence Analysis on Discriminant Coordinates J. M. Muñoz-Pichardo , A. Enguix-González , J. Muñoz-García & J. L. Moreno- Rebollo To cite this article: J. M. Muñoz-Pichardo , A. Enguix-González , J. Muñoz-García & J. L. Moreno- Rebollo (2011) Influence Analysis on Discriminant Coordinates, Communications in Statistics - Simulation and Computation, 40:6, 793-807, DOI: 10.1080/03610918.2011.556288 To link to this article: https://doi.org/10.1080/03610918.2011.556288 Published online: 16 Mar 2011. Submit your article to this journal Article views: 109 View related articles Citing articles: 1 View citing articles Communications in Statistics—Simulation and Computation®, 40: 793–807, 2011 Copyright © Taylor & Francis Group, LLC ISSN: 0361-0918 print/1532-4141 online DOI: 10.1080/03610918.2011.556288 Influence Analysis on Discriminant Coordinates J. M. MUÑOZ-PICHARDO, A. ENGUIX-GONZÁLEZ, J. MUÑOZ-GARCÍA, AND J. L. MORENO-REBOLLO Departamento de Estadística e I.O., Universidad de Sevilla, Sevilla, Spain Discriminant analysis (DA), particularly Discriminant Coordinates (DC), is broadly applied in the scientific literature and included in many statistical software packages. DC is used to analyze biomedical data, especially for differential diagnosis on the basis of laboratory profiles. Articles handling influence analysis in DA can be found in the literature; however, this topic has been scarcely touched upon in DC. In this article, the case-deletion approach is followed to introduce a perturbation in the data and influence measures are proposed to assess the effect on three statistics of interest: the transformation matrix, canonical directions, and configuration, of the sample centroids. Keywords Deletion approach; Discriminant coordinates; Influence analysis. Mathematics Subject Classification 62H30; 62H99. 1. Introduction Discriminant analysis (DA), also named “classification” and “statistical pattern recognition,” is a methodology commonly used to classify a set of observations into predefined classes or to distinguish between a set of groups or sub-populations. DA has been broadly applied in the scientific literature in almost all scientific fields. In particular, DA has been frequently applied in medical research in order to distinguish between several diseases or between health and disease states. Many medical research articles apply DA for the differential diagnosis on the basis of a laboratory profile. DA includes several techniques. Linear discriminant analysis (LDA), DC (also named canonical variates), and logistic discrimination are probably the most frequently used in differential diagnosis. Mirkin et al. (2004) studied DC in a study of human endometrium; Galanaud et al. (1999) studied DA for a non invasive diagnostic assessment of brain tumors; Struijk et al. (2006) studied a new method for discriminating between people with a normal genotype and those with the congenital long-QT syndrome; Guo et al. (1999) and Perelman et al. (2003) studied DA in microarray data, all illustrate relevant applications of DA in biomedical research. Received December 10, 2009; Accepted January 17, 2011 Address correspondence to J. M. Muñoz-Pichardo, Departamento de Estadística e I.O., Universidad de Sevilla, Avd. Reina Mercedes sn., Sevilla 41012, Spain; E-mail: [email protected] 793 794 Muñoz-Pichardo et al. Generally speaking, influence analysis (IA) deals with the study and assessment of the variations caused in statistical conclusions by perturbations. Several perturbation schemes can be considered, although case-deletion may be the most commonly used in IA. Articles handling IA in almost all statistical techniques can be found in the literature. Obviously, accuracy of the estimation and the classification in DA might be affected by outliers and influential observations. This fact justifies the interest of IA in DA. The study of influence in linear discriminant analysis is normally carried out by using the common case-deletion approach, and usually by assessing its effect on the estimated total probability of misclassification. This approach is applied in the following articles: Campbell (1978), Critchley and Vitiello (1991), and Fung (1992, 1995b,c, 1996). Recently, Moreno-Roldán et al. (2007) proposed two case-deletion diagnostics based on the L2-norm which evaluate the effect of the omission on the linear functions which determine Fisher’s linear discriminant rule. Riani and Atkinson (2001) provided a unified approach to study influential observations and outliers in Quadratic Discriminant Analysis. As far as we know, there is no article which deal with influence in discriminant analysis focused on DC in the literature, perhaps because analysis based on DC and LDA lead to identical results if the complete set of discriminant coordinates is considered. However, for convenience of the subsequent analysis, in particular for the graphical representation of the transformed feature data, only the first two discriminant coordinates are often considered in practice. The results obtained from DC and LDA might differ in such a case. In this article, case-deletion diagnostics are proposed on three statistics of interest in DC when only the first two discriminant coordinates are considered: the transformation or projection matrix, the directions of the projection matrix, and the configuration of the sample centroids of the first two discriminant coordinates. •The transformation matrix. Two diagnostic measures related to this statistic are proposed. The first one assesses the effect of the omission through a matrix norm, the classic norm of Frobenius, and the second one through a ratio of determinants. •The directions of the projection. In this case, the effect of the perturbation is measured through the angle between the non perturbed and perturbed directions of the projections. •The configuration of the sample centroids. The Euclidean distance between the non perturbed and perturbed centroids is considered as influence measure on this statistic. This article is organized as follows. In the next section, the notation, the discriminant rule and the basic statistics in canonical discriminant analysis are introduced. The influence measures that we propose are presented in Sec. 3. In Sec. 4, these influence measures are illustrated with two data sets. Finally, some conclusions are set out in Sec. 5. 2. Discriminant Coordinates First a short introduction to DC, which serves as a means to establish the notation, is given. Let Gii=1g be gmutually exclusive groups or populations and let X=X1X ptbe a p-dimensional random vector. We assume that the distribution of Xfor Giis Npi,i=1g. Influence Analysis on Discriminant Coordinates 795 The basic question in DC is to determine linear combinations of X1X p (discriminant coordinate variables) which reflect the differences between groups, that is, the linear transformations which lead to the greatest separation among the mean vectors. To this end, a sample of size ni(“training” data) is selected from each population, xij ∈pj=1n i,i=1gThe sample means of the groups are denoted by xi•i=1g and the global sample mean by x, that is, x= 1 ng i=1ni j=1xij, where n=g ini.Wand Bdenote the within-group and betweengroup matrices, respectively, that is, W= g  i=1 ni  j=1 xij −xi•xij −xi•tand B= g  i=1 nixi•−xxi•−xt The computation of the vectors (coefficients of the linear combinations), ckk= 1d d=minp g −1, that determine the discriminant coordinates is well covered in standard textbooks on multivariate analysis (see, for instance, Seber, 1984, Ch. 5). The vectors ck,k=1d are the solutions of the optimization problems sup c=0 ctBc ctWc for k=1 and sup c=0ct jWc=0j=1k−1 ctBc ctWc for k=2d (1) If 1≥···≥dare the non zero eigenvalues of W−1Bwith associated unit eigenvectors e1ed, then the kth discriminant coordinate variate is given by ct kXwhere ck=n−g h2 k 1/2ekand h2 k=et kWek. Then the discriminant coordinates have null sample correlations, ct kWcr=0k= r Given k≤d Ck=c1cktis called the transformation or projection matrix. The centroids of the groups in the transformed space are zi•=Ckxi•i=1g. A future observation xis assigned to the ith-group if dECkxCkxi•=min l=1g dECkxCkxl• where dEdenotes the vector Euclidean distance. At this point, it should be borne in mind that the classification rule based on DC is derived under two assumptions: normality and homoscedasticity. Under these assumptions, the discriminant coordinates are obtained by maximizing the F-ratio statistic of the analysis of variance. As this technique is moderately robust to longer-tailed symmetric distributions, the classification rule can be used under moderate violation of the normality assumption. The discriminant coordinates are determined in order to emphasize the separation between groups, but with decreasing effectiveness. From a practical point of view, it is necessary to fix the number kof discriminant coordinates to be considered in the statistical analysis. The relative magnitudes of the eigenvalues 1 dare frequently used to this end. For k=2 and k=3, plots of the transformed data are helpful to study the degree and nature of the separation between the groups. This option of the discrimination problem is provided by several statistical packages which allow DC to be broadly applied in statistical data analysis. Obviously, considering a dimension of the transformed space greater than 3 is not adequate to display the behavior of the groups and the data properly, although plots of pairs of coordinates still can be explored. 796 Muñoz-Pichardo et al. In this article, only k=2 is considered although the diagnostics that we propose can easily be generalized to any value of k. Hence, the most relevant statistics in DC are the transformation matrix C2, the directions that determine the linear transformation, that is, the rows c1and c2 of C2and the sample means in the transformed space zi•i=1g. 3. Influence Measures In the following, influence diagnostics are proposed in order to assess the effect of this perturbation on the most relevant statistics in DC. From now on, for any sample statistic T,Tsl denotes the statistic perturbed by deleting xsl, the lth case of the sth group, from the sample. In particular, xr•sl = xr•if r= s,xs•sl =xs•−1 ns−1xsl −xs•,C2sl is determined by the eigenvectors of W−1 slBsl corresponding to the two largest eigenvalues, and zr•sl =C2slxr•sl. 3.1. Transformation Matrix Two diagnostics are proposed to quantify the effect caused on the transformation matrix C2by the omission of xsl: •The Frobenius norm of the difference matrix C2−C2sl, Msl = C2−C2sl F where AF=ija2 ij for any matrix A=aij. Obviously, Msl can be considered as an influence diagnostic of the effect of the omission of xsl on C2. This diagnostic is called “M-measure.” Large values of Mcorrespond to cases that lead to large changes in the projection matrix upon which the discriminant rule is based. •The aim of canonical discriminant analysis is to determine the linear transformations of the vector of the observed variables that lead to the greatest separation between the groups in the transformed space. Therefore, measuring the effect that the deletion of each observation exerts on discrimination ability of the discriminant coordinates is of interest. The ratio between the between- and within-group dispersion is a measure of discrimination ability, and the determinant of the dispersion matrix of the discriminant coordinates, 1 n−gC2WCt 2, is a measure of the withingroup dispersion in the transformed space. Therefore, if the omission of an observation causes a significant change in the value of the determinant, it also will provoke a significant change in discrimination ability. Since DC leads to standardized discriminant coordinates, 1 n−gC2WCt 2=1, we also propose Rsl = 1 n−gC2slWCt 2sl as an influence measure of the omission of xsl on the dispersion matrix. We will call “R-measure” this diagnostic. Influence Analysis on Discriminant Coordinates 797 Taking into account that W=Wsl +ns/ns−1xsl −xs•xsl −xs•t R-measure can be expressed by R=1−1 n−g2 1+ut slusl where usl =ns n−s−1n−g−11/2zslsl −zs•sl, with zslsl =C2slxsl. Therefore, values of Rfar from 1 are associated with influential observations. Those verifying Rsl −1>005 or 010 are proposed as influential observations. In should be pointed out here that the determinant ratio approach has previously been applied in the literature as an influence diagnostic and to identify outliers. For example, Barnett and Lewis (1978) and Munoz-García et al. (1990) used this approach to identify outliers in a sample from a q-dimensional normal population, while Belsley et al. (1980) proposed an influential diagnostic in linear regression. 3.2. Directions of the Projection Matrix The M-measure quantifies the effect of the omission on the coefficients of the linear projection as a whole. However, separate conclusions about its effect on the directions c1and c2cannot be obtained using this measure. The directions c1and c2 are not orthogonal (ct 1c2= 0) but uncorrelated (ct 1Wc2=0). Therefore, the study of the effect of the omission on c1and c2has to be carried out separately. We propose the angle between cjand cjslj=12as influence measures of the omission of xsl on the direction cj, A1 sl =c1c1sl=100 arccos ct 1c1sl c1c1sl A2 sl =c2c2sl=100 arccos ct 2c2sl c2c2sl We will call “A1-measure” and “A2-measure”, respectively, these diagnostics. For convenience, A1and A2are stated in hexadecimal degrees, and therefore A1A 2∈0100. This fact enables reference values to be fixed in such a way that those cases with “A-measures” greater than the reference values would be considered as influential observations. The reference values for high and moderate influence are fixed at 5 and 2.5, respectively. It should be noted that c1sl is an eigenvector corresponding to the largest eigenvalue of W−1 slBsl, thereby verifying the condition ct 1slWslc1sl =n−g−1. Obviously, the opposite vector −c1sl also verifies this condition. Therefore, c1sl has to be chosen appropriately so that the effect of the omission can be accurately assessed. A similar consideration has to be made for c2sl. 3.3. Centroids We propose assessing the effect of the omission of xsl on the centroids through the sum of the Euclidean distances between the perturbed zr•slr =1g, and 798 Muñoz-Pichardo et al. the non perturbed centroids zr•r =1g, Dsl =g  r=1zr•sl −zr•tzr•sl −zr•1/2  We will call “D-measure” this diagnostic. Obviously, a large value of Dsl means a significant change in the centroid’s configuration and in consequence a significant change in the classification rule. Therefore, cases with large values in the D-measure can be considered as influential observations. To conclude this section, it should be taken into account that the diagnostics here proposed can obviously be generalized if k>2 canonical discriminant variates are necessary to obtain a discriminant rule with admissible error rates. 4. Applications In this section, two examples with real data are presented to illustrate the diagnostics proposed. In the first example, the Lubischew data set (Lubischew, 1962) is used. This data set has been considered by other authors in several studies on discriminant analysis. See, for example, Moreno-Roldán et al. (2007), Bremner and Taplin (2002), Fung (1995a), Schott (1990), and McKay (1977). Two variables from the Lubischew data set have been selected, thereby enabling a simple display and facilitating the interpretation of the measures proposed. The second example is an application of discriminant analysis to a medical data set (Plomteux, 1980). This medical application attempts to determine a differential diagnosis of diseases of the liver on the basis of a laboratory profile determined by liver enzymes. 4.1. Lubischew Data Set This application has been included due to two reasons. Firstly, the Lubischew data set has been widely used in the scientific literature. Secondly, it simplifies the interpretation of the measures here proposed. Lubischew (1962) analyzed three groups of genus of flea beetle: Chaetocnema Concinna (G1), Chaetocnema Heikertingeri (G2), and Chaetocnema Heptapotamica (G3). The groups consist of 21, 31, and 22 observations, respectively. They are labeled as cases 1–21, 22–52, and 53–74, respectively. Each observation consists of six variables from which two have been selected: the fourth variable (the maximal width of the aedeagus in the fore-part in microns) and the sixth variable (the aedeagus width from the side in microns). Figures 1(a) and (b) display the original data set and the transformed data set through the discriminant coordinates, respectively. The resultant eigenvectors of the analysis are ct 1=0147500690and ct 2= −0165001268. Table 1 shows the results of the classification through the discriminant coordinates, whereby 9095% of cases are correctly classified. Cases 6, 8, 9, 16, and 17 of G1are misclassified in G3, and cases 53 and 66 of G3are misclassified in G1through DC. Obviously, these results reflect what Figs. 1(a) and (b) show: cases in G2are clearly separated from those in G1and G3; but some cases corresponding to G1and G3are mixed. Figures 2(a)–(d) display the index plots of the diagnostics proposed in Sec. 3: M-measure, D-measure, A1-measure, and R-measure. The A2-measure plot is Influence Analysis on Discriminant Coordinates 799 Figure 1. Lubischew data set: Scatter plots. ∗-G1;•-G2;-G3. Table 1 Lubischew data set. Results of the classification rule Predicted group Actual group G1G2G3Total G116 (76.2%) 0 5 (23.8%) 21 G20 31 (100%) 0 31 G32 (9.1%) 0 20 (90.9%) 22 omitted since no case is identified as influential from this measure. Table 2 summarizes the results of the influence analysis through these diagnostics. From the results obtained by applying the diagnostics proposed, it can be concluded that: •Cases 17 and 46 are the most influential. Case 17 is identified as influential by the four diagnostics and case 46 by three diagnostics. From Fig. 1(a), it is observed that cases 17 and 46 are extreme values in G1and G3, respectively. Cases 16, 18, 23, 60, and 67 can also be considered as influential observations. •Different cases are identified by the measures proposed here. That is, the influence measures M,D,A1,A2, and Rprovide different information. Each one of these measures assesses the effect of the omission on a different statistic of interest. Hence, each measure provides useful information to the researcher. •The reader could consider the Rand Ameasures as the most interesting diagnostics since they have reference values to determine influential cases. However the information that they provide may not be comprehensive. For example, cases 16, 23, and 60 are not identified by these measures, but they are identified by Mand Dmeasures. •It could be suspected that outliers and influential cases coincide. However, this fact is not necessarily true as the results of this example illustrate. There 800 Muñoz-Pichardo et al. Figure 2. Lubischew data set. Index plot of influence measures (in all graphics, horizontal axis: cases). Table 2 Lubischew data set. Summary of the influence measures Case M-measure D-measure A1-measure R-measure 16 ∗ 17 ∗∗ ∗∗ ∗ ∗ 18 ∗∗ 23 ∗∗ 46 ∗∗ ∗ ∗∗ 60 ∗ 67 ∗ (**) High influence; (*) moderate influence. Influence Analysis on Discriminant Coordinates 807 Moreno-Roldán, D., Munoz-Pichardo, J. M., Enguix-González, A. (2007). Influence diagnostics in multiple discriminant analysis. Test 16(1):172–187. Munoz-García, J., Moreno-Rebollo, J. L., Pascual-Acosta, A. (1990). Outliers: A formal approach. International Statistics Review 58(3):215–226. Perelman, S., Mazzella, M. A., Muschietti, J., Zhu, T., Casal, J. J. (2003). Finding unexpected patterns in microarray data. Plant Physiology 133:1717–1725. Plomteux, G. (1980). Multivariate analysis of an enzyme profile for the differential diagnosis of viral hepatitis. Clinical Chemistry 26:1987–1899. Riani, M., Atkinson, A. (2001). A unified approach to outliers, influence and transformations in discriminant analysis. Journal of Computational and Graphical Statistics 10(3):513–544. Schott, J. R. (1990). Canonical mean projections and confidence regions in canonical variate analysis. Biometrika 77:587–596. Seber, G. A. F. (1984). Multivariate Observations. New York: John Wiley & Sons. Struijk, J., Kanters, J. K., Andersen, M., Hardahl, T. B., Graff, C., Christiansen, M., Toft, E. (2006). Classification of the long-qt syndrome based on discriminant analysis of t-wave morphology. Medical and Biological Engineering and Computing 44(7):543–549.