On the effect of measurementmodel misspecification in PLS Path Modeling: the reflective case

Minotti et al. Measurement model misspecification in PLS-PM 2nd International Symposium on Partial Least Squares Path Modeling, Seville (Spain), 2015 1 On the effect of measurement model misspecification in PLS Path Modeling: the reflective case. Completed Research Paper Simona C. Minotti Università degli Studi di Milano-Bicocca [email protected] Tomàs Aluja-Banet Universitat Politecnica de Catalunya [email protected] Giuseppe Lamberti Universitat Politecnica de Catalunya [email protected] Antonio Ciampi McGill University [email protected] Abstract The specification of a measurement model as reflective or formative is the object of a lively debate. Part of the existing literature focuses on measurement model misspecification. This means that a true model is assumed and the impact on the path coefficients of using a wrong model is investigated. The majority of these studies is restricted to Structural Equation Modeling (SEM). Regarding PLS-Path Modeling (PLS-PM), a few authors have carried out simulation studies to investigate the robustness of the estimates, but their focus is the comparison with SEM. The present paper discusses the misspecification problem in the PLSPM context from a novel perspective. First, a real application on Alumni Satisfaction will be used to verify whether different assumptions for the measurements models influence the results. Second, the results of a Monte-Carlo simulation study, in the reflective case, will help to bring some clarity on a complex problem that has not been sufficiently studied yet. Keywords: PLS Path Modeling, Measurement Model Misspecification, Alumni Satisfaction, Monte-Carlo Simulation Study . 1. Introduction The specification of a measurement model as reflective or formative is the object of a lively debate (Diamantopoulos and Siguaw, 2006; Diamantopoulos et al., 2008; Edwards, 2010; Hardin et al., 2011; Hardin and Marcoulides, 2011; MacKenzie et al., 2011; Howell et al., 2013a; Howell et al., 2013b; Howell et al., 2013c). This debate, which covers different applied areas, reflects the fact that there is no universally accepted underlying theory to guide the choice. Part of the existing literature focuses on measurement model misspecification. By this we mean that a true model is assumed and the impact on the path coefficients of using a wrong model is investigated. The majority of these studies is restricted to Structural Equation Modeling (SEM) approach (Bollen and Lennox, 1991; Edwards and Bagozzi, 2000; Diamantopolous and Winklhofer, 2001; Jarvis et al., 2003; MacKenzie et al., 2005; Petter et al.; 2007; Kim et al., 2010; Hardin et al., 2011). However, the problem persists because: first, these studies lead to different conclusions (see for example the allegations of Aguirre-Urreta and Marakas, 2012 against Jarvis et al., 2003 and Petter et al., 2007 about the use of standardized coefficients); second, these studies are not comparable because they treat different models and applications, and, even in the case of simulation-based papers, it is not Minotti et al. Measurement model misspecification in PLS-PM 2nd International Symposium on Partial Least Squares Path Modeling, Seville (Spain), 2015 2 clear how the data were simulated; third, none of these papers was published in statistical journals. If we move to the PLS-Path Modeling (PLS-PM) context (Tenenhaus et al., 2005), which is the focus of our work, then we find a lack of published results. Roy et al. (2012) for example study the misspecification problem with respect to an application in the Operations and Manufacturing Management Research, but they don’t discuss any simulation study. Then, in the statistical literature, a few authors have carried out simulation studies to investigate the robustness of PLS-PM estimates, but they refer to the European Customer Satisfaction Index (ECSI) model and their focus is the comparison with SEM (Cassel et al., 1999; Cassel et al., 2000; Vilares et al., 2010; Dolce and Lauro, 2014). In the present paper we will approach the misspecification problem from a perspective which is novel in the PLS-PM context. First, we will start from a real application referred to Alumni Satisfaction, where obviously the real model is not known, and assume different measurement models to verify if the results are sensitive to that choice. Second, we will present the results of a Monte-Carlo simulation study, in the reflective case, and focus on the consequences of the problem of measurement model misspecification, in terms of properties of the estimates (i.e., mean squared error, bias and variance). 2. A PLS Path Model for Alumni Satisfaction We will present in the following an analysis of a real application referred to Alumni Satisfaction. Here the real model is obviously not known, and the aim is to verify if assuming reflective or formative measurement models leads to important divergences in the estimation of the path coefficients. 2.1. The dataset The dataset comes from a survey realized in 2008, collected on 147 alumni of the Barcelona School of Informatics (BSI) three years after their graduation. The questionnaire was based on a simplified version of the European Customer Satisfaction Index (ECSI) model (Fornell, 1992) and it consisted of 24 questions referred to 5 latent constructs (see Table 1). The students were asked to provide measures on a 11-point ordinal scale ranging from very satisfied (10) to very dissatisfied (0). The goal of the study was to explore the Alumni Satisfaction about the formation received at BSI in connection to their actual work conditions. In particular, the aim was to study the relationships between Alumni Satisfaction (Satisfaction), which is the main proxy for school reputation and recommendation, and the following drivers: perceived image of the school (Image), perceived quality on generic skills 1 (Specific Quality), perceived quality on technical skills 2 (Technical Quality), and advantage or profit that the alumni could draw from the school degree (Value). 1 The specific skills refer to a broad spectrum of capabilities not specific to a profession or organizational environment, such as the ability of problem solving, communication, time management, team working, initiative… 2 The technical skills refer to the knowledge and abilities, specific to a profession, either mathematical or engineering based, or specific to accomplish technical tasks. Minotti et al. Measurement model misspecification in PLS-PM 2nd International Symposium on Partial Least Squares Path Modeling, Seville (Spain), 2015 3 Table 1: Description of the manifest variables for each latent construct Latent Variables Description Image 1 - It is the best college to study Information Engineering 2 - It is internationally recognized 3 - It has a wide range of courses 4 - The professors are good 5 - Facilities and equipment are good 6 - It is leader in research 7 - It is well regarded by the companies 8 - It is oriented to new needs and technologies Specific Quality 1 - Basic skills 2 - Specific technic skills 3 - Applied skills Generic Quality 1 - Achieved abilities in solving problem 2 - Training in business management 3 - The written and oral communication skills 4 - Planning and time management acquired 5 - Team-work skills Value 1 - It has allowed me to find a well paid job 2 - I have good perspectives in improvement and promotion 3 - It has allowed me to find a job that motivates me 4 - The training received is the basis on which I will develop my career Satisfaction 1 - I am satisfied with the training received 2 - I am satisfied with my current situation 3 - I think I'll have a good career 4 - What do you think is the prestige of your work 2.2. The model The PLS path model for Alumni Satisfaction is represented in Figure 1 and described in more detail in Lamberti and Aluja-Banet (2015). It is an adaptation of the model presented in Martensen et al. (2000) and Eskildsen et al. (1999). The measurement models related to the latent variables are all assumed to be reflective, that is, we assume that each latent construct is a cause of the corresponding indicators. The estimates of the path coefficients obtained by the plspm R package (Sánchez, 2012) are also reported In Figure 1 (the asterisk indicates non-significant values). For a detailed discussion and interpretation of the results see Lamberti and Aluja-Banet (2015). 2.3. The misspecified model We report in Figure 2 the results of the estimation of a different PLS path model. Here the measurement models for Specific and Generic Quality are assumed to be formative, as well as some literature on customer satisfaction suggests (see for example Müller et al., 2006). Since there is disagreement in the literature on this point, we want to verify if the choice of the type of measurement model affects the estimates of the path coefficients. Minotti et al. Measurement model misspecification in PLS-PM 2nd International Symposium on Partial Least Squares Path Modeling, Seville (Spain), 2015 4 Figure 1: The PLS path model for Alumni Satisfaction Figure 2: The misspecified PLS path model for Alumni Satisfaction What emerges is that assuming different measurement models does not seem to have a considerable impact on the estimation of the path coefficients, which are in fact very similar in the two cases (compare the estimates reported in Figure 1 and 2). These results motivated the simulation study we are going to present in order to understand misspecification better and in which cases it produces a considerable effect. Minotti et al. Measurement model misspecification in PLS-PM 2nd International Symposium on Partial Least Squares Path Modeling, Seville (Spain), 2015 5 3. The Monte-Carlo simulation study The aim of the proposed Monte-Carlo simulation study is generalizing the findings of the previous section to a broader class of situations where we actually know what the real model is. This is a misspecification study in the sense that it allows us to observe the behavior of the path coefficient estimates (in terms of mean squared error, bias and variance) when we assume the right model and the wrong models. The study is restricted to the reflective case and represents a first step in the attempt to bring some clarity on a complex problem that has not been sufficiently studied yet. 3.1. The model As a starting point, we consider a PLS path model with one exogenous and one endogenous latent variable (LV), both of reflective type. An exogenous LV can be defined as a variable “of external origin”, with no causes included in the model (i.e., no arrows pointing to the variable; only arrows pointing out). An endogenous LV can be defined as a variable “of internal origin” and is represented as the effect of other variables (i.e., at least one arrow pointing to it). The measurement model for the exogenous LV is: X = λ x ξ + ε x where ξ is a random variable which denotes the exogenous LV; X=(X 1 ,…,X q )’ is a pdimensional vector of observable random variables; λ x =(λ x1 ,…, λ xp )' is a p-dimensional vector of unknown loadings; ε x =(ε x1 , …, ε xp )' is a p-dimensional vector of errors of measurement, with expected value equal to zero and uncorrelated to ξ. The measurement model for the endogenous LV is: Y = λ y η + ε y where η is a random variable which denotes the endogenous LV; Y=(Y 1 ,…,Y q )’ is a qdimensional vector of observable random variables; λ y =(λ y1 ,…, λ yq )' is a q-dimensional vector of unknown loadings; ε y =(ε y1 , …, ε yq )' is a q-dimensional vector of errors of measurement, with expected value equal to zero and uncorrelated to η. The model is completed by the structural equation which describes the relationship between the LVs: η = βξ + ζ where β is the unknown path coefficient which represents the relationship between η and ξ, while ζ is a random variable with expected value equal to zero and indicating the error in equation associated with η. By assuming to have three indicators for each latent variable, the considered path model is represented in Figure 3. Minotti et al. Measurement model misspecification in PLS-PM 2nd International Symposium on Partial Least Squares Path Modeling, Seville (Spain), 2015 6 Figure 3: The path model for the simulation study 3.2. Description of the simulation study The parameters of the simulations which should be fixed in advance are: the number k of simulations, the sample size n, the path coefficient β, the loadings λ x =(λ x1 ,λ x2 ,λ x3 )' and λ y =(λ y1 ,λ y2 ,λ y3 )' and the standard deviations s x and s y . The data were simulated according to the following process. Repeat for k times the following steps: 1. Calculate the 2x2 correlation matrix between ξ and η as: Σ1 β β 1 2. Simulate n realizations of ξ and η, ξ and η, from a bivariate Gaussian random variable with expected value equal to zero and correlation matrix equal to Σ. 3. Standardize the n-dimensional vectors ξ and η, by obtaining ξ s and η s . 4. Given ξ s and η s , estimate the path coefficient β ^ by regressing η s on ξ s . 5. Given ξ s , λ x and s x , and η s , λ y and s y , , respectively, calculate the n realizations of the manifest variables x and y as: x = ξ s Λ x + ε x , and y = η s Λ y + ε y , by simulating the n realizations of ε x and ε y from two univariate Gaussian random variables with expected value equal to zero and standard deviations equal to s x and s y , respectively. 6. Estimate the PLS path coefficient β PLS in the reflective-reflective case and in the misspecified cases, i.e. formative-formative, formative-reflective, reflective-formative. The experimental conditions which were considered are: (a) the sample size (n=100, 400, 1000), (b) the random fluctuation of the manifest variables (s x =s y =0.05, 0.2, 1.0), (c) the strength of the path coefficient (β=0.1, 0.5, 0.9), (d) the strength of the loadings (λ x =λ y =0.8, 0.2). The parameter k was fixed to a value of 500. 3.3.1. The results For each group of simulations we compared the simulated β ^ with the assumed theoretical β. We also compared the PLS coefficients, β PLS , estimated in the misspecified cases with the one estimated in the reflective-reflective case (which represents the true model). The comparisons were made in terms of mean squared error, bias and variance. What emerged from the analysis of the single simulations is that a misspecification effect Minotti et al. Measurement model misspecification in PLS-PM 2nd International Symposium on Partial Least Squares Path Modeling, Seville (Spain), 2015 7 appears when the path coefficient, β, is low (i.e., equal to 0.1, see Table 2 and 4) and the random fluctuation of the manifest variables, s x and s y , are low (i.e., 0.05 and 0.2); it is almost negligible when β increases (i.e., equal to 0.5 and 0.9, see Table 3 and 5). As the sample size n increases, the pattern doesn’t change, but the misspecification effect reduces drastically (compare Table 2 and 3 with Table 4 and 5, respectively). Table 2: Results of the simulations n=100, β=0.1, s x =s y =0.05 Case a. λ x =λ y =0.8 Simulated versus theoretical path coefficient PLS versus simulated path coefficient Ref/Ref Form/Form Form/Ref Ref/Form MSE 0.0098 0.0001 0.0389 0.0085 0.0095 Bias -0.0071 0.0003 0.0530 0.0405 0.0412 Var 0.0097 0.0001 0.0362 0.0069 0.0078 Case b. λ x =λ y =0.2 Simulated versus theoretical path coefficient PLS versus simulated path coefficient Ref/Ref Form/Form Form/Ref Ref/Form MSE 0.0093 0.0011 0.0391 0.0086 0.0094 Bias -0.0092 0.0029 0.0578 0.0433 0.0442 Var 0.0092 0.0011 0.0358 0.0067 0.0075 Table 3: Results of the simulations n=100, β=0.9, s x =s y =0.05 Case a. λ x =λ y =0.8 Simulated versus theoretical path coefficient PLS versus simulated path coefficient Ref/Ref Form/Form Form/Ref Ref/Form MSE 0.0004 0.0000 0.0000 0.0000 0.0000 Bias -0.0015 -0.0013 0.0030 0.0008 0.0008 Var 0.0004 0.0000 0.0000 0.0000 0.0000 Case b. λ x =λ y =0.2 Simulated versus theoretical path coefficient PLS versus simulated path coefficient Ref/Ref Form/Form Form/Ref Ref/Form MSE 0.0004 0.0004 0.0003 0.0003 0.0003 Bias -0.0022 -0.0176 -0.0127 -0.0152 -0.0152 Var 0.0004 0.0001 0.0001 0.0001 0.0001 Table 4: Results of the simulations n=400, β=0.1, s x =s y =0.05 Case a. λ x =λ y =0.8 Simulated versus theoretical path coefficient PLS versus simulated path coefficient Ref/Ref Form/Form Form/Ref Ref/Form MSE 0.0024 0.0000 0.0050 0.0012 0.0012 Bias -0.0019 0.0002 0.0367 0.0223 0.0227 Var 0.0024 0.0000 0.0036 0.0007 0.0007 Case b. λ x =λ y =0.2 Simulated versus theoretical path coefficient PLS versus simulated path coefficient Ref/Ref Form/Form Form/Ref Ref/Form MSE 0.0024 0.0001 0.0055 0.0013 0.0011 Bias -0.0017 0.0017 0.0351 0.0239 0.0208 Var 0.0024 0.0001 0.0042 0.0008 0.0006 Minotti et al. Measurement model misspecification in PLS-PM 2nd International Symposium on Partial Least Squares Path Modeling, Seville (Spain), 2015 8 Table 5: Results of the simulations n=400, β=0.9, s x =s y =0.05 Case a. λ x =λ y =0.8 Simulated versus theoretical path coefficient PLS versus simulated path coefficient Ref/Ref Form/Form Form/Ref Ref/Form MSE 0.0001 0.0000 0.0000 0.0000 0.0000 Bias -0.0002 -0.0011 -0.0001 -0.0006 -0.0006 Var 0.0001 0.0000 0.0000 0.0000 0.0000 Case b. λ x =λ y =0.2 Simulated versus theoretical path coefficient PLS versus simulated path coefficient Ref/Ref Form/Form Form/Ref Ref/Form MSE 0.0001 0.0004 0.0003 0.0003 0.0003 Bias -0.0003 -0.0182 -0.0169 -0.0176 -0.0175 Var 0.0001 0.0000 0.0000 0.0000 0.0000 To verify if the symmetry of the Gaussian distribution could have affected the results, we generated the data from a beta random variable with parameters (6,3); as expected, the results didn’t substantially change. Then, as a summary of the previous results, we present the following three separated analyses of the misspecification effect, by marginalisation of the obtained results according the sample size, the random fluctuations and the path coefficient, respectively. 3.3.2. Behavior of the bias and the MSE according to the sample size (n=100, 400, 1000) The effect of misspecification tends to slow down with sample size; it is clear with n=100, but diminishes when the sample size increases, being very small with n=1000 (see Figure 4). Figure 4: Behavior of the bias and MSE according to the sample size Minotti et al. Measurement model misspecification in PLS-PM 2nd International Symposium on Partial Least Squares Path Modeling, Seville (Spain), 2015 9 In all cases the misspecification follows the same pattern. When the model is correctly estimated (Ref/Ref), then the bias and the MSE of the PLS-PM estimates are almost null; when the model has been estimated as being Form/Form, the bias and the MSE get the highest values, whereas the other cases (Form/Ref and Form/Ref) appear in a middle position between the two extremes. As expected, both the bias and the MSE decrease when increasing the sample size. Then, according to the quality of the measurement model (λ x =λ y =0.8 or 0.2), in the case of a measurement model with low loadings (λ x =λ y =0.2), the previous statement is true, with slightly worse results for bias. We would expect a worse results, but it seems harming that the measurement model does not interfere so much in the estimate of the inner model. With low loadings it appears a tendency to move down the estimation of β, i.e., to underestimate the path coefficient (i.e. negative bias). 3.3.3 Behavior of the bias and the MSE according to the random fluctuation of the manifest variables (s x =s y =0.05, 0.2, 1.0) Figure 5 shows that increasing the variance of the random fluctuations of the manifest variables implies an increase of the variance of the estimates, and hence the MSE, for all specifications Ref/Ref, Form/Form, Form/Ref and Ref/Form, and also an increase of the bias (however negatively); the latter meaning that high values of random fluctuation come out finding lower estimates of the “true=simulated” relation between constructs. Figure 5: Behavior of the bias and MSE according to the random fluctuation of the manifest variables Also, when the model is correct regarding the loadings (λ x =λ y =0.8), it appears a misspecification effect, following the same pattern as before (Ref/Ref corresponds to the lowest values of the estimates, Form/Form to the highest, and Form/Ref and Ref/Form in between). Regarding the MSE it appears also a clear misspecification effect when the random