Synthetic control and inference

Hahn, Jinyong; Shi, Ruoyao Article Synthetic control and inference Econometrics Provided in Cooperation with: MDPI – Multidisciplinary Digital Publishing Institute, Basel Suggested Citation: Hahn, Jinyong; Shi, Ruoyao (2017) : Synthetic control and inference, Econometrics, ISSN 2225-1146, MDPI, Basel, Vol. 5, Iss. 4, pp. 1-12, https://doi.org/10.3390/econometrics5040052 This Version is available at: https://hdl.handle.net/10419/195436 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ econometrics Article Synthetic Control and Inference Jinyong Hahn 1,† and Ruoyao Shi 2,*,† ID 1 Department of Economics, University of California Los Angeles, 8283 Bunche Hall, Los Angeles, CA 90095, USA; [email protected] 2Department of Economics, University of California Riverside, 3136 Sproul Hall, Riverside, CA 92521, USA *Correspondence: [email protected]; Tel.: +1-951-827-1494 † These authors contributed equally to this work. Academic Editors: In Choi and Ryo Okui Received: 27 October 2016; Accepted: 3 November 2017; Published: 28 November 2017 Abstract: We examine properties of permutation tests in the context of synthetic control. Permutation tests are frequently used methods of inference for synthetic control when the number of potential control units is small. We analyze the permutation tests from a repeated sampling perspective and show that the size of permutation tests may be distorted. Several alternative methods are discussed. Keywords: synthetic control; permutation test; symmetry JEL Classification: C12 1. Introduction Synthetic control method, proposed and discussed by Abadie and Gardeazabal (2003) and Abadie et al. (2010), is a very useful way of conducting comparative studies when exact matches are unavailable. Estimation of treatment effects usually takes the form of comparing outcomes between the treated unit and the control unit. Common sense suggests that, for the comparison to be meaningful, the control unit needs to be similar to the treated unit in the absence of the treatment in various dimensions. Such a requirement may not be satisfied in many observational studies. In some cases, availability of panel data makes such comparisons reasonable, the difference-in-differences method being a very well-known example. The difference-in-differences method requires a very specific set of assumptions, i.e., the common trend assumption, which may not be plausible for many applications. The synthetic control method offers a sensible generalization of the difference-in-differences. The synthetic control is a linear combination of the potential control outcomes, where the weights are manufactured by analyzing the pre-intervention outcomes. For the purpose of statistical inference with synthetic control, i.e., confidence interval and hypothesis testing, various versions of placebo tests are often adopted. The idea underlying the placebo tests is the usual permutation tests, where the critical value of a test statistic is computed under all possible permutations of the “treatment” assignments in the control units. The idea of permutation test is very intuitive and attractive. Applying the synthetic control method to every potential control unit presumably allows researchers to assess the distribution of a test statistic under the null hypothesis of no treatment effects, and the inference is seemingly exact in the sense that the burden of asymptotic approximation can be obviated. The purpose of this paper is very specific. We ask whether the permutation test is a reasonable idea in the context of the synthetic control method, and argue that the intuitive appeal of the permutation test is misplaced. The validity of permutation tests usually requires certain symmetry assumption, which is often violated in the context of synthetic control studies. Using Monte Carlo simulations, Econometrics 2017,4, 52; doi:10.3390/econometrics5040052 www.mdpi.com/journal/econometrics Econometrics 2017,4, 52 2 of 12 we document the size distortion of the permutation tests. We also discuss a few alternative methods of inference. Alberto Abadie kindly pointed out that the placebo test in synthetic control is often based on randomization inference idea, under which the symmetry restriction is built-in, while our analysis is predicated from the usual random sampling perspective, which leads to the violation of symmetry. This perspective is shared with an anonymous referee, who notes that (i) the synthetic control literature uses permutation tests in the context of design-based inference, and, as such, the permutation tests have exact size; (ii) the present article shows that permutation tests may not have correct size under a different mode of inference based on repeated sampling, although interpreting the permutation tests in the previous literature as tests based on repeated sampling would be incorrect; and (iii) the present article also proposes some alternatives that are valid in a repeated sampling setting. It would be useful to understand the exact mechanism through which the difference between the two perspectives manifests itself. The same referee points out that the present paper adopts a setting where T0→∞ , while the original litereature assumes fixed T0. 2. Placebo Test and Synthetic Control In this section, we provide a brief discussion of the placebo test in the context of the synthetic control method. We begin with an overview of the synthetic control, borrowing heavily from discussions in Abadie et al. (2010) and Doudchenko and Imbens (2016). We then move on to describe the placebo test, and point out the importance of the symmetry assumption. We argue that the symmetry assumption is violated in general for placebo tests using linear combinations of outcomes, such as synthetic control. We conclude this section that such violation should be expected in general even when a normalized version of the test statistic is adopted. We start with the overview of the synthetic control method. Consider a panel data with J+ 1 cross sectional units observed over the time periods t= 1, . . . , T . Units j= 1, . . . , J are the control units that receive the treatment in none of the time periods. The unit j= 0 receives no treatment in periods 1, . . . , T0 , and receives active treatment in time periods t=T0+ 1, . . . , T . For simplicity, we will often assume that T=T0+ 1. The outcome variable Yj,t is such that Yj,t=Yj,t(1) if the j th unit receives treatment in time t, and Yj,t=Yj,t(0)otherwise. Obviously, Yj,t=Yj,t(0),j=1, . . . , J;t=1, . . . , T, Y0,t=Y0,t(0),t=1, . . . , T0, Y0,t=Y0,t(1),t=T0+1, . . . , T. The idea underlying the synthetic control is that if there were some weights 1ˆ ω1 , . . . , ˆ ωJ such that Y0,t≈ J ∑ j=1 ˆ ωjYj,t(1) during the pre-intervention periods ( t= 1, . . . , T0 ). Then, ∑J j=1ˆ ωjYj,t can be used as a (synthetic) control for Y0,t during the post-intervention periods ( t=T0+ 1, . . . , T ). Abadie et al. (2010) and Doudchenko and Imbens (2016) discuss various methods of finding the ˆ ω ’s so that the requirement in Equation (1) is satisfied. We analyze the weights and the nature of approximation from the asymptotic 1 Doudchenko and Imbens (2016) also consider a slightly more general requirement Y0,t≈α+∑J j=1wjYj,t . This is a sensible way to enhance accuracy of synthetic control viewed as a point estimator. It also provides a link to the difference-in-differences estimator. Because our focus is on inferential aspects of the problem, we simplify notation and analysis by abstracting away from the intercept term. Econometrics 2017,4, 52 3 of 12 perspective where T0→∞ . Note that a special case of the estimator discussed by Abadie et al. (2010, p. 496) solves min ω1,...,ωJ ¯ Y0− J ∑ j=1 ωj¯ Yj!2 . (2) Under our interpretation, ¯ Yj=T−1 0∑T0 t=1Yj,tabove is an estimator of EYj,t. Consider the linear factor structure2as in Abadie et al. (2010): Yj,t=Yj,t(0)=αj+θt+γ0 jδt+ej,t,t=1, . . . , T0. (3) Suppose that θt , δt , and ej,t satisfy strict stationarity. Without loss of generality, we also assume that E[δt]= 0 and Eej,t= 0. We would have ¯ Yj→αj+E[θt] in probability as T0→∞ . Assuming that ω∗jsatisfies α0+E[θt]= J ∑ j=1 ω∗jαj+E[θt], (4) we can understand that the population version of the synthetic control ∑J j=1ω∗jYj,t is such that the difference Y0,T0+1(0)−∑J j=1ω∗jYj,T0+1(0)is designed to have a mean zero. Our T0→∞ asymptotic interpretation is not the only possible one. Doudchenko and Imbens (2016) provide an in-depth analysis of many possible methods. Our interpretation, however, is helpful for two reasons. First, it makes a concrete interpretation of ˆ ω s as estimates of some pseudo-parameter, say ω∗ ’s, along with analytic expressions of the ω∗ ’s, which makes it easy to understand the potential pitfalls of permutation methods afterwards. Second, it helps us to motivate alternative methods of inference exploiting time series variation. We now discuss how placebo tests can be used in the context of synthetic control. For this purpose, we first present a summary of the placebo tests/permutation tests. The tests are motivated to deal with the case where the number of the treated is small and the number of controls is relatively large. In order to focus on the salient feature of the tests, we will consider an extreme case and assume that there is only one treated unit. The basic intuition underlying the general placebo test can be gleaned by examining a standard textbook case of randomized treatments. Suppose that there is cross sectional data with J+ 1 units, where the units j= 1, . . . , J are the control units and the unit j= 0 receives the active treatment. A reasonable estimator of the treatment effect is the difference Y0−¯ Y , where Y0 is the outcome of the unit j= 0, and ¯ Y=J−1∑J j=1Yj denotes the average of the outcomes of the controls. Suppose that we are interested in testing whether the treatment had impact. Given that there is only one treated unit, the standard t -test comparing the difference of the mean outcomes is not applicable. On the other hand, common sense suggests that we may implement such a test by “assigning” each control unit to fictitious treatment. More precisely, one can estimate the empirical distribution of Yk−(J−1)−1∑j6=kYj for k= 1, . . . , J , and use it as if it were the distribution of the treatment effect under the null hypothesis.3 Implementation of the placebo test with synthetic control requires a bit more notation. First let ˆ ω=ˆ ω1, . . . , ˆ ωJ0 denote the estimator of ω∗=ω∗1, . . . , ω∗J0 . Although we will use the method of exact balancing later in our Monte Carlo simulations, we do not need to restrict ourselves to this particular estimator. For now, we can view ˆ ω as an output from a blackbox and let ω∗ denote its 2 Using the notation consistent with this paper, Equation (1) in Abadie et al. (2010) takes the form Yj,t(0)=λt+(θt−λt)0Zj+ γ0 jδt+ej,t , so the factor structure in Equation (3) of this paper is a special case of Equation (1) in Abadie et al. (2010), where Zj=1, δ1t=1 and γ1j=αj, i.e., it is a special case where the Zjdoes not exist and the first element of δtis time invariant. 3 Conley and Taber (2011), who proposed a similar test, cite Bertrand et al. (2004) when they discuss placebo tests. Abadie et al. (2010) reference many other papers that precede Bertrand et al. (2004). Econometrics 2017,4, 52 4 of 12 probability limit as T0→∞ . Second, let ˆ ω(−k) denote the outcome of the same blackbox except that we use the k th unit as the outcome of the treated unit, and Yj,t with j6=k as our control units. The placebo test then uses the empirical distribution of Yk,T0+1−∑j6=kˆ ω(−k) jYj,T0+1 for k= 1, . . . , J as if it were the distribution of the treatment effect under the null hypothesis of no treatment effect. If the estimated effect Y0,T0+1−∑J j=1ˆ ωjYj,T0+1 belongs to the extreme tails of the empirical distribution, it is understood to be the evidence that the null hypothesis is incorrect. In order to understand the size property of the placebo test, it helps to recall that the placebo test is a version of the permutation test, which requires for its validity what may be called the symmetry assumption. For review of this property, we will borrow the short discussion in Canay et al. (2017). 4 Suppose that a researcher observes a vector of observations X , whose joint distribution is P . The objective is to test whether P∈P0 , where P0 is a collection of probability distributions such that the distribution of X is equal to that of gX for every g in G , where G is a finite collection of transformations. The permutation test has the exact size if, for the test statistic T(X) , the critical value is taken from the distribution of T(gX) for every g in G . In the context of the placebo test above, one can understand Xto be the vector Y1, . . . , YJ, and gX to be the permutation of the Ys. We note that the symmetry is not mathematically obvious in the context of synthetic control. In order for the permutation test to be valid, it is necessary for the distribution of Y0,T0+1− ∑J j=1ω∗jYj,T0+1 and those of Yk,T0+1−∑j6=kω(−k) ∗jYj,T0+1 for k= 1, . . . , J to be identical. Even for the relatively simple model in Equation (3), the nature of the synthetic control is such that the symmetry does not naturally follow. Using the restriction in Equation (4), we may write Y0,T− J ∑ j=1 ω∗jYj,T=(θT−E[θT] ) 1− J ∑ j=1 ω∗j!+ γ0− J ∑ j=1 ω∗jγj!0 δT+ e0,T− J ∑ j=1 ω∗jej,T!. (5) Even if the first two terms on the right-hand side of Equation (5) were identically equal to zero over the permutations, we believe that the third term is not likely to satisfy the symmetry property. This is because we believe that under the further restriction that the e ’s have a finite variance, the term can be symmetric only when they are normally distributed. We show that normality is necessary if the distribution of the error term e0,T−ω0 ∗eT in Equation (5), where eT=e1,T, . . . , eJ,T0 , is to be symmetric up to normalization. 5 Suppose that e0,T , . . . , eJ,T are i.i.d., and their common distribution is such that the variance is finite and the characteristic function does not disappear. If ω is a nontrivial function of α s and γ s, then symmetry over the permutations requires that the marginal distributions of ek,T−∑j6=kω(−k) jej,T for k= 0, . . . , J should remain invariant over all possible ω(−k) s. Without loss of generality, we can focus on the distribution of e0,T−ω0eT , and conclude that the symmetry requires that there exists a random variable Y such that the distribution of e0,T−ω0eT is the same as that of cY for some scalar c . Because the standard deviation of e0,T−ω0eT is proportional to √1+ω0ω , we may without loss of generality take c=√1+ω0ω . This implies that the distribution of ω0eT only depends on ω0ω . In other words, for ω6=˜ ω such that ω0ω=˜ ω0˜ ω , the distribution of ω0eT is identical to that of ˜ ω0eT . In particular, let all components of ˜ ω be zero except for the first one. Then, the distribution of ω0eT is identical to that of ˜ ω0eT=˜ ω1e1,T=√˜ ω0˜ ωe1,T . This implies that ej,T should have a stable distribution. 6 Because the only stable distribution with a finite variance is the normal distribution, we should conclude that normality is a necessary condition of the symmetry (up to normalization). Note that the third term in Equation (5) arises in an ideal 4 The same test was first discussed by Hoeffding (1952), which is a generalization of the randomization test proposed by Fisher (1949). 5 We are using the fact that the symmetry implies the equality of marginal distributions, and therefore, the lack of equality of marginal distributions is a sufficient condition for violation of symmetry. 6See Nolan (2015), or Wikipedia Contributors (2017). Econometrics 2017,4, 52 5 of 12 situation where the weights ω do not need to be estimated and the first two terms completely disappear. Our analysis suggests that even if we normalize the third term by its standard deviation, the symmetry requires normal distribution. The necessity of normality assumption is about any linear combination so it applies a fortiori to synthetic control. 3. Monte Carlo The discussion at the end of the previous section casts doubt on the placebo test, even for the simple case where the first two terms in Equation (5) can be ignored. In order to understand the roles that the first two terms may play, we adopt Monte Carlo simulations. We try to find data generating processes (DGPs hereafter) that generate a large amount of size distortions. This is helpful in understanding the potential problem of the placebo test from the uniformity perspective; after all, the mathematical definition of the “size” of a test is the maximum probability of rejection under the null, and here the null hypothesis is a composite hypothesis where the only requirement on the DGP is that the treatment effect is zero, which allows many possibilities on the terms in Equation (5). For this purpose, we found it most convenient to work with the first two terms in Equation (5), although we acknowledge that there may be other important sources of size distortion that we have not explored. Since the last paragraph of Section 2 showed that normalization does not abate the symmetry requirement, we examine the importance of the first two terms in Equation (5) using a more natural statistic. The version of the synthetic control that we use in the Monte Carlo is the method of exact balancing, the population version of which minimizes ∑J j=1ω2 j subject to E[Y0,t(0)] =∑J j=1ωjEYj,t(0)and 1 =∑J j=1ωj.7 The method of exact balancing may not be an ideal version of the synthetic control, but it reflects a certain ambiguity in the method of synthetic control. In the factor model in Equation (3), it is impossible to find weights ω such that Y0,t=∑J j=1ωjYj,t for every t= 1, . . . , T0 , if T0 is large enough, as long as ej,t is continuously distributed. In other words, the condition (2) in Abadie et al. (2010) is incompatible with the factor model unless Var ej,t= 0. The assumption Var ej,t= 0 has at least two implications. 8 First, the weights ωj can be estimated without error with sufficiently large T0 . Second, the distribution of the permutation test would have the point mass at zero, and as such, there is no reason to conduct any test. Both implications are questionable. In any case, under the assumption Var ej,t= 0, the weights can be estimated (without error) by the method of least squares that minimizes ∑T0 t=1Y0,t−∑J j=1ωjYj,t2 . If the assumption Var ej,t= 0 is violated, the method of least squares would be subject to a version of measurement error problem; the true regressor there is αj+θt+γ0 jδt in Equation (3), and the Yj,t plays the role of a regressor with measurement error ej,t . 9 Note that such a problem is avoided by the method of exact balancing. We consider the method of exact balancing in this section not because it is necessarily an ideal version of the synthetic control, but because it is a convenient way of examining the impact of the first two terms in Equation (5). As mentioned at the beginning of this section, our analysis at the end of the previous section suggests that the placebo test may have a problem even when these two terms are dismissed, and the purpose of our Monte Carlo exercise is to focus on the potential impact of these two terms. 7Abadie et al. (2010) also impose the positivity restriction, i.e., ωj≥0 for all J. 8 It is straightforward to prove that under stationarity assumption, the only model that allows the synthetic controls to trace the trajectory of the outcome for the treated (i.e., Y0,t=∑J j=1ωjYj,tfor some ω) is a linear factor model with Var(ej,t) = 0. 9See Ferman and Pinto (2017) for related discussion on the bias of the synthetic control estimator. Econometrics 2017,4, 52 6 of 12 For our Monte Carlo analysis, we adopted a simplified version of the factor model in Equation (3) such that (i) θt∼ N (0, 1) ; (ii) δt is a scalar; (iii) δt∼ N 0, σ2 δ ; (iv) ej,t∼ N 0, σ2 e is i.i.d. over j and t . In matrix notation, our estimator ˆ ωsolves min ω0ωs.t. (i) ¯ Y0ω=¯ Y0, (ii) `0ω=1, (6) where ` is a vector of ones. Because EYj,t=αj , we can see that the population counterpart ω∗ solves min ω0ωs.t. (i) α0ω=α0, (ii) `0ω=1, where α=α1, . . . , αJ0. We now write Y0,T− J ∑ j=1 ˆ ωjYj,T=A+B+C+D, where A= J ∑ j=1ω∗j−ˆ ωjαj B=θT 1− J ∑ j=1 ω∗j! | {z } B(i) +θT J ∑ j=1ω∗j−ˆ ωj | {z } B(ii) C= γ0− J ∑ j=1 ω∗jγj!0 δT | {z } C(i) + J ∑ j=1ω∗j−ˆ ωjγj!0 δT | {z } C(ii) D= e0,T− J ∑ j=1 ω∗jej,T! | {z } D(i) + J ∑ j=1ω∗j−ˆ ωjej,T! |{z } D(ii) . Note that the term B(i) is equal to 0 by design here, although it can be in principle different from 0 depending on the DGP and the estimator chosen. We speculate that the placebo test is used in the hope that (a) Y0,T−∑J j=1ˆ ωjYj,T is dominated by the term D(i) above; (b) the four terms A, B(ii), C(ii) and D(ii) above, which reflect the noise of estimating ω∗ by ˆ ω , are ignorable; and (c) the two terms C(i) and D(i) more or less satisfy the symmetry property. We argued in the previous section that the term D(i) is likely to violate the symmetry property. In order to assess the impacts of other terms, we consider the following variations in DGPs: 1. Vary the values of α ’s such that (a) none of the components of ω∗ dominates; (b) only two of the elements are non-zero. 2. Vary the values of γ ’s such that the unbalanced unobservable factors C(i) (a) disappear; and (b) are present. 3. Vary T0such that the estimation errors in the weights are (a) prominent; and (b) negligible. Combinations of the first two variations give us four different DGPs, shown as DGP No. 1 to No. 4 in Table 1. Econometrics 2017,4, 52 7 of 12 Table 1. Data Generating Processes (DGPs) that generate size distortion. DGP No. α’s γ’s Variations 1α0= (2J+1)/3, α1=1, ··· ,αJ=Jγ0=γ1=··· =γJ=0 1(a), 2(a) 2α0= (2J+1)/3, α1=1, ··· ,αJ=Jγ0=2, γ1=··· =γJ=1 1(a), 2(b) 3α0=5/3, α1=1, α2=2, α3=··· =αJ=0γ0=γ1=··· =γJ=0 1(b), 2(a) 4α0=5/3, α1=1, α2=2, α3=··· =αJ=0γ0=2, γ1=··· =γJ=1 1(b), 2(b) We considered two versions of the placebo tests: the first one is what might be called a feasible version of the test. Formally, for j= 0, 1, . . . , J , let Yj be a T× 1 vector of outcomes for the j th control unit, let Y= (Y1 , . . . , YJ) , and let Y−j be a T×(J− 1 ) matrix that deletes the j th column from Y . Then, S(Y0,Y)≡Y0,T− J ∑ j=1 ˆ ω∗jYj,T. (7) Similar to Equation (6), define the leave-one-out synthetic control weights ˆ ω−j for the j th control unit as a solution to min ω0ωs.t. (i) ¯ Y0 −jω=¯ Yj, (ii) `0ω=1, (8) where ¯ Y−j is to delete the j th element from ¯ Y . We likewise define the population counterpart ω∗−j as a solution to min ω0ωs.t. (i) α0 −jω=αj, (ii) `0ω=1. For j= 1, . . . , J and k6=j , let ˆ ω−j,k be the element in ˆ ω−j that corresponds to the k th control unit. In addition, define ˆ ω−j,j≡0 for j=1, . . . , J. Then, for j=1, . . . , J, we can compute SYj,Y−j≡Yj,T− J ∑ k=1 ˆ ω−j,kYk,T. (9) Let S(1) , . . . , S(J) be the order statistics of S(Yj , Y−j) ’s. We reject H0 if S(Y0 , Y)>S(J(1−α 2)) or S(Y0,Y)<SJα 2. The second test is an infeasible version of the test, which is identical to the first test, except that we use the true value of ω∗, i.e., Strue(Y0,Y)≡Y0,T− J ∑ j=1 ω∗jYj,T, Strue(Yj,Y−j)≡Yj,T− J ∑ k6=j ω∗−j,kYk,T, and we reject H0if Strue(Y0,Y)>Strue,(J(1−α 2)) or Strue(Y0,Y)<Strue,Jα 2. For each DGP, we try T0∈ { 40, 80, 400, 800 } , J∈ { 20, 40, 80 } and σ2 e= 0.1. For all designs, we set the level of the tests to be α=10%, and the number of Monte Carlo runs to be 1000. The results are summarized in Table 2. 10 We see size distortions in Table 2, especially DGP No. 2 and No. 4. The size distortion there cannot be attributed to the noise of estimating ω . First, the problem persists even as T0 approaches unrealistically large values. Second, the size distortion is similar over the feasible and infeasible versions of the test. We suspect that the problem is a fundamental problem 10 We set θt∼N(0, 1) in Table 2. We also considered the case where θt= 0. Although the results for this case are not reported here in the paper, they were qualitatively similar to the θt∼N(0, 1) case. They are available upon request. (When the adding-up constraint was imposed, the two cases gave the same results. Without the adding-up constraint, these two specifications give slightly different results.) Econometrics 2017,4, 52 8 of 12 that may have something to do with the violation of symmetry. (An anonymous referee pointed out that DGPs No. 2 and No. 4 cannot produce synthetic controls that approximate the trajectory of the outcome for the treated, and that synthetic controls should not be applied in those settings.) Table 2. Null rejection rates of permutation tests. DGP Jσ2 δ Estimated Weights True Weights T0=40 T0=80 T0=400 T0=800 T0=40 T0=80 T0=400 T0=800 1 0.1 0.076 0.067 0.082 0.090 0.074 0.070 0.079 0.091 J=20 1 0.076 0.067 0.082 0.090 0.074 0.070 0.079 0.091 10 0.076 0.067 0.082 0.090 0.074 0.070 0.079 0.091 0.1 0.079 0.080 0.080 0.093 0.079 0.090 0.084 0.096 J=40 1 0.079 0.080 0.080 0.093 0.079 0.090 0.084 0.096 10 0.079 0.080 0.080 0.093 0.079 0.090 0.084 0.096 0.1 0.100 0.087 0.113 0.083 0.094 0.087 0.114 0.085 J=80 1 0.100 0.087 0.113 0.083 0.094 0.087 0.114 0.085 10 0.100 0.087 0.113 0.083 0.094 0.087 0.114 0.085 2 0.1 0.193 0.178 0.175 0.190 0.174 0.176 0.174 0.190 J=20 1 0.568 0.530 0.559 0.529 0.557 0.518 0.561 0.531 10 0.838 0.843 0.837 0.837 0.839 0.828 0.838 0.838 0.1 0.193 0.219 0.198 0.208 0.200 0.213 0.193 0.207 J=40 1 0.561 0.623 0.586 0.589 0.556 0.593 0.588 0.583 10 0.809 0.853 0.863 0.844 0.832 0.858 0.854 0.846 0.1 0.238 0.253 0.245 0.230 0.239 0.245 0.241 0.233 J=80 1 0.617 0.623 0.631 0.619 0.610 0.629 0.624 0.622 10 0.865 0.870 0.870 0.866 0.861 0.894 0.867 0.868 3 0.1 0.115 0.123 0.110 0.125 0.114 0.113 0.109 0.123 J=20 1 0.115 0.123 0.110 0.125 0.114 0.113 0.109 0.123 10 0.115 0.123 0.110 0.125 0.114 0.113 0.109 0.123 0.1 0.152 0.151 0.149 0.153 0.153 0.154 0.146 0.154 J=40 1 0.152 0.151 0.149 0.153 0.153 0.154 0.146 0.154 10 0.152 0.151 0.149 0.153 0.153 0.154 0.146 0.154 0.1 0.180 0.168 0.173 0.153 0.178 0.164 0.166 0.153 J=80 1 0.180 0.168 0.173 0.153 0.178 0.164 0.166 0.153 10 0.180 0.168 0.173 0.153 0.178 0.164 0.166 0.153 4 0.1 0.201 0.219 0.193 0.214 0.207 0.213 0.195 0.209 J=20 1 0.536 0.518 0.525 0.522 0.533 0.520 0.525 0.520 10 0.837 0.827 0.817 0.814 0.841 0.809 0.824 0.812 0.1 0.238 0.253 0.249 0.253 0.241 0.268 0.239 0.249 J=40 1 0.549 0.597 0.583 0.579 0.552 0.576 0.583 0.582 10 0.809 0.842 0.856 0.852 0.828 0.843 0.859 0.848 0.1 0.287 0.293 0.300 0.275 0.279 0.301 0.297 0.278 J=80 1 0.610 0.623 0.655 0.629 0.608 0.625 0.653 0.634 10 0.861 0.871 0.874 0.860 0.866 0.880 0.865 0.866 Our Monte Carlo analysis indicates that the placebo test does have the size distortion problem. The results in Table 2suggest that the size problem is potentially bigger in DGPs No. 2 and No. 4. DGPs No. 2 and No. 4 differ from No. 1 and No. 3 in that the γ ’s are nonzero and the aggregate shock δt plays a role as a consequence. Therefore, it is of interest to investigate further sources of asymmetry. For this purpose, we revisit the decomposition in Equation (5) of Y0,T0+1(0)−∑J j=1ω∗jYj,T0+1(0) , assuming that the first and second terms in the factor model in Equation (3) are not present:11 11 This can be done by assuming that α0=∑J j=1ωjαjand θt=0.