A Meta-Analysis of the Generalization of the Reliability of State/Trait Depression Inventory Scores

María Guillot-Valdés, Alejandro Guillén-Riquelme, and Gualberto Buela-Casal 476 Depression is one of the biggest public health problems worldwide (Schürmann & Margraf, 2018). Around 4.2% of the global population suffered from depression in 2015 (World Health Organization, 2017). The costs of the disorder are very high, not only in economic terms but also with regard to social and familial aspects and leading to the need for time off work (more than 70% of sick leave was for this reason in 2010; Castellón Leal et al., 2016). Depression also has associations with quality of life (Becker et al., 2018), with high levels of disability (Aguilera et al., 2019) and with obesity (Mulugeta et al., 2019). When the symptoms are very serious, depression can even lead to suicide (Brailovskaia et al., 2019; Castellón Leal et al., 2016; Teismann et al., 2018). Low mood and anhedonia are the most important symptoms in a depressive disorder. However, the disorder also includes physical symptoms (fatigue, insomnia, weight gain, etc.), cognitive symptoms (loss of concentration, suicidal thoughts, feelings of guilt, etc.), and motor symptoms (psychomotor agitation or retardation; American Psychiatric Association, 2013). The symptoms are also characterized by the differential estimation of frequency and intensity in their presentation. Among the explanatory models of mental disorders is the Spielberger model, with its distinction between state and trait. State is considered a transitory condition where levels of depression are detected at the time of the evaluation, while trait refers to individual differences in susceptibility to the disorder and to perceiving stimuli as depressive. The state/trait model has been adapted to depression disorder, referring to it as a dimensional construct, that is, depression would be defi ned within a continuum. According to Agudelo et al. (2007), two aspects of depression can be evaluated: the frequency (trait) and the degree of involvement (state). Thus, it could be said that the presence of depressive traits increases the probability of the occurrence of depressive states and a person’s vulnerability to the disorder. Hence, the State/Trait Depression Questionnaire was developed (ST-DEP; Spielberger, 1995; Spielberger & Reheiser, 2009; Spielberger & Ritterband, 1996; Spielberger et al., 2002a, 2002b). This test was adapted to several languages (i.e. Spanish, Portuguese, German...). In the development of the Spanish adaptation, only two of the items were kept, creating the rest of ISSN 0214 - 9915 CODEN PSOTEG Copyright © 2020 Psicothema www.psicothema.com A Meta-Analysis of the Generalization of the Reliability of State/Trait Depression Inventory Scores María Guillot-Valdés, Alejandro Guillén-Riquelme, and Gualberto Buela-Casal Universidad de Granada Abstract Resumen Background: Depression is a disorder that is highly prevalent nowadays. Within the dimensional explanatory model of depression, the State/Trait Depression Inventory was developed. Its objective is to identify the degree of affectation (state) and the frequency of occurrence (trait) of the affective component of depression. The instrument has proven reliable and comprises two factors in its structure: state and trait, with two euthymia and dysthymia subscales in each one. The objective of this meta-analysis is to fi nd an average alpha for the questionnaire. Method: A bibliographical search was conducted on Web of Science and Scopus. Forty-fi ve articles were selected. Results: The alpha ranges from .84 to .89 for all factorizations, and in most studies a bifactorial structure of state and trait depression was found. Conclusions: The State/Trait Depression Inventory is a reliable and suitable instrument for measuring depression. Keywords: Depression, meta-analysis, reliability, Cronbach’s alpha, STDEP. Meta-Análisis de Generalización de la Fiabilidad de las Puntuaciones del Inventario de Depresión Estado/Rasgo. Antecedentes: la depresión es un trastorno de alta prevalencia en la actualidad. Dentro del modelo explicativo dimensional de la depresión, se desarrolló el Inventario de Depresión Estado/Rasgo, cuyo objetivo es identifi car el grado de afectación (estado) y la frecuencia de ocurrencia (rasgo) del componente afectivo de la depresión. Este instrumento ha demostrado ser fi able y poseer dos factores en su estructura: estado y rasgo con dos subescalas eutimia y distimia en cada uno. El objetivo de este meta-análisis es hallar un alfa medio de las puntuaciones cuestionario. Método: se realizó una búsqueda bibliográfi ca en Web of Science y Scopus. Se seleccionaron 45 artículos. Resultados: el alfa oscila entre 0,84 y 0,89 para todas las factorizaciones, y en la mayoría de los estudios se halló una estructura bifactorial de depresión estado y rasgo. Conclusiones: el Inventario de Depresión Estado/Rasgo es un instrumento fi able y adecuado para medir la depresión. Palabras clave: depresión, meta-análisis, fi abilidad, alfa de Cronbach, IDER. Psicothema 2020, Vol. 32, No. 4, 476-489 doi: 10.7334/psicothema2020.106 Received: April 5, 2020 • Accepted: June 11, 2020 Corresponding author: María Guillot-Valdés Centro de Investigación Mente, Cerebro y Comportamiento (CIMCYC) Universidad de Granada 18011 Granada (Spain) e-mail: [email protected] A Meta-Analysis of the Generalization of the Reliability of State/Trait Depression Inventory Scores 477 the questionnaire from scratch. This Spanish version was the one that provided the basis for other adaptations to countries in Latin America or Portugal. The Spanish version was called State/ Trait Depression Inventory (in Spanish Inventario de Depresión Estado/Rasgo, Spielberger et al., 2008). Its main objective is to identify the presence of negative affects (dysthymia) and the absence of positive affects (euthymia) in state and trait depression components. This offers four possible combinations that can be evaluated: state euthymia, trait euthymia, state dysthymia, and trait dysthymia (Agudelo, 2009). This distinction between state and trait makes the instrument more sensitive to small changes in the level of involvement, given their transitory nature and makes it easier to apply the instrument to a non-clinical population (Agudelo et al., 2005). Structural validity and score’s reliability have been corroborated in various studies around the world. Most of them coincide with respect to the existence of two factors in the structure of the questionnaire: state (S-DEP) and trait (T-DEP), with two subscales of euthymia and dysthymia in each one. Notably, its use should be considered within the evaluation of the affective component of depression but not as a diagnostic test in itself (Agudelo et al., 2014). Although the ST-DEP has adequate psychometric properties in different populations, a generalization of its score’s reliability has not been made. Therefore, this study aims to establish the reliability of the questionnaire’s scores for various samples of adults (both clinical and general population) from around the world as well as to know which factors infl uence in its reliability. The hypothesis is that the questionnaire’s scores will have high reliability for both the trait and state scales, with neither being more reliable than the other. Furthermore, the score’s reliability of the unifactorial and tetrafactorial models will be lower than that of the bifactorial model. Method Literature sampling Search strategy For its preparation, the recommendations of Rubio-Aparicio et al. (2018) were followed, as well as the PRISMA guidelines (Liberati et al., 2009). The identifi cation of key scientifi c articles was performed by searching the Web of Science and Scopus databases with the search terms ‘Inventario de depresión estado rasgo’, OR ‘ider’, OR “State Trait Depression’, OR ‘st/dep’. As the instrument appears with different names in the literature and its use is frequently not refl ected in the titles or summaries of articles, complementary strategies were used. Specifi cally, in addition to the database search, a more exhaustive manual search was carried out via the citations of the original ST-DEP documents and in those works where their psychometric properties were refl ected. For this, cites from creation documents or cultural adaptations of the questionnaire were searched both in Scopus and in Google Scholar. In this case, cites were searched from original article and from its adaptations (Spanish, Portuguese, Cuban, Czech, Korean and German) as well as cultural adaptations of the Spanish version (i.e., Chile, Colombia and Mexico). Additionally, it was searched in the citations of articles whose fundamental objective was the development or study of the psychometric properties of the ST-DEP. This procedure will allow including articles from specifi c journals but without impact. This fact will help to mitigate the gray literature bias. This search was made during June 2019 and actualized in May 2020. Study selection: Inclusion and exclusion criteria The articles included in the review were required to meet the following criteria: they must be empirical studies published in scientifi c journals (with no limit on year of publication); the participating samples must be adults and adolescents of any gender and culture; the articles must be available in English, Spanish, or German (an article in Czech that provided suffi cient data for analysis in its English abstract was included); they must report Cronbach’s alpha—whether global alpha, total state alpha, or total trait alpha, either together or divided by euthymia and dysthymia. Works with children’s samples (under 11 years old) were excluded since the inventory is not designed for this population. A total of 977 studies were initially obtained. All articles that did not meet these inclusion criteria were excluded, as well as those that, after analysis, did not provide relevant data for the present review. This fi nally left 45 relevant articles (see Figure 1). Instruments All analysis included were performed using the R software and, specifi cally, the metafor package (Viechtbauer, 2010). Procedure After articles selection, the following data were extracted: 1) author, 2) year of publication, 3) country/ies in which the study was conducted, 4) sample size, 5) type of sample (adolescents, adults, elderly, and clinical samples), 6) percentage of women, 7) mean age and standard deviation, 8) minimum and maximum age, 9) Cronbach’s alpha for each of the factorizations, and 10) means and standard deviations for each of the factorizations. The coding process of half of the articles was made for all the variables by to independent researchers. The discrepancies were resolved by consensus. With all variables a Cohen’s Kappa was obtained. The mean was .93 (SD = .20), ranked between .68-1. Finally, the Strengthening the Reporting of Observational Studies in Epidemiology methodological quality survey was applied to check the bias risk of the selected studies. This survey is made up of 22 elements that evaluate aspects related to the presentation of relevant data for the study’s replication. For its correction, one point was given per item that was properly refl ected. The average methodological quality of the studies was 18.13 (SD = 0.83, range = 17–20). Data analysis To estimate the average alpha of all the selected studies, their non-normality was assumed. The alpha coeffi cient in each study was transformed using the formula presented by Bonett (2010) to achieve normalization. After obtaining the average transformed score, it was transformed back to alpha to make it more easily interpretable. A random effects model was used with the DerSimonian–Laird estimation method and the values María Guillot-Valdés, Alejandro Guillén-Riquelme, and Gualberto Buela-Casal 478 were weighted by the inverse of the studies’ variance. To check heterogeneity in the results the Q statistics and the I2 were used, interpreting the value upper than .25 as a small variability, upper than .5 medium, and upper than .75 as high variability. The moderator analysis was performed using metaANOVA for the categorical variables, and a multiple metaregresion for continuous variables. The publication bias analysis was performed using the regression test of Egger. Results Description of studies After the search and screening, a total of 45 studies were selected. The total sample was 19,582 with an average of 338 participants per study. Regarding the sample types, twenty-three studies focused on the general population, fourteen on university students, four on adolescents, three on both, and a single study consisted of a clinical sample (depressive patients). Around 30% were from Germany, 13% from Spain and Austria, 11% from Colombia and Peru, 6% from Mexico, 4% from China, United States and Chile, and 2% from the Czech Republic, China and South Korea. Most of the studies (23) proposed a bifactorial structure of state and trait depression; while seventeen proposed a four-factor solution in which state and trait depression included the euthymia and dysthymia dimensions. Finally, fi ve works provided the total alpha (see Table 1). Internal consistency of different factorizations As can be seen in Table 1, the alpha ranged between .85 and .89 for all factorizations. The homogeneity of the results was analysed. The signifi cance associated with Q was always less than .01. So, it can be affi rmed that there was a high variability. Likewise, two factorizations of possible publication bias were observed, according to the signifi cance obtained by the Egger test (see Table 2). It is important to interpret these results with caution. For more details of these results, see the funnels plots (Figures 2 to 8). Documents obtained through formal sources: Web of Science n = 307 Documents obtained through formal sources: Scopus n = 31 Documents obtained through other sources: n = 639 Total studies containing the descriptors in the title, abstract and/or keywords (in English, Spanish or German) n = 977 Excluded: n = 932 Not related to the theme or duplicates or do not provide the alpha (neither global nor in any of its dimensions): n = 919 Not articles: n = 13 Total of studies included in the review after full reading: n = 45 Figure 1. Flow diagram of selected studies A Meta-Analysis of the Generalization of the Reliability of State/Trait Depression Inventory Scores 479 Figures 9 and 10 refl ect the results of the alpha scores in depression–state and depression–trait for the bifactorial model. It was observed that all the studies had a score’s reliability greater than .75 (except in two samples). Figures 11 and 12 present the alpha scores of the state subscale understood as euthymia and dysthymia. In this case, most studies achieved an alpha greater than .70. Similarly, in Figures 13 and 14, the alpha subscale scores for trait (euthymic and dysthymia) are presented. Again, almost all of them exceed .70 (except in two samples). Finally, Figure 15 shows the global alpha of the ST-DEP (greater than .8 in four studies and lower in one study). Moderator analysis A moderator analysis was performed for the state and trait factorizations, since they are the most used and mainly supported by the theory. For the state scale, the country of the version was signifi cant (F(8;12) = 3.86; p = .018), where the values ranged from .74 (in samples from Peru; the only country below .8), up to .95 (in Chinese samples). For the sample type in both adults and adolescents, the alpha was .86. The mean age was neither signifi cant nor the percentage of women (p > .1). The standard deviation of the samples had a signifi cant effect on the mean reliability (p < .001). In the trait factorization, the country was also signifi cant (F(10;27) = 5.81; p < .001), with a range between .73 (for Peruvian samples, there being no other countries with reliability lower than .8) and .94 (for United States). The type of sample was signifi cant since in adults was .86, while in adolescents was .68 (however, there were only two studies with this factorization in adolescents’ samples). Finally, neither the mean age of the sample nor the Table 1 Average alpha according to the factorizations used Factorization k Average alpha SE 95% CI t Q I2 Unifactorial 5 .85 .13 [.77,.90] 12.7*** 33.36*** 91.35 Bifactorial State Trait 23 41 .89 .88 .12 .08 [.86,.91] [.86,.90] 18.73*** 27.40*** 1255.12*** 1221*** 98.25 96.72 Tetrafactorial Stateeuthymia State-dysthymia Trait-euthymia Trait-dysthymia 23 25 25 26 .88 .87 .85 .84 .12 .14 .11 .11 [.85,.91] [.83,.90] [.81,.88] [.80,.87] 16.89*** 14.75*** 17.44*** 16.98*** 1115.6*** 1632.04*** 815.18*** 938.51*** 98.03 98.53 97.06 97.34 Table 2 Analysis of publication bias based on the factorizations used Factorization Egger Test T p Unifactorial 4.37 .02 Bifactorial State Trait 2.19 1.36 .04 .18 Tetrafactorial State-euthymia State-dysthymia Trait-euthymia Trait-dysthymia 1.07 0.69 0.63 -0.12 .30 .53 .54 .90 0.163 0.122 0.082 0.041 0 1.5 2 2.5 3 Standard Error Transformed coefficient alpha Figure 2. Funnel plot for Depression-State scores María Guillot-Valdés, Alejandro Guillén-Riquelme, and Gualberto Buela-Casal 480 0.215 0161 0108 0.054 0 1.51 2 2.5 3 3.5 Standard Error Transformed coefficient alpha Figure 3. Funnel plot for Depression-Trait scores 1.5 2.5 3.523 0.173 0.129 0.086 0.043 0 Transformed coefficient alpha Standard Error Figure 4. Funnel Plot for State/Euthymia scores A Meta-Analysis of the Generalization of the Reliability of State/Trait Depression Inventory Scores 481 1.5 2 2.5 3 3.51 0.173 0.129 0.086 0.043 0 Standard Error Transformed coefficient alpha Figure 5. Funnel plot for State/Dysthymia scores 00.202 0.152 0.101 0.051 1 1.5 2 32.5 Transformed coefficient alpha Standard Error Figure 6. Funnel pot for Trait/Euthymia scores María Guillot-Valdés, Alejandro Guillén-Riquelme, and Gualberto Buela-Casal 482 percentage of women were signifi cant (p > .1). The standard deviation of the samples had a signifi cant effect on the mean reliability (p = .001). Discussion The objectives of this study were to provide an average alpha of the ST-DEP through the analysis of several studies and analyze its sources of variability. In most of the 45 included studies, Cronbach’s alpha offers values greater than .80, and all the average alphas are greater than .84. These values are indicators of adequate reliability, according to Charter (2003). Therefore, it can be stated that the ST-DEP is a questionnaire with adequate score’s reliability for the evaluation of depression. The high accumulated sample size of the selected works allows increasing confi dence in the results. Furthermore, the methodological quality of the papers included was high in all cases, suggesting that the conclusions were derived from objective, accurate and verifi able data. The bifactorial structure is the most common in the ST-DEP. This factorization is based on the theoretical state and trait model used on the development of the ST-DEP (Spielberger et al., 2008). The effi cacy and usefulness of these scales lies not only in their use in non-clinical samples but also when clinically relevant scores are not reached, which can be useful in the applied setting (Agudelo et al., 2014). The measurements of stable traits and specifi c states of depression help to understand the debate on the dimensionality of 0.202 0.152 0.101 0.051 0 Standard Error Transformed coefficient alpha 1 1.5 2 32.5 3.5 Figure 7. Funnel plot for Trait/Dysthymia scores 0.192 0.096 0 Standard Error Transformed coefficient alpha 1.2 1.4 1.6 1.8 2 2.21 Figure 8. Funnel plot for ST-DEP global scores A Meta-Analysis of the Generalization of the Reliability of State/Trait Depression Inventory Scores 483 Figure 9. Forest plot of Cronbach’s alpha for Alpha Scores in Depression-State Figure 10. Forest plot of Cronbach’s alpha for Alpha Scores in Depression-Trait María Guillot-Valdés, Alejandro Guillén-Riquelme, and Gualberto Buela-Casal 484 Figure 11. Forest plot of Cronbach’s alpha for Alpha Scores in State/Euthymia Figure 12. Forest plot of Cronbach’s alpha for Alpha Scores in State/Dysthymia