Closing the achievement gap in mathematics: Evidence from a remedial program in Mexico City

Gutiérrez, Emilio; Rodrigo, Rodimiro Article Closing the achievement gap in mathematics: Evidence from a remedial program in Mexico City Latin American Economic Review Provided in Cooperation with: Centro de Investigación y Docencia Económica (CIDE), Mexico City Suggested Citation: Gutiérrez, Emilio; Rodrigo, Rodimiro (2014) : Closing the achievement gap in mathematics: Evidence from a remedial program in Mexico City, Latin American Economic Review, ISSN 2196-436X, Springer, Heidelberg, Vol. 23, Iss. 1, pp. 1-30, https://doi.org/10.1007/s40503-014-0014-2 This Version is available at: https://hdl.handle.net/10419/108958 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. http://creativecommons.org/licenses/by/2.0/ ORIGINAL RESEARCH Closing the achievement gap in mathematics: evidence from a remedial program in Mexico City Emilio Gutie ´rrez •Rodimiro Rodrigo Received: 11 August 2013 / Revised: 31 July 2014 / Accepted: 7 October 2014 / Published online: 11 November 2014 ÓThe Author(s) 2014. This article is published with open access at Springerlink.com Abstract This paper evaluates the impact of an intervention targeted at marginalizedlow-performance students in public secondary schools in Mexico City. The program consisted in offering free additional math courses, taught by undergraduate students from some of the most prestigious Mexican universities, to the lowest performance students in a set of marginalized schools in Mexico City. We exploit the information available in all students’ (treated and not treated by the program) transcripts enrolled in participating and non-participating schools. Before the implementation of the program, participating students lagged behind non-participatingones by more than a half base point in their GPA (over 10). Using a difference-in-differences approach, we find that students participating in the program observed a higher increase in their school grades after the implementation of the program, and that the difference in grades between the two groups decreases over time. By the end of the school year (when the free extra courses had been offered, on average, for 10 weeks), participating students’ grades were not significantly lower than non-participating students’ grades. These results provide some evidence that short and low-cost interventions can have important effects on student achievement. Keywords Impact evaluation Remedial education programs Difference-indifferences estimators JEL Classification C23 I21 I28 O15 E. Gutie ´rrez (&) Centro de Investigacio ´n Econo ´mica, Instituto Tecnolo ´gico Auto ´nomo de Me ´xico, ITAM, Camino a Sta. Teresa #930, He ´roes de Padierna, 10700 Me ´xico DF. CP, Mexico e-mail: [email protected] R. Rodrigo Department of Economic Productivity, Secretarı ´a de Hacienda y Cre ´dito Pu ´blico, Palacio Nacional, 06600 Me ´xico DF. CP, Mexico e-mail: [email protected] 123 Lat Am Econ Rev (2014) 23:14 DOI 10.1007/s40503-014-0014-2 1 Introduction While the evidence on the efficacy of different programs aimed at increasing a population’s schooling accumulates, the evidence on the effectiveness of programs targeted at improving school quality is not as vast. Existing evidence has measured the impact of conditional transfer programs (Behrman et al. 2011), information provision (Jensen 2010), school construction (Duflo 2001), and other school resources, such as teachers (Banerjee et al. 2004) and text books (Glewwe et al. 2002) on school enrollment or schooling years. However, studies finding that particular interventions increase school attendance or schooling do not always find evidence that these programs improve students’ performance on standardized tests. In other words, ‘‘students often seem not to learn anything in the additional days that they spend at school’’ (Banerjee et al. 2008). Such is the case of Mexico, where despite successful efforts to increase schooling (e.g., Oportunidades), children’s cognitive skills have arguably increased by very little or nothing (Behrman et al. 2011). In addition, interventions aimed at improving school inputs, such as text books and number of teachers, find very little or no effects on students’ performance (Banerjee et al. 2004; Glewwe et al. 2002). Recently, more attention has been put into the design and evaluation of programs aimed at improving learning. Examples of such interventions include Conditional Cash Transfers programs like the one studied by (Barham et al. 2013) in Nicaragua, and pedagogical strategies aimed at better matching teaching to students’ learning needs, like the one studied in (Banerjee et al. 2008) and in this paper, specifically targeted at the worst performing students within classrooms. However, their effectiveness is likely to depend on their design and the setting in which they are put in place. This paper evaluates the impact on students’ grades of a low-cost intervention in public secondary schools in Mexico City, which consisted in offering free additional math courses to students lagging behind their peers in marginalized low-income schools in Mexico City. We exploit the information available in all students’ (treated and not treated by the program) transcripts enrolled in participating and non-participating schools. Before the implementation of the program, participating students lagged behind non-participating ones by more than a half base point of their GPA (over 10). Using a difference-in-differences approach, we find that students participating in the program observed a higher increase in their school grades after the implementation of the program, and that the difference in grades between the two groups decreased over time. By the end of the school year (when the free extra courses had been offered, on average, for 10 weeks), participating students’ grades were not significantly lower than non-participating students’ grades. After accounting for the presence of (i) mean reversion, and (ii) differences in testing and grading among teachers, our results suggest that the impact on math grades associated with the program is positive and significant in the order of 0.21 14 Page 2 of 30 Lat Am Econ Rev (2014) 23:14 123 and 0.26 standard deviations in the fourth and fifth partial exams. 1 The estimated impact is similar in magnitude to other studies’ findings. For example, Banerjee et al. (2008) find that a similar intervention, the Balsakhi Program in India, increased participating students’ grades by 0.14 standard deviations in the first year. 2 The paper is organized as follows. Section 2reviews the recent literature that analyzes similar interventions, describing their design and findings in detail. Section 3describes the setting in which the program evaluated in this paper was put in place, and motivates the need for its evaluation. Section 4describes the data and the empirical strategy used in this paper. Results are presented in Sect. 5. Section 6 concludes. 2 Literature Despite the growing number of programs targeting underperforming students in different countries, there exists very little evidence on their effectiveness. Two recent exceptions, which analyze similar interventions to the one studied here, are Banerjee et al. (2008), and Lavy and Schlosser (2005). Banerjee et al. (2008) evaluate a remedial education program that hired women to teach third and fourth grade students in India lagging behind their peers in basic literacy and math skills in small groups. The evaluation design for this intervention used a sample of 15,000 students in two Indian cities, to which the treatment was randomly assigned. The treatment consisted in offering additional courses taught by an instructor (typically a young woman who received 2 weeks of training for this purpose) to a subset of lower performing students from each of the treated classrooms. These additional courses lasted for 2 h a day, took place during school hours, and were taught to groups of 15–20 children. The courses covered basic material that children were supposed to have learned in first and second grade. Finally, as the one studied in this paper, the cost of this intervention was relatively low, since teachers were local personnel trained for 2 weeks and paid 15 dollars per month. The authors look at the impact of the program on learning levels, measuring learning using annual pre-intervention tests administered during the first few weeks of the school year and post-intervention tests, administered at the end. They found that it increased average test scores of all children in treatment schools by 0.14 standard deviations, mostly due to large gains experienced by children at the bottom of the test-score distribution. Their results suggest that the remedial course program 1 In Mexico, grading throughout the school year is based on five evaluations (one every 2 months), called ‘‘exa ´menes parciales’’. The final GPA is calculated as the simple average of these five exams’ grades. Throughout the paper, we call each of these evaluations a ‘‘partial exam’’. 2 The Balsakhi program provided schools with a teacher (local personnel paid 15 dollars a month) to work for 2 h a day during the school year, with groups of 15–20 children in the third and fourth grades identified as falling behind their peers. Banerjee et al. (2008) estimate improvements in average test scores of 0.14 standard deviations in the first year, and 0.28 in the second. Lat Am Econ Rev (2014) 23:14 Page 3 of 30 14 123 was more cost effective than hiring new teachers 3 and than a computer-assisted learning program implemented at the same time in a similar geographic area. 4 Lavy and Schlosser (2005) evaluate the short-term effects of a remedial education program, which provided additional instruction to underperforming high school students in Israel. The program targeted tenth to twelfth graders in need of additional help to pass the matriculation exams. As a control, they use a comparison group of schools with similar characteristics to those treated (schools that enrolled later in the program), and apply a difference-in-differences empirical strategy. The intervention consisted in individualized instruction in small study groups of up to five tenth, eleventh and twelfth graders. The main goal of these study groups was, through individualized instruction based on underperforming students’ needs, to increase their matriculation rate and enhance their scholastic and cognitive abilities, self-image, and leadership aptitudes. The participants were chosen by their teachers based on their perceived likelihood that each student could pass his or her matriculation exam. The evaluation focuses on the first year of implementation of the program. 4,100 students were affected by the intervention (one-fifth of all students in treated schools). The authors look for the effects of the program on matriculation status, which is a comparable outcome for 12th graders. They find that the program raised the school mean matriculation rate by 3.3% points, mainly through its impact on targeted participants, rejecting the existence of externalities on their untreated peers. In conclusion, as summarized in Kremer et al. (2013), programs that reduce the costs of attending school and improve students’ health and the availability of information may have large impacts on school attendance, although not necessarily impacting their performance. Moreover, interventions that increase existing school inputs, such as teachers and textbooks, have been shown to be generally ineffective. However, programs that are tailored to each individual student’s learning needs, such as remedial programs, not only have been shown to have large impacts on students’ performance, but are also cost effective. Nevertheless, the effectiveness of programs of this kind relies on a correct design and the context of their implementation. 3 Mexico’s education system Pre-college education in Mexico is divided into three main stages: primary or elementary school, secondary school, and high school. Private schools must cover the same curriculum as schools in the public system, and the content of this curriculum is exclusively designed by the Ministry of Public Education (Secretarı ´a de Educacio ´nPu ´blica, SEP) for the first nine grades (all of primary 3 A program reducing class size appears to have had little or no impact on test scores, and the remedial course program costs 2.25 dollars per student per year. 4 The computer-assisted learning program costs approximately 15.18 dollars per student per year, including the cost of computers, assuming a 5-year depreciation. 14 Page 4 of 30 Lat Am Econ Rev (2014) 23:14 123 and secondary school). SEP shares the responsibility of designing the curriculum and regulating private education at the high school level with the public university system. Primary school lasts for 6 academic years, and is typically attended by children aged 6–12. Secondary school and high school last three academic years each. Unlike the United States, Mexico does not regulate the age at which children can legally drop out of school. Instead, although the law is not enforced, it is compulsory for all Mexicans to graduate from secondary school. There exists a variety of different programs within both primary and secondary public education in Mexico. Primary schools can be ‘‘general’’ (the most common), indigenous (where courses are taught in indigenous languages), and community schools (which target the most isolated communities and where students from different grades generally share a classroom). In addition to general and community schools, secondary schools can also be technical, ‘‘tele-secundarias’’ (which teach the general program but through television and pre-recorded classes), and secondary schools for adults (offered to individuals aged 15 or older who have completed their primary education). After secondary school, there are 3 years of high school which must be completed to attend college. Table 1shows some descriptive statistics about the Mexican education system, obtained from the Mexican Institute for the Evaluation of Education (INEE). The primary school system has nearly reached universal coverage. One hundred percent of children aged 6–11 are enrolled in school. Secondary school, despite being compulsory by law, does not show such high enrollment rates: 90 % of children aged 12–14 are enrolled in school. Enrollment rates in high school are considerably lower: only 60 % of children aged 15–17 are enrolled in school. Mexico City shows slightly different enrollment rates: full enrollment in primary and secondary schools and 86.7 % enrollment in high school. It is then clear that the largest drop in enrollment rates, particularly in Mexico City, takes place in the transition from secondary school to high school. However, perhaps surprisingly, from all those students graduating from the last grade of secondary school, nearly all of them enroll in high school, particularly in Mexico City. The drop in enrollment seems then primarily a consequence of students’ low performance in secondary school, which does not allow them to graduate and continue with their education. 16 % of secondary school students do not pass the grade in which they are enrolled. According to the standardized test applied by INEE, ENLACE, in 2010, 40 % of secondary school students had insufficient verbal skills and another 40 % just reached basic verbal skills’ levels; 53 % of secondary school students had an insufficient math skills and another 34.7 % just reached basic skills in math. According to the PISA 2009 test, 51 % of 15-yearold students in Mexico’s educational system performed below level 2 in mathematics (55 % in PISA 2012), and just 5 % ranked in the highest level (5). It seems then crucial to design and evaluate interventions aimed at improving students’ performance in secondary schools in Mexico, if increasing educational attainment is a desired goal. Lat Am Econ Rev (2014) 23:14 Page 5 of 30 14 123 4 The program The program evaluated in this paper offers a remedial math course to lowperformance students enrolled in the last grade of secondary school from marginalized schools in Mexico. It was put in place by the representation of the SEP in Mexico City in collaboration with the Laboratory of Initiatives for Development (LID), a local NGO. It ran during the second half of the academic year 2009–2010, from April to June 2010 in 33 schools in 11 different delegaciones. 5 The remedial course was taught by a group of undergraduate students. There were, in total, 55 advisors from three of the most prestigious universities in the country: UNAM, ITAM, and UP. UNAM is a public university, while the other two are private. These advisors fulfilled their ‘‘social service’’ requirement by participating in this program. 480 h of social service activities (understood as activities beneficial to society, the State or a university), after covering 70 % of the undergraduate program credits, is a legal requirement for students to obtain a college degree in Mexico. There is a wide range of activities that undergraduates can do to fulfill this requirement, which go from being a research assistant, participating in reforestation campaigns, or taking a job for a social organization or government agency, with the latter being a common case. Employers are not legally obliged to offer any remuneration for students during their social service. However, generally, students do receive payments to cover their transportation needs (as was the case for the students participating in the remedial program). The program seems then easily replicable and, the extent to which it will be displacing social service activities, its opportunity cost seems potentially low. The program was advertised through the universities’ social service offices and participating advisors enrolled voluntarily. Table 2shows the distribution of advisors recruited, by university and gender. Advisors’ attendance to the remedial Table 1 Mexico’s education system. Descriptive Statistics (2010) Percentage enrolled in school All of Mexico Mexico City Children aged 6–11 100 100 Children aged 12–14 90 100 Children aged 15–17 60.3 86.7 Continuation rate* Primary to secondary 97 99 Secondary to high school 100?100? Source: INEE (2012) * The continuation rate is calculated as the ratio between the number of individuals enrolling in one education level in a given year divided by the total number of individuals graduating from the preceding education level in the previous academic year. 100?indicates that this ratio was higher than one 5 Mexico City is divided into 16 delegaciones, which are the smallest administrative entities in the city. 14 Page 6 of 30 Lat Am Econ Rev (2014) 23:14 123 courses was controlled by the principals of all participating schools. In addition, random visits to the remedial sessions were put in place by the implementing NGO to verify their attendance to the remedial courses. In case the advisors were absent for more than two consecutive sessions, they were not given the approximately 80 dollars for transportation costs that they were entitled to. Prior to the intervention, the advisors were required to attend a brief but useful training session of 4 h in total given by the Mexican Academy of Science, where they had access to the mathematics syllabus at the secondary school level, a previous final exam, and were instructed to look at children’s notebooks and continuously ask the students for specific questions to regularly adapt the course’s content to the group’s needs. The intervention in each of the participating schools typically consisted of a meeting of two advisors with a group of up to 20 students, 2 days a week for 2 h, after school. This extra course focused on helping children develop the mathematical skills needed to improve their grades to pass the course. The assignment of advisors to schools occurred as follows: (a) each of the volunteers stated the neighborhood to which they would prefer to be assigned. Table 2 Administrative data Female Male Total Advisors Number of advisors 29 26 55 UNAM 10 7 17 ITAM 14 16 30 UP 5 3 8 Group size Initial size 13.1 13.7 13.4 Final size 12.0 13.0 12.4 Sessions 11.2 7.7 9.9 Total hours 27.6 31.2 29.2 Schools Type of secondary school General 34 Technical 21 Shift Morning school 18 Afternoon school 29 Both 8 Geographical location North 14 South 7 East 30 Center 4 Lat Am Econ Rev (2014) 23:14 Page 7 of 30 14 123 (b) coordinators located three schools in the said neighborhood, choosing the worst ranked in the ENLACE 2009 test who were willing to receive the program, (c) the students ranked them according to their preferences and finally, taking into account this information, coordinators assigned advisors to a single school. Table 2presents descriptive statistics for the outcome of the assignment of advisors to schools. 34 advisors were assigned to general secondary schools and 21 to technical secondary schools; 18 taught the remedial courses to morning shift students, and 29 to students in the afternoon shift. 8 advisors taught to students in both shifts. Finally, from a total of 55 advisors, 14 volunteered at schools located in northern Mexico City, 7 in the south, 30 in the east and 4 in the center. 6 The assignment of students to the remedial course was decided by the schools’ principals. The general guidelines suggested by the coordinators of the program were to identify students at risk of failing the school year, based on their performance in the first two partial exams that they had already taken during that academic year. The third partial exam had not yet been administered in any of the schools at the time of the assignment of students to the remedial course. The intervention began after the third partial exam on a date that varies from school to school but in all cases before the fourth partial exam had been administered, and lasted until the end of the school year. Table 3shows the fraction of students participating in the remedial course that scored a grade average that was lower than the minimum passing grade after the first two partial exams administered. 55 % of participating students had a passing grade average in their first two partial exams, while 78 % of those not participating had a passing grade average. The difference in this fraction is statistically significant from zero. The second panel in Table 2shows that dropout rates from the remedial course were low. At the beginning of the program, on average, 13.4 students attended the course. The average remedial course class size by the end of the school year was 12.4 students. 5 Evaluation design For the evaluation of this program, we collected detailed information on the math grades obtained by a sample of participating and non-participating students in the five partial exams administered during the academic year. As the program only started after the third partial exam, we then have information on students’ performance before and after its implementation. There are two different groups of non-participating students in our sample: non-participating students in schools in which the program was put in place, and all students in a set of 60 schools (similar in observable characteristics to the treated ones), in which the remedial course was not offered. 6 The odd numbers reported in each school category are due to a few cases in which one single advisor was assigned to a class, or to more than one school. 14 Page 8 of 30 Lat Am Econ Rev (2014) 23:14 123 Scoreijt ¼X 5 t¼3;tPartialtþX 5 t¼3 btTreatmentiPartialt þX 5 t¼3 ctScorei1PartialtþX 5 t¼3 ptðScorei2Scorei1ÞPartialt þX 5 t¼3 otðScorei2Scorei1ÞScorei1Partialtþeijt ð4Þ Equations (3) and (4) allow us to further relax the identification assumption necessary for the estimates from Eq. (2) to measure the effect of the program. Now, the assumption would be that treated and non-treated students with similar grades in the first partial exam and similar changes in grades between the first and second exams were equally likely selected to receive the intervention. Moreover, there is no further assumption needed with respect to the existence of noise when measuring the pre-program performance of the students, because this specification controls for that, allowing the occurrence of transitory shocks. The idea now is that apart from the possibility of having experienced a negative transitory shock during the first partial, it could also be the case that students in the treatment (or in the control) group experienced a negative (or positive) shock during the second exam correlated with the shock in the previous one. In this case, we would observe improvements in grades on further exams even in the absence of the program, caused by mean reversion. Therefore, we need to add to Eq. 2the interactions between the score in the second partial exam and the dummies of partial exams 3–5. 7 This specification allows us to further relax the assumption in Eq. (1), Eu ijt  ¼0;8i;8j;8t, and allows for potential transitory shocks suffered in periods 1 and 2 leading to mean reversion . Also, note that adding Scorei2or ðScorei2Scorei1Þto Eq. (2) solves the concern about mean reversion explained in the previous paragraph, since the difference ðScorei2Scorei1Þis a linear transformation of Scorei2. Therefore, the specification suggested by Eq. (3) not just controls for the ability of students to improve, but also for mean reversion in a more satisfactory way. If this specification is correct, we would expect the coefficients for the interaction of the treatment and the third period dummy to be not significantly different from zero, as the program had not been implemented by that time. And the coefficients for the interaction of the treatment with periods 4 and 5 dummies would reflect the impact associated with the program. 7 Specifically, we would need to estimate the following equation: Scoreit ¼P 5 t¼2;tPartialtþP 5 t¼2 btTreatmentiPartialt þP 5 t¼2 ctScorei1PartialtþP 5 t¼2 utScorei2Partialtþeit Lat Am Econ Rev (2014) 23:14 Page 15 of 30 14 123 6.3 Differences in testing and grading among teachers An important difference between this paper and similar studies (listed above) is that the scores observed for each student correspond to their grades on exams designed and graded by their specific teachers and not to standardized tests. In this section, we analyze the advantages and disadvantages of considering this measure of students’ performance. Standardized tests are uniform in subject matter, format, administration, and grading procedure across all test takers, while a course grade might depend on a particular teacher’s judgment. However, course grades and standardized tests both reflect students’ skills and knowledge, and are thus generally highly correlated (Willingham et al. 2002). Trying to provide an explanation to why these two evaluation tools are not entirely interchangeable, recent literature has underlined aspects of personality that seem essential to earning strong course grades because of what is required of students to earn them. Almlund et al. (2011) analyze how the contribution of personality to performance varies among these two evaluation methods. One key point is that despite the power of standardized achievement tests to predict later academic and occupational outcomes (for example in Kuncel and Hezlett (2007); Sackett, et al. (2008)), cumulative high school GPA predicts graduation from college much better than standardized test scores do (as shown by Bowen et al. (2009)). Similarly, high school GPA more powerfully predicts college rank-in-class (Bowen et al. 2009; Geiser and Santelices 2007). Duckworth et al. (2012) compare the variance explained in standardized test scores and GPA at the end of the school year by self-control and intelligence measured at the beginning of the school year. In a sample of children, they found that fourth graders’ self-control was a stronger predictor of ninth graders’ GPA than fourth graders’ IQ. At the same time, fourth graders’ self-control was a weaker predictor of ninth graders’ standardized test scores than was fourth graders’ GPA. Similarly, Oliver et al. (2007) found that parent and self-reported ratings of distractibility at age 16 predicted high school and college GPA, but not SAT test scores. Given these arguments, analyzing grades instead of standardized scores seems to be a better idea to measure the impact of the program according to the initial objective. Unfortunately, the possibility of high subjectivity on our performance measure still remains. However, as long as this subjectivity is teacher specific and not correlated with the treatment within classrooms, we can evaluate to what extent treated students caught up with their untreated peers. Taking advantage of the panel structure of our data, which contains a measure of each student performance in five different periods, we can control by school level fixed effects. In this way, we relax the implicit assumption in every previous specification that EðsjÞ¼0;8j, in words, we are capturing the differences in students’ scores driven by differences across school characteristics, such as the teacher’s specific preferences for testing and grading. 14 Page 16 of 30 Lat Am Econ Rev (2014) 23:14 123 7 Results Results for all specifications described above are presented in Tables 5and 6. Table 5uses all non-treated students in participating schools as the control group, while Table 6includes all non-treated students as a control (including those in the 60 non-participating schools for which we have information). Columns 1, 2, 3 and 4 show the results for the specification described in the equation with the same number in the previous section. The results of estimating Eq. (1) are shown in Column 1 of Tables 5and 6.As can be seen, the coefficients for the interaction between the treatment and partial variables are negative and significantly different from zero for the first three partials both when restricting the sample to all students in treated schools (Table 5) and when including all students in treated and non-treated schools (Table 6). The coefficient for the interaction between the treatment dummy and partial 4 is significantly lower in magnitude than those for the first three partial exams. The grade gap between the treated and non-treated students seems then to decrease after the implementation of the program. Perhaps more interestingly, the coefficient for the interaction between the treatment and period 5 dummies is close to zero and insignificant when the sample is restricted to students in treated schools (suggesting that the program might have completely closed the performance gap between the two groups after 3 months of its implementation). When including students in nontreated schools in the sample, this coefficient remains significantly negative, although still smaller in magnitude than that for partial 4. Consistently, when testing whether the coefficients of the interactions between the treatment dummy and the partial exam dummy variables are statistically different for both control groups (non-treated students within treated schools and in all schools), reported in Table 7, we fail to reject that the estimated coefficients for the interaction with partial 1 and the interaction with partial 2 are equal, and the same with the estimated coefficients for the interaction with partial 2 and the interaction with partial 3. Nevertheless, based on a confidence level of 95 %, the F test rejects that the estimated coefficients for the interaction between the treatment dummy with partial 3 and the interaction with partial 4 are equal (when doing this exercise for all schools, we get the same result but with a confidence level of 90 %), and the same result is obtained when testing for the interaction with partial 3 and the interaction with partial 5. We fail to reject that the estimated coefficients for the interaction with partial 4 and the interaction with partial 5 are equal. Nevertheless, as stated above, for this empirical strategy to correctly identify the program’s effects, we would expect the trends in grades between treated and nontreated students before the implementation of the program to not be different. Figure 1, which shows the regression results graphically, suggests that this is perhaps not the case. Grades for treated students seem to increase (relative to nontreated students’ grades) since before the program’s implementation (period 3). The following three columns in both tables present the regression results for the specifications described in Eqs. (2)–(4), respectively. Column 2 compares scores in partial exams 2–5 of students from the treatment and control groups controlling for the score in the first partial exam. As can be seen, students in the treatment group, by Lat Am Econ Rev (2014) 23:14 Page 17 of 30 14 123 Table 5 Treatment effects on the average math scores within treated schools (1) (2) (3) (4) Partial 1 6.863*** [0.028] Partial 2 6.945*** [0.028] 2.108*** [0.126] Partial 3 6.663*** [0.027] 2.390*** [0.126] 0.991*** [0.128] 0.992*** [0.128] Partial 4 6.679*** [0.027] 2.253*** [0.126] 0.802*** [0.128] 0.810*** [0.128] Partial 5 6.729*** [0.028] 2.505*** [0.126] 0.945*** [0.128] 0.952*** [0.128] Treatment * Partial 1 -0.569*** [0.082] Treatment * Partial 2 -0.678*** [0.082] -0.278*** [0.079] Treatment * Partial 3 -0.438*** [0.081] -0.084 [0.079] 0.100 [0.077] 0.100 [0.077] Treatment * Partial 4 -0.162** [0.081] 0.205*** [0.079] 0.396*** [0.077] 0.393*** [0.077] Treatment * Partial 5 -0.013 [0.081] 0.337*** [0.079] 0.543*** [0.077] 0.540*** [0.077] Controls by Partial 1 grade No Yes Yes Yes Controls by trends in grades before the treatment No No Yes Yes Controls by the interaction of trends and grade in Partial 1 No No No Yes Observations 29,410 23,528 17,646 17,646 R-squared 0.92 0.93 0.93 0.93 This table reports the estimates of the treatment effect as specified in Eq. (1,2,3, and 4), presented in the same order in Columns (1)–(4), taking all the non-treated students of the treated schools as the comparison group. Standard errors are in brackets * Significant at 10 %; ** significant at 5 %; *** significant at 1 % 14 Page 18 of 30 Lat Am Econ Rev (2014) 23:14 123 Table 6 Treatment effects on the average math scores for all the sample’s schools (1) (2) (3) (4) Partial 1 6.923*** [0.014] Partial 2 6.931*** [0.039] 2.026*** [0.059] Partial 3 6.807*** [0.023] 1.989*** [0.062] 0.608*** [0.059] 0.612*** [0.060] Partial 4 6.751*** [0.013] 1.809*** [0.059] 0.457*** [0.059] 0.463*** [0.060] Partial 5 6.897*** [0.014] 2.317*** [0.055] 0.951*** [0.056] 0.963*** [0.057] Treatment * Partial 1 -0.628*** [0.079] Treatment * Partial 2 -0.664*** [0.078] -0.219*** [0.073] Treatment * Partial 3 -0.582*** [0.078] -0.145** [0.072] 0.005 [0.070] 0.004 [0.070] Treatment * Partial 4 -0.234*** [0.078] 0.215*** [0.073] 0.361*** [0.070] 0.359*** [0.070] Treatment * Partial 5 -0.180** [0.078] 0.236*** [0.073] 0.384*** [0.070] 0.382*** [0.070] Controls by Partial 1 grade No Yes Yes Yes Controls by trends in grades before the No No Yes Yes Controls by the interaction of trends and grade in Partial 1 No No No Yes Observations 110,005 88,004 66,003 66,003 R-squared 0.92 0.93 0.94 0.94 This table reports the estimates of the treatment effect as specified in Eq. (1,2,3, and 4), presented in the same order in Columns (1)–(4), taking all the non-treated students of the schools in the sample as the comparison group. Standard errors are in brackets * Significant at 10 %; ** significant at 5 %; *** significant at 1 % Lat Am Econ Rev (2014) 23:14 Page 19 of 30 14 123 the fifth partial exam, had experienced a 0.33 point increase in their grades relative to their non-treated peers (Table 5) and a 0.24 points increase relative to the whole sample of non-treated students. This change is roughly equivalent to half the grade gap between the groups in their first partial exam. However, it is worth noting that the coefficient on the interaction between the treatment dummy and the second period is, for both samples, negative and statistically different from zero, but a closing in the gap is still observed by the third period, when the program had not yet begun. Adding the interaction between the first score and the period dummies does 5.8 6 6.2 6.4 6.6 6.8 7 12345 Treatment Control 1 -0.7000 -0.6000 -0.5000 -0.4000 -0.3000 -0.2000 -0.1000 0.0000 0.1000 12345 Diffs 5.8 6 6.2 6.4 6.6 6.8 7 12345 Treatment Control2 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 012345 Diffs Fig. 1 Trends in partial grades between treatment groups. The left panel shows the average test scores for treated and untreated students in treatment schools. The right panel includes non-treated schools in the control group Table 7 Ftests of differences on treatment effects across partial exams Null hypothesis Treated schools All sample’s schools Ftest Pvalue Ftest Pvalue Partial 1 =Partial 2 0.900 0.342 0.100 0.747 Partial 2 =Partial 3 4.330 0.037 0.550 0.460 Partial 3 =Partial 4 5.690 0.002 9.760 0.002 Partial 3 =Partial 5 13.550 0.000 13.050 0.000 Partial 4 =Partial 5 1.680 0.195 0.240 0.626 Observations 29,400 110,005 This table presents the Ftest and Pvalues’ result of testing the null hypothesis of estimated coefficients for the treatment effects of each couple of partial exams to be equal. First two columns for the regressions run for treated schools (reported in Table 5), the last two for those run for all sample’s school (in Table 6) 14 Page 20 of 30 Lat Am Econ Rev (2014) 23:14 123 not seem to fully solve the problem of mean reversion caused by the use of a noisy measure of the students’ pre-treatment performance. The last two columns in Tables 5and 6include further controls measuring students’ performance in the first two partial exams. The results are quantitatively similar when excluding or including (Columns 3 and 4) the interaction between the grade in the first partial and the change in grades between the first and the second exam as controls. Once we include all the observable information that the principals had at the moment of selecting the treated students (the scores in partial 1 and partial 2, and the difference between them), the coefficient of the interaction between the treatment dummy and the score in the third partial exam is insignificantly different from zero. Students’ grades are then uncorrelated with the treatment before the implementation of the program. With this specification, the difference in grades between treated students and their non-treated peers (Table 5) decreases by 0.54 points by the fifth partial exam. This magnitude is very close to the grade gap in the first partial exam, suggesting that the grade gap between treated students and their peers was fully closed by the fifth partial exam. The same estimate using all nontreated students in our sample as the control group (Table 6) suggests that the grade gap was closed by 0.38 points by the fifth partial exam. Tables 8and 9are analogous to Tables 5and 6but include school fixed effects to correct by differences in testing and grading among teachers. 8 The estimated coefficients increase with respect to those without fixed effects (when it was 0.54), to 0.66 points by the fifth partial when using the non-treated students in treated schools as the control group. Tables 10 and 11 present the results of the same specifications, splitting the sample into boys and girls, respectively, and using the non-treated students in treatment schools as the control group. According to the results of specification (1), in both tables, the differences between treated and untreated boys’ and girls’ scores in the first three partial exams are similar in magnitude, and the decreasing pattern in this gap in the fourth and fifth partial exams documented for the full sample remains for both groups. Looking at the graphed magnitudes in Fig. 2, it seems that the changes in grades are larger for boys than girls, although it could be an artifact of larger mean reversion in the boys’ sample, as suggested by the reduction in the gap in the third partial exam, before the remedial course had been offered. When including further controls and school fixed effects, the difference in grades between treated students and their non-treated peers decreases by 0.73 points by the fifth partial exam for boys, and by 0.56 points for girls. Finally, to be able to compare the results of this low-cost program with the impact of other interventions, Table 12 reports our results in standard deviations taking the untreated students in treatment schools as the control group. The estimations correspond exactly to those from Table 8but with standardized variables. Here, the estimated impact amounts to 0.26 standard deviations. 8 This exercise was made at the school level and not at class level as was suggested in the Evaluation Design section, due to data limitations (we count with school identifier but not class identifier). Nevertheless, according with the educational authorities, it is always the same teacher who teaches the same subject to all classes in a school (but rare exceptions). In which case, school or class fixed effects estimations would equally correct for differences in testing and grading among teachers. Lat Am Econ Rev (2014) 23:14 Page 21 of 30 14 123 Table 8 Treatment effects on the average math scores within treated schools, FE schools (1) (2) (3) (4) Partial 1 6.859*** [0.208] Partial 2 6.941*** [0.165] 1.726*** [0.273] Partial 3 6.658*** [0.076] 2.007*** [0.348] 0.638*** [0.204] 0.637** [0.215] Partial 4 6.674*** [0.067] 1.870*** [0.356 0.448*** [0.203] 0.456** [0.134] Partial 5 6.724*** [0.101] 2.123*** [0.360] 0.591** [0.292] 0.597** [0.202] Treatment * Partial 1 -0.527** [0.205] Treatment * Partial 2 -0.637*** [0.178] -0.095 [0.108] Treatment * Partial 3 -0.396* [0.203] 0.099 [0.146] 0.216 [0.115] 0.215 [0.116] Treatment * Partial 4 -0.121 [0.205] 0.387*** [0.158] 0.511*** [0.123] 0.507*** [0.123] Treatment * Partial 5 0.029, 0.208 0.520*** [0.173] 0.658*** [0.142] 0.655*** [0.143] Controls by Partial 1 grade No Yes Yes Yes Controls by trends in grades before the treatment No No Yes Yes Controls by the interaction of trends & grade in Partial 1 No No No Yes Observations 29,410 23,528 17,646 17,646 R-squared 0.92 0.93 0.94 0.94 This table reports the estimates of the treatment effect as specified in Eq. (1,2,3, and 4), presented in the same order in Columns (1)–(4), taking all the non-treated students of the schools in the sample as the comparison group. Standard errors are in brackets * Significant at 10 %; ** significant at 5 %; *** significant at 1 % 14 Page 22 of 30 Lat Am Econ Rev (2014) 23:14 123 Table 9 Treatment effects on the average math scores for all the sample’s schools, FE schools (1) (2) (3) (4) Partial 1 6.919*** [0.036] Partial 2 6.927*** [0.033] 1.929*** [0.117] Partial 3 6.803*** [0.046] 1.891*** [0.122] 0.151*** [0.071] 0.587** [0.071] Partial 4 6.747*** [0.058] 1.711*** [0.141] 0.435*** [0.094] 0.438*** [0.093] Partial 5 6.893*** [0.065] 2.220*** [0.170] 0.494*** [0.103] 0.937*** [0.103] Treatment * Partial 1 -0.501** [0.212] Treatment * Partial 2 -0.537*** [0.178] -0.008 [0.137] Treatment * Partial 3 -0.455** [0.187] 0.067 [0.134] 0.221* [0.123] 0.218* [0.123] Treatment * Partial 4 -0.107 [0.189] 0.426*** [0.136] 0.577*** [0.121] 0.574*** [0.121] Treatment * Partial 5 -0.053 [0.183] 0.447*** [0.155] 0.599*** [0.143] 0.596*** [0.143] Controls by Partial 1 grade No Yes Yes Yes Controls by trends in grades before the treatment No No Yes Yes Controls by the interaction of trends and grade in Partial 1 No No No Yes Observations 110,005 88,004 66,003 66,003 R-squared 0.92 0.93 0.94 0.94 This table reports the estimates of the treatment effect as specified in Eq. (1,2,3, and 4), presented in the same order in Columns (1)–(4), taking all the non-treated students of the schools in the sample as the comparison group. Standard errors are in brackets * Significant at 10 %; ** significant at 5 %; *** significant at 1 % Lat Am Econ Rev (2014) 23:14 Page 23 of 30 14 123 Table 10 Treatment effects on the average math scores of boys within treated schools, FE schools (1) (2) (3) (4) Partial 1 6.691** [0.191] Partial 2 6.723*** [0.097] 1.628*** [0.338] Partial 3 6.412*** [0.165] 1.854*** [0.273] 0.476** [0.205] 0.487** [0.194] Partial 4 6.396*** [0.168] 1.451*** [0.158] 0.272** [0.166] 0.295** [0.167] Partial 5 6.413*** [0.261] 1.535*** [0.267] 0.235*** [0.271] 0.253*** [0.269] Treatment * Partial 1 -0.567*** [0.196] Treatment * Partial 2 -0.590*** [0.189] -0.025 [0.144] Treatment * Partial 3 -0.298** [0.064] 0.219 [0.191] 0.246 [0.144] 0.242 [0.143] Treatment * Partial 4 0.04 [0.225] 0.572** [0.214] 0.602*** [0.173] 0.596*** [0.171] Treatment * Partial 5 0.167 [0.210] 0.693*** [0.188] 0.735*** [0.155] 0.730*** [0.156] Controls by Partial 1 grade No Yes Yes Yes Controls by trends in grades before the treatment No No Yes Yes Controls by the interaction of trends and grade in Partial 1 No No No Yes Observations 15,042 12,034 9,026 9,026 R-squared 0.564 0.568 0.81 0.81 This table reports the estimates of the treatment effect as specified in Eq. (1,2,3,and4), presented in the same order, with fixed effects by school for boys, taking all nontreated students of the treated schools in the sample as the comparison group. Cluster-robust standard errors are in brackets * Significant at 10 %; ** significant at 5 %; *** significant at 1 % 14 Page 24 of 30 Lat Am Econ Rev (2014) 23:14 123