The effects of teaching and assessment methods on academic performance: a study of an Operations Management course

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=riie20 Download by: [Universidad de Sevilla] Date: 02 November 2017, At: 04:16 Innovations in Education and Teaching International ISSN: 1470-3297 (Print) 1470-3300 (Online) Journal homepage: http://www.tandfonline.com/loi/riie20 The effects of teaching and assessment methods on academic performance: a study of an Operations Management course Macarena Sacristán-Díaz, Pedro Garrido-Vega, Rafaela Alfalla-Luque & María-del-Mar González-Zamora To cite this article: Macarena Sacristán-Díaz, Pedro Garrido-Vega, Rafaela Alfalla-Luque & María-del-Mar González-Zamora (2016) The effects of teaching and assessment methods on academic performance: a study of an Operations Management course, Innovations in Education and Teaching International, 53:5, 497-507, DOI: 10.1080/14703297.2015.1008546 To link to this article: http://dx.doi.org/10.1080/14703297.2015.1008546 Published online: 09 Feb 2015. Submit your article to this journal Article views: 196 View related articles View Crossmark data Citing articles: 1 View citing articles The effects of teaching and assessment methods on academic performance: a study of an Operations Management course Macarena Sacristán-Díaz*, Pedro Garrido-Vega, Rafaela Alfalla-Luque and María-del-Mar González-Zamora Department of Financial Economics and Operations Management, University of Seville, Sevilla, Spain Whether the use of more active teaching–learning methods has a positive impact on academic performance remains unanswered. This article seeks to contribute to the issue by conducting a study of an Operations Management course with almost 1000 students per year over three consecutive academic years. The study compares three scenarios with differing degrees of student involvement and Information Technology usage. The findings show no differences in terms of exam grades, but there is an increase in the percentage of passes in the most active scenario analysed, in which continuous assessment tests are used. Notwithstanding, student involvement in the optional assessable activities was found to be a significant factor for predicting the likelihood of passing the subject, especially when continuous assessment tests are used, since their grades show a higher correlation with exam grades. Keywords: teaching methods; assessment; academic performance; passes; grades; logistic regression Introduction and theoretical background Teaching and assessment methods (hereinafter T&AM) are usually considered to affect students’academic performance (Tejedor, 2003). However, the results of research that measures their impact on performance are inconclusive. Friday, Friday-Stroud, Green, and Hill (2006) and McLaren (2004)find that the implementation of new T&AM has not always resulted in improved performance. By contrast, other research studies determine that improvements in T&AM have a positive impact on student performance (Marklin Reynolds & Hancock, 2010). In general, improving T&AM consists of making the students take a more active role in the teaching–learning process by carrying out different activities throughout the course. Another way of creating more active learning environments, sometimes used to complement the above method, is the use of Information Technology (IT). IT is increasingly viewed as a key resource for enabling innovative and enhanced learning experiences in management education (Holbert & Karady, 2009; Medina-López, Alfalla-Luque, & Arenas-Márquez, 2011; Pascual-Ezama, Camacho-Miñano, Urquía-Grande, & Müller, 2011). Internet-based technologies are rapidly gaining acceptance as a supplement to traditional classroom instruction for management education, enriching the teaching and learning experiences for students and lecturers *Corresponding author. Email: [email protected] © 2015 Informa UK Limited, trading as Taylor & Francis Group Innovations in Education and Teaching International, 2016 Vol. 53, No. 5, 497–507, http://dx.doi.org/10.1080/14703297.2015.1008546 Downloaded by [Universidad de Sevilla] at 04:16 02 November 2017 (Liaw, Huang, & Chen, 2007; Medina-López, Alfalla-Luque, & Marín-García, 2011; Wang, Wang, & Shee, 2007). Although the promises of IT-based instruction in the literature are extensive and there is some evidence that it may enhance the classroom environment and support student learning (Arquero & Romero-Frías, 2013; Daymont & Blau, 2008), further empirical research would seem to be necessary (Ginns & Ellis, 2009). The present paper focuses on the context of the Spanish University to provide new empirical evidence by evaluating the influence of more active T&AM on academic performance. Academic performance is a complex concept that can be measured by various indicators, with the exam grade being the most used in previous research. There is evidence that more active T&AM have a positive influence on the exam grade (Fish, 2008; López-Pérez, Pérez-López, & Rodríguez-Ariza, 2011; Polito, Kros, & Watson, 2004). However, findings to the contrary also exist, showing that more active methods do not improve results in comparison with traditional classes or seminars (Alfalla-Luque, Medina-López, & Arenas-Márquez, 2011; Collaud, Gurtner, & Coen, 2000; McLaren, 2004; Merril & Galbraith, 2009; Naslund, 2005). To a lesser extent, the pass rate of students who sit the exam (success rate) has also been used as an indicator of performance. Some research shows that more active T&AM have a positive effect on the success rate (Arquero-Montaño, Jiménez-Cardoso, & Joyce, 2004; Dziuban, Hartman, & Moskal, 2004; López-Pérez et al., 2011). As far as is known, there is no research that analyses the pass rate of enrolled students (performance rate). However, in our particular context, Spanish universities, it is also important to consider this performance indicator. The reason for this is that student absenteeism and high dropout rates are widespread problems in Spanish universities (López-Bonilla & López-Bonilla, 2015; Sacristán-Díaz, Garrido-Vega, González-Zamora, & Alfalla-Luque, 2012). Some studies quantify levels of absenteeism at between 10 and 35% (Crespo-Tejero, Palomo-Vadillo, & Méndez-Suárez, 2012). Other studies find percentages of no-shows at examinations of 31.2% (Gracia-Expósito & De la Iglesia-Villasol, 2007) or even 49% (Parra, Baena, Jiménez, & Valencia, 2008). This means that each academic year has a significant percentage of students who fail the course (repeater students in the following academic year). The importance and concern for this problem is reflected by both the success and performance rates being included among other performance indicators in the Spanish University Degree Quality Assurance System. Therefore, besides the usual exam grade, this paper also focuses on these two other academic performance measures when analysing the impact of different types of T&AM. In view of the above, the present paper specifically poses the following research question: Do more active T&AM help to improve academic performance? With the aim of contributing to the prior literature, the present study also seeks to evaluate the impact of student involvement in optional assessable activities as an explanatory variable or predictor of academic performance. In this regard, Rúa-Vieytes and Gónzález-Martínez (2004) conclude that the grades achieved in tests taken during the course help to predict students’final academic performance. Meanwhile, Barkley (2006) analyses the impact of different evaluation methods (homework, quizzes and exams) on final exam grades, finding that these are influenced by the grades of mid-term exams, but not by those of assignments and questionnaires. Finally, Durán-Santomil, Maside-Sanfíz, Catorna-Agra, and 498 M. Sacristán-Díaz et al. Downloaded by [Universidad de Sevilla] at 04:16 02 November 2017 Rodeiro-Pazos (2013)find that the final exam grade is influenced by the marks of partial tests during the course, while the ratings of active participation in the classroom and group work have no influence. To the best of our knowledge, there are no previous studies that focus on the impact of student involvement in some optional assessable activities on both the performance rate and the success rate. Taking into account the above, a second research question is thus added: To what extent can student involvement in more active T&AM predict academic performance? Research methodology Subject and scenarios A study was conducted at the University of Seville (Spain) using data on an Operations Management (OM) second-year subject worth 9 ECTS credits that forms part of an undergraduate degree in Business Administration, compiled over three academic years. Almost 1000 students are enrolled on the course every year and are randomly split into nine different groups according to their surnames. Overcrowding is therefore rife in the subject. Another habitual problem is the high dropout rate, which results in low attendance at the first sitting of the final examination, usually well below 50% of enrolled students. The theoretical–practical tests for the students assessment consisted of 20 theoretical multiple-choice test questions weighted at 40%, plus another 20 practical multiple-choice questions weighted at 60% that the students could only answer after solving a series of problems. A survey of the students on the course some years earlier revealed that they had difficulties with practical exercises and multiple-choice tests. This led teaching staff to concentrate on these two angles and to include a greater IT use and more active T&AM activities which, although voluntary, would be graded. This resulted in the teaching and assessment scenario evolving over three consecutive academic years. The scenarios are summarised in Table 1and can be described as follows: Scenario 0 (S0): is the control scenario in which traditional teaching was used (lectures and textbook). The assessment system included only examinations (theoretical–practical multiple-choice tests, as described above). The subject website was used solely as a repository of information. Table 1. Summary of the three teaching and assessment scenarios analysed. Scenario (S) S0 S1 S2 Optional assessable activities % of exam grade in final grade 100 90 80 Theoretical–practical assignments (weighting) No Yes (10%) No Continuous assessment tests (weighting) No No Yes (20%) IT support Website (information repository) Yes Yes Yes Self-assessment via web No Yes Yes Teaching platform (WebCT) No Yes Yes WebCT use for assessable activities No No Yes Innovations in Education and Teaching International 499 Downloaded by [Universidad de Sevilla] at 04:16 02 November 2017 Scenario 1 (S1): four theoretical–practical assignments were recommended as assessable activities. These were not compulsory but represented 10% of the final grade. The remaining 90% corresponded to the theoretical and practical multiple-choice examination (same structure as in S0). Self-assessment tests were also included on the subject website and the WebCT virtual platform (but were not taken into account for the final grade). Scenario 2 (S2): the assessable activities consisted of doing eight continuous assessment theoretical and practical multiple-choice tests via WebCT. They were not compulsory and were worth 20% of the final grade. The remaining 80% corresponded to the theoretical–practical multiple-choice examination (same structure as in S0 and S1). Non-graded self-assessment was retained but no assignments were done. Variables and measures The scenario and student involvement (yes/no) in the optional assessable activities were used as independent or explanatory variables (Table 2). As for the dependent or outcome variables, academic performance was measured using three indicators: performance rate (passes over the number of students enrolled on the course), success rate (passes over the number of students that actually sat the exam) and exam grades obtained by students (0–10, with 5 being the pass-grade). Tejedor (2003) identifies five categories of variables that influence academic performance (identification, psychological, academic, educational and socio-family). Of these, three dichotomous control variables have been considered in our research (Tables 2and 3) since they were available for the teachers: shift (morning or Table 2. Descriptive data for variables analysed. S0 S1 S2 No. students enrolled 805 988 939 Gender N %N%N% Female 410 50.93 536 54.25 514 54.74 Male 395 49.07 452 45.75 425 45.26 Repeating subject Yes 374 46.46 466 47.17 431 45.90 No 431 53.54 522 52.83 508 54.10 Shift Morning 417 51.80 546 55.26 534 56.87 Evening 388 48.20 442 44.74 405 43.13 Did recommended activities Yes ––268 27.13 492 52.40 No ––720 72.87 447 47.60 Drop-out rate Students sitting exam 331 41.12 421 42.61 484 51.54 Students not sitting exam 474 58.88 567 57.39 455 48.46 Performance Passed over enrolled 251 31.18 314 31.78 365 38.87 Passed over sat exam 251 75.83 314 74.58 365 75.41 Mean SD Mean SD Mean SD Exam grade 5.67 1.62 5.40 1.44 5.61 1.90 500 M. Sacristán-Díaz et al. Downloaded by [Universidad de Sevilla] at 04:16 02 November 2017 evening classes), gender and whether the student was repeating the subject (i.e. previously enrolled but failed the course). Statistical methods of analysis Two types of analysis have been carried out: Comparison of scenarios: analysis of differences of proportions or means (according to the performance indicator). Analysis of assessable activities as a predictor of academic performance: logistic regression and analysis of linear correlations (according to the performance indicator) in the two most active scenarios (S1 and S2). Regarding the first type of analysis, proportions tests and post hoc proportions tests were done to assess the statistical significance of differences in percentages between scenarios (performance and success rates). For the performance indicator exam grades, an ANOVA was conducted to compare the means of the three scenarios. The Levene test was carried out to verify the equality of variances in the case of exam grades. Since the assumption of equal variances is not met, the robust statistical Welch test was performed. The Games-Howell method was used for post hoc comparisons not assuming equal variances (to determine the scenarios between which the differences can be seen). For the second type of analysis, a logistic regression was carried out for the ‘pass the course’performance indicator (categorical variable: 0 = No, 1 = Yes), and the grades for the assessable activities were correlated with those for the examination. Table 3shows the variables and observations used in the logistic regression for S1 and S2. All statistical assumptions of logistic regression (Peng, Lee, & Ingersoll, 2002) were checked for. Results and discussion Comparison of scenarios Performance rate The proportion test shows that differences between the three scenarios with regard to the performance rate can be considered statistically significant (p= 0.000), i.e. the change to more active methodologies influences the number of passes over total Table 3. Control Variables and observations used in regressions. Variables Groups Codification Frequency S1 S2 Shift Morning 1 278 310 Evening 0 142 174 Gender Female 1 243 278 Male 0 177 206 Student repeating subject Not repeating 1 285 310 Repeating 0 135 174 Doing assessable activities Yes 1 233 394 No 0 187 90 Innovations in Education and Teaching International 501 Downloaded by [Universidad de Sevilla] at 04:16 02 November 2017 enrolled students. The post hoc proportions tests show that only the comparisons of S2 with S0 and S1 are statistically significant (p= 0.001 in both cases). This means that S2 presents a better performance rate than the other two scenarios. The theoretical–practical assignment methodology (S1) therefore does not seem to help improve the number of passes over enrolled students. Success rate Proportions tests were conducted for the percentage of passes over the number of students who sat the exam in the corresponding academic year. No significant differences can be seen (p= 0.920), which indicates that the use of more active methodologies does not help to increase the success rate, despite the continuous assessment tests (S2) significantly increasing the percentage of exam shows. Exam grades The ANOVA test indicates that there are statistically significant differences between the means from one scenario to another. The post hoc analyses show that there is only a significant difference between the exam grades for scenarios S0 and S1 (p= 0.044). This difference is, however, contrary to what had been expected: as can be seen in Table 2, the highest mean exam grade is for S0 and the lowest for S1. To summarise, it can be stated that the shift to more active methodologies does not guarantee an improvement in academic performance when measured by the success rate or the exam grade. However, an important finding is that in the teaching and assessment scenario with the most active methodologies used, i.e. S2, in which continuous assessment tests were done, the percentage of passes over the total number of students enrolled, i.e. the performance rate, significantly increases when compared to both a single examination (S0) and theoretical–practical assignments (S1). The same is not true of the teaching tools used in S1 compared to S0, since in that case the number of students sitting the exam did not change significantly compared to S0. This means that, for the course and context considered, where there are high noshow rates and students find it difficult to pass tests, the continuous assessment tests (S2) appear to be an appropriate way of improving academic performance. They may not result in higher grades in the exam, or increase the percentage of passes over students that sit the exam, but they do have a clear effect on the rate of students that sit the exams, i.e. they significantly reduce the no-show rate and consequently increase the number of passes in absolute terms, by increasing the rate of passes over enrolled students. Assessable activities as a predictor of academic performance This section analyses the strength of the relationship between the optional assessable activities and academic performance in S1 and S2. Pass the course We first focused on the pass variable as the academic performance indicator. The variable that indicates student participation in each year’s activities and the three 502 M. Sacristán-Díaz et al. Downloaded by [Universidad de Sevilla] at 04:16 02 November 2017 control variables were included as independent variables. What is being tested is the extent to which his/her doing the activities can help predict or explain a student’s passing the course. Tables 4and 5set out the basic data for the logistic regression. In these, the Omnibus tests on the model coefficients show that there are significant improvements in the models over the reference models for each regression, and the Hosmer– Lemeshow goodness-of-fit tests show that the models are also significant. According to the Cox and Snell and Nagelkerke pseudo R 2 , the percentage of variance explained by the models is about 9.5–14% in S1, and about 16–24% in S2. The predictive ability of the two models is 72.9% in S1 and 80.8% in S2, representing a clear improvement over random classification. With regard to the predictors, in the upper halves of Tables 4and 5, it can be seen that only the variable of interest (doing assessable activities) and the constant in S1 (but not in S2) are statistically significant in the resulting models. The control variables (shift, gender and student repeating subject) are not significant. The regression coefficient (column B) for the doing assessable activities variable is greater than zero in both cases, which indicates that it has a positive effect on passing. Nevertheless, there is a difference in the strength of the impact in the two cases. In fact, when the reasons for the advantages (column Exp(B)) are compared in the two tables, the pass advantage of students who do the assessable activities compared to those who do not do them is noticeably greater (over double) in S2 than in S1 (specifically, 9.9 times for S2 and 4.4 for S1). Exam grades For S1 and S2, the grades for the assessable activities have been correlated with those for the examination. Both correlation coefficients are statistically significant, but there is a higher correlation between the exam grade and the continuous assessment test grade (r= 0.54; p< 0.001) than the theoretical–practical assignment grade (r= 0.31; p< 0.001) and it therefore has a higher predictor value. The continuous assessment test grade can explain up to 30% of the examination grade’s variance, whilst the assignment grade only explains 9%. Table 4. Logistic regression analysis of passes in S1. Predictors BSE Wald df Sig. Exp(B) 95% CI for EXP(B) Lower Higher Shift −0.367 0.268 1.881 1 0.170 0.693 0.410 1.171 Gender −0.278 0.244 1.300 1 0.254 0.757 0.470 1.221 Repeating subject 0.237 0.269 0.778 1 0.378 1.268 0.748 2.148 Doing activities 1.476 0.261 31.983 1 0.000 4.376 2.624 7.299 Constant 0.633 0.253 6.282 1 0.012 1.883 Fit tests χ 2 df Sig. Variance explained Omnibus tests 42.052 4 0.000 Cox and Snell R 2 Nagelkerke R 2 Hosmer & Lemeshow 4.217 6 0.647 0.095 0.140 Innovations in Education and Teaching International 503 Downloaded by [Universidad de Sevilla] at 04:16 02 November 2017 Conclusions This study has analysed two research questions: (1) Do more active T&AM help to improve academic performance? The results indicate that grades do not improve in any of the two most active scenarios analysed compared to the control scenario. In fact, there has even been a fall in the case of S1, which can be attributed to students’overconfidence due to the marks received in the assessment activities. However, we did find that S2 significantly reduces the number of students who drop out of the subject throughout the course and obtains a higher pass rate with regard to enrolled students (but not with respect to exam shows). S1, however, fails to have any effect in this regard. The overall conclusion is that a more active type of scenario can result in improved academic performance, but it is the choice of suitable activities for implementing active T&AM that could mark the difference (Tang & Austin, 2009; Wang, 2008). S1 does not increase the number of passes and, in addition, worsens the exam grades, so it is not a suitable scenario for our OM course. However, although S2 fails to improve exam grades, it is able to increase the number of passes. Therefore, in our case, multiple choice tests throughout the course have proven to be much more suitable than periodic assignments. These findings are in line with some prior studies. For instance, Peters, Kethley, and Bullington (2002) found that requiring graded homework had a negative effect on exam performance, and Douglas, Wilson, and Ennis (2012) argued that multiple-choice tests work more effectively when used in conjunction with other methods. (2) To what extent can student involvement in more active T&AM predict academic performance? We found that student participation in the proposed assessable activities increases the likelihood of passing and, to some extent, predicts their exam grades. These results are significantly higher in S2 than in S1. Specifically, there is twice the likelihood of passing and the predictive power of the assessable activities grade is more than three times higher. This study presents some limitations that should be borne in mind when assessing and interpreting the results. One limitation is that participation in assessable activities was voluntary and therefore its effect on performance cannot be accurately measured. For this reason, no comparisons have been made between the students Table 5. Logistic regression analysis of passes in S2. Predictors BSE Wald df Sig. Exp(B) 95% CI for EXP(B) Lower Higher Shift −0.006 0.256 0.001 1 0.981 0.994 0.602 1.641 Gender −0.378 0.246 2.355 1 0.125 0.685 0.423 1.100 Repeating subject 0.198 0.253 0.608 1 0.435 1.218 0.742 2.002 Doing activities 2.298 0.269 73.211 1 0.000 9.952 5.879 16.846 Constant −0.494 0.292 2.870 1 0.090 0.610 Fit tests χ 2 df Sig. Variance explained Omnibus tests 86.157 4 0.000 Cox and Snell R 2 Nagelkerke R 2 Hosmer & Lemeshow 10.224 7 0.176 0.163 0.243 504 M. Sacristán-Díaz et al. Downloaded by [Universidad de Sevilla] at 04:16 02 November 2017