Impact of a Gamification Learning System on the Academic Performance of Mechanical Engineering Students

Proofs Impact of a Gamification Learning System on the Academic Performance of Mechanical Engineering Students* ROSA PA `MIES-VILA ` Department of Mechanical Engineering, Universitat Polite `cnica de Catalunya, Av. Diagonal, 647, 08028, Barcelona, Catalonia, Spain. E-mail: [email protected] ALBERT FABREGAT-SANJUAN Department of Mechanical Engineering, Universitat Rovira i Virgili, Av. Paı¨sos Catalans, 26, 43007, Tarragona, Catalonia, Spain. E-mail: [email protected] JOAN PUIG-ORTIZ and LLUI ¨SA JORDI NEBOT Department of Mechanical Engineering, Universitat Polite `cnica de Catalunya, Av. Diagonal, 647, 08028, Barcelona, Catalonia, Spain. E-mail: [email protected], [email protected] ANTONI HERNA ´NDEZ FERNA ´NDEZ Institute of Education Sciences, Universitat Polite `cnica de Catalunya, Plac¸a Eusebi Gu ¨ell, 6, 08034, Barcelona, Catalonia, Spain. E-mail: [email protected] This study examines the effects of using a gamification tool as a teaching strategy. Specifically, Kahoot! is evaluated as a tool for enhancing student learning. The activities were part of the laboratory sessions of the subject Mechanism and Machine Theory during two consecutive academic years. We analyze the effect of a gamification learning system on both, students’ grades and motivation, in a course with a large number of students (n 1 = 283 students, n 2 = 306 students). The students were divided into three different groups (control group, gamification group and writing group) and their results were evaluated depending on the learning method applied during the class. In terms of gamification, this project introduces real-time feedback to stimulate the interest of students and help them use the typical tools and methodologies of game-based learning. The analysis of their performance in the laboratory exam shows significant differences between the group that used gamification and the groups that did not. The results suggest that gamification in engineering lab activities has a positive effect on students’ motivation and learning outcome. The study concludes that game-based elements and competitive activities enhanced student performance. Keywords: gamification; game-based learning; higher education; mechanical engineering 1. Introduction The combination of teaching and games can be traced back to the humanistic approach, but in recent years game design elements have started to be used for non-playful purposes [1]. Although the term is still being revised conceptually – see [2] for a theoretical review – gamification can be defined as using game-based mechanics, aesthetics and thinking to engage people, motivate action, promote learning, and solve problems [3]. In education, the idea is to motivate and stimulate students by using activities other than traditional ones, and facilitate – almost without them being aware – teachinglearning itself, especially in a social context in which student engagement needs to be increased [4]. There is a broad debate among game designers, researchers and educators, about what games are, how they impact individuals and, in general, how they can be used in classrooms. Insufficient attention has been paid to gamification grounded in both theories and evidence from empirical studies [5]. In this regard, [6] describes the advantages and disadvantages of gamification Among the advantages, he says, are that games and gamification can lead to high levels of learner engagement and motivation since they connect with the skills of 21st-century students [7]. On the other hand, there is a risk of applying reproduction without prior design, resulting in problems such as exploitation or the creation of hostile and tense environments. Gamification models in education domain could help gamification practitioners to make new strategies in learning activities to increase students’ motivation, achievement and involvement[8]. Rigorous studies are required to fully examine the effects of gamification and determine how learning is best achieved [9]. In general, gamification techniques have positive effects on the involvement and motivation of students [4, 10, 11], – see [12] for scoping review –. Students value its competitive nature, the immediacy of feedback on their knowledge and structured opportunities for further discussion [13] and they also identify gamification as a multifaceted tool for a great learning experience [14]. Gamified learning environments contribute to the learning and teach- * Accepted 10 July 2022.1434 IJEE 4266 PROOFS International Journal of Engineering Education Vol. 38, No. 5(A), pp. 1434–1442, 2022 0949-149X/91 $3.00+0.00 Printed in Great Britain #2022 TEMPUS Publications. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 ing process by raising levels of engagement, creating enjoyable learning environments and ensuring active participation [11, 15–20]. However, some studies have not identified any significant effects on learning or have even detected worse academic results when students are forced to use game elements [21–23]. However, all reviews to date agree that there is insufficient evidence to support the long-term benefits of gamification in educational contexts [1, 13, 17, 24, 25], so more empirical evidence is needed to justify that gamification is better than other pedagogical alternatives [26, 27]. Therefore, the present study aims to provide new evidence on the effects of gamification in the classroom. 1.1 Gamification with Personal Response Devices Personal response devices (PRDs) – sometimes called classroom response systems, student response systems, or audience response systems – appeared at the beginning of the 21st century and they consist of an emitter and a receiver that, together with the corresponding software, enable teachers to ask their students a multiple-choice question (the question is projected on a screen) and students to send an answer using their individual control or clicker [28, 29]. Clickers provide a simple way to generate an atmosphere of student interaction that can enhance teacher-student communication [30]. Several PRDs can be used with iPads, Android tablets, mobile phones and computers. These new systems have the same utilities as other fast response methods such as clickers, none of the corresponding technical-logistical problems, and new features like gaming elements, music, modalities, and design [28, 31] Usually, the integration of this devices do not present technical difficulties and gaming is successful in enabling active participation and interactive learning [13]. Some of these applications are Mentimeter, Infuse Learning, Socrative, Quiz Socket, Kahoot!, Verso, Poll Everywhere or VoxVote, which enable you to prepare multiplechoice questionnaires, true/false questionnaires and, in some cases, questions with short answers (i.e. Socrative). With these tools, students answer all questions simultaneously in class, data is collected and statistics on the responses of the students are given immediately. The timing is programmed by the teachers, who can detect common errors, highlight aspects that are most deficient for the students and provide immediate feedback [32]. The individual response system creates an environment of immediate interactive learning and discussion in the classroom [33, 34]. It also provides formative feedback on learning (for both teachers and learners) [35]. However, the benefits of using student response systems are also controversial [36, 37]. While there is considerable evidence to suggest that university students have very positive opinions about the use of these systems [28, 30, 33], some studies conclude that these tools do not guarantee better learning [35, 38]. It seems that it is the implementation of pedagogical strategies in combination with the technology that ultimately influences student success. Studies on individual response systems often compare the statistics and feedback given with the student’s final grade. The correlations are often positive but weak, which shows that they can be useful for formative assessment but not for summative [30, 35, 39]. Kahoot! is a free virtual tool that has gained in popularity among teachers for its user-friendly nature and its ability to establish working dynamics in the classroom. It is highly appreciated by students [28]. Kahoot! allows teachers to create surveys, questionnaires, puzzles and debates, and obtain students’ answers in real time. Various studies on Kahoot! agree that this tool improves participation and the positive relationship between class members [13, 39–41]. 1.2 Case Study in Mechanical Engineering Mechanism and Machine Theory is a core subject taught in the fourth semester of the Degree in Industrial Engineering at Universitat Polite `cnica de Catalunya. It is one of the first times that the Industrial Engineering students have come into contact with the world of mechanical engineering. The formative assessments from previous semesters showed that the students did not acquire the required skills at laboratory classes: the percentage of students who passed the laboratory exam was very low, and the teachers considered it a problem since it suggests that students were not able to put into practice the knowledge they had acquired in the theory classes. On average, the percentage of students passing the course is 70%, whereas the percentage passing the practical examinations is 40%, notably lower. Therefore, a new method was needed in order to improve the teaching/learning process. We hypothesize that the introduction of gamified feedback will help to highlight the most important concepts at the end of each laboratory session, and therefore, improve the learning process. Another important issue is which of the factors involved in gamification is most related to the improvement in learning (if indeed there is an improvement). To address this issue using a multifactorial approach [42], we differentiated two factors: the fact that students receive feedback (this will be checked by a control group doing written tests) and the fact Impact of a Gamification Learning System on the Academic Performance of Mechanical Engineering Students 1435 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 that gamification promotes other variables (motivation, engagement, competitiveness, etc.). For this reason, the second hypothesis of this study is that the first of these two factors (gamification) is more important than the second (the feedback itself). In order to test this hypothesis, the questions that the students in the gamification group were asked were also presented to another experimental group in which students did a written test without using Kahoot!. The solutions of the test (feedback) were also provided after the laboratory session. 2. Methods The present study uses an empirical-analytical methodology to study gamification as a tool in laboratory sessions. The subject Theory of Machines and Mechanisms has a large number of students each semester (between 270 and 320) so the students were randomly distributed into 11 laboratory groups taught by 4 different lecturers. The aim of our intervention was to improve learning in the laboratory sessions. The overall course grade is calculated according to the following weighted average, rounded to one decimal place: Mcourse ¼Maxð0:6Mfe þ0:2Mpe;0:8MfeÞ þ 0:10 Mlab1 þ0:10ðMlab2 MsimÞ1=2ð1Þ where, Mcourse is the final grade for the course, Mfe is the mark for the final exam, Mpe is the mark for the midterm exam, Mlab1 is the mark for the first laboratory exam (assessing sessions 1, 2 and 3), and Mlab2 is the mark for the second laboratory exam (assessing sessions 4 and 5). Finally, Msim is the mark for a simulation exercise. The interventions took place during the second term of the academic years 2016–17 and 2017–18 and aimed to improve the marks for 3 laboratory sessions which account for 10% of the final grade. A test questionnaire has been introduced as a feedback tool. Quick feedback helps students become aware, and they have greater perception of what has happened in the laboratory. This feedback has been introduced as a test questionnaire that has to be answered in the last 15–30 minutes of each session. Two different feedbacks are analyzed. The first uses Kahoot! questionnaires. Since Kahoot! is a fast response system for the student, it is expected to be effective at improving knowledge retention and skill acquisition. The second uses a traditional questionnaire which, therefore, involves no competition or cooperative learning. To determine the effect of introducing not only a feedback tool but a feedback gamification tool, the laboratory groups were divided into three groups: An experimental group given feedback through the Kahoot! questionnaires– (Gamification group, GG). These learners use the mobile version of the app. An experimental group given a written test at the end of the session (with the same questions as in Kahoot!), acting as reinforcement and feedback, but without the other components that Kahoot! may have (Writing group, WG). A control group subject to no intervention (Control group, CG). The students were divided up in this way to avoid teacher and timetable factors. Table 1 summarizes the number of students in each group. Note that some students do not participate in the laboratory sessions. Academic performance was assessed by comparing the marks of students in each of the pedagogical groups. The mean mark, standard deviation and number of students who passed the exam were calculated for each evaluation (M lab1 ,M lab2 ,M pe , M sim ,M fe ). A Student’s T-Test was also used to find significant differences between the experimental (GG and WG) and the control (CG) groups. Therefore, for the GG the relation between the Kahoot! test score and the grades in the other evaluations was studied. Likewise, for the WG, the relation between the writing test score and the grades in other evaluations was examined. To this end, linear correlations were calculated and Pearson, Spearman and Kendall coefficients determined. Finally, whether or not there was a teacher effect was studied (that is to say, whether a particular student gets a better or a worse mark depending on the teacher who has taught the subject). Therefore, the students were grouped according to the lecturer who taught the sessions and a Student’s T-Test was used to determine significant differences between the four groups. Finally, it was not considered appropriate to measure success only by comparing the summative marks, because this is not the only purpose of the gamification tool. A 12-question survey was preRosa Pa `mies-Vila `et al.1436 Table 1. Overall number of students for each group and academic year Number of students 2016–17 2017–18 Gamification Group – GG 37 41 Writing Group – WG 115 86 Control Group – CG 113 100 Not attending 41 56 Total 306 283 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 pared for students in the gamification group (See the Appendix). 3. Results Because feedback is now a part of laboratory sessions 1, 2 and 3, differences in the laboratory exam 1 marks (M lab1 ) can be expected among the three groups. During the academic year 2016–17, 62.16% of the students who took part in the gamification passed the exam while only 54.87% of the control group and 58.26% of the writing group did the same. Similarly, during the academic year 2017–18, 87.80% of the students who took part in the gamification passed the exam while in the control group and the writing group the percentages were 74.74% and 77.91%, respectively. Table 2 shows the mean and standard deviation of the marks for each evaluation (M lab1 ,M lab2 ,M pe , M sim ,M fe ) for both of the academic years analyzed. It can be seen that for laboratory exam 1 (M lab1 ), the mean grade obtained by the students who took part in the gamification sessions (5.59 2.43, academic year 2016–17; and 6.90 1.68, academic year 2017–18) is more than one point higher than the control group (4.50 2.17, academic year 2016– 17; and 5.75 2.30, academic year 2017–18). However, this difference is not so clear for the writing group (4.71 2.37, academic year 2016– 17; and 5.57 2.24, academic year 2017–18). A Student’s T-test between GG and CG demonstrated that there is a significant difference between these two groups (p-value < 0.05). Moreover, the differences between the WG and the CG group are not statistically significant. Fig. 1 shows the boxplot obtained for M lab1 for both academic years and for each teaching methodology. The central block is delimited by the position of Q1 and Q3 quartiles and the line representing the median is drawn in the box. It can be seen that the median is also higher for the gamification group than for the writing and control groups. Differences between GG, CG and WG are not presented for the other evaluation marks (M lab2 , Impact of a Gamification Learning System on the Academic Performance of Mechanical Engineering Students 1437 Table 2. Mean SD of the marks for each evaluation Group M lab1 M lab2 M pe M sim M fe Academic year 2016–17 GG 5.59 2.43* 4.22 2.97 5.24 2.34 7.16 2.16 2.88 1.82 WG 4.71 2.37 3.69 2.86 5.57 2.24 7.13 1.76 3.12 1.62 CG 4.50 2.17 3.49 2.54 5.53 2.37 7.06 1.84 2.90 1.87 Academic year 2017–18 GG 6.90 1.68* 4.94 2.36 5.90 2.76 7.06 2.23 4.94 2.30 WG 6.08 2.23 5.56 2.59 6.28 2.50 7.14 2.23 4.83 2.10 CG 5.75 2.30 4.95 2.78 5.92 2.11 7.26 2.27 4.32 2.16 *p-value < 0.01. Fig. 1. Boxplot of laboratory exam 1 marks. (a) Academic year 2016–17 (b) Academic year 2017–18. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 M pe ,M sim ,M fe ) (Table 2): the p-values are greater than 0.05 and therefore statistical differences cannot be assumed. This shows that students are randomly distributed among groups. Differences only appear when a gamification methodology is applied. Three different correlation coefficients (Pearson, Spearman and Kendall) and the p-values of the statistical tests were calculated. For GG, the pvalues were much lower than 0.05 (Table 3), so there is a significant positive correlation between the marks obtained in the Kahoot! test and the ones obtained in the laboratory 1 exam (correlations between 0.6 and 0.77 depending on the indicator used). However, the correlations for the writing group are really low and they are not significant (see Table 3). Furthermore, the relation between the feedback tests and the grades obtained in the other evaluations were studied using the same coefficients. The correlations in these cases were poor (0.180-0.337) and non significant. Fig. 2 shows, the relationship between the calculated Kahoot! grades (M Kahoot! ) and the grades obtained by students on the laboratory exam 1(M lab1 ) for the gamification group (GG). The graphs also show the polynomial regression line that adjusts these values and the corresponding R 2 parameter. The teacher effect was also analysed. As explained above, the sessions are taught by four different lecturers. For this analysis, students were grouped according to the lecturer who taught their laboratory session. However, the Student’s T-test does not detect any significant differences between the four groups studied (p-value > 0.05). Therefore, it cannot be affirmed that the teaching staff has an effect on the grades of the students. Finally, the opinion poll shows whether students see gamification as an improvement in their learning process and their motivation. The results for both courses were similar. They are presented in Fig. 3 where the dotted blocks represent a positive answer (strongly agree or agree) and lined blocks Rosa Pa `mies-Vila `et al.1438 Table 3. Correlation coefficients between feedback test and laboratory exam 1. M Kahoot! and M lab1 for gamification group and M WT and M lab1 for writing group. Coefficient Academic year 2016–17 Academic year 2017–18 Value (p-value) Value (p-value) Gamification Group Pearson 0.726 (3.63 10 –7 ) 0.774 (2.92 10 –9 ) Spearman 0.754 (7.34 10 –8 ) 0.709 (2.13 10 –7 ) Kendall 0.603 (7.32 10 –7 ) 0.615 (2.74 10 –7 ) Writing Group Pearson –0.010 (0.913) –0.101 (0.357) Spearman 0.003 (0.970) –0.174 (0.110) Kendall 0.003 (0.965) –0.136 (0.084) Fig. 2. Scatter graph of the lab exam 1 marks (M lab1 ) versus the Kahoot! marks (M Kahoot! ). (a) Academic year 2016–17, (b) Academic year 2017–18. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 represent a negative answer (disagree or strongly disagree).. In general, Kahoot! has been very well accepted. On the basis of the answers to the first three questions, it can be said that more than 90% of the students think that gamification has helped them to understand the subject which, in turn, makes them more motivated. Moreover, students confirm that the time spent on the activity is offset by improved learning (Q3). In terms of what the Kahoot! results demonstrate (Q4), students are divided: some agree that they are proof of knowledge acquired, but others do not. Similar results were obtained for Q5 and Q7. From these results, it is difficult to see if enough time was given to answer the Kahoot! questions and if Kahoot! questionnaires make them more attentive to the laboratory session. All students affirm that Kahoot! questionnaires will be positive in other subjects (Q6) and they all agree that discussion after they have given their answers allows them to clarify concepts (Q12). Most of them positively evaluate the feedback they get through these quizzes (Q11). According to 80% of the students, if Kahoot! grades were added to the summative evaluation grade they would pay more attention in class (Q8). A similar percentage perceives the competitiveness created by Kahoot! as a positive stimulus to learning (Q10). 4. Discussion There is a lack of research on the real effects of gamification on the learning process and whether these effects are better than those obtained with traditional approaches [43]. This paper presents an experimental study with 589 students enrolled. As has been mentioned, there was a need to elucidate whether Kahoot! is effective or not. The main objective was to contrast if gamification through game-based student response systems improves active student learning, participation and retention of concepts [30, 44], or on the contrary, it is no guarantee of better learning [35, 38]. So, the results presented here can be regarded as a pilot test which shows that game-based student response systems (Kahoot! here) can improve academic performance. In addition, this study also contributes to evaluate if both students and teachers think that gamification is stimulating, revealing, motivating and, in essence, fun [28]. Results show that the gamification group (GG) had a higher success rate in the laboratory exam (Laboratory exam 1) than the control group (CG). Moreover, on this evaluation, the average grade of GG students was statistically greater than the average of CG students. Furthermore, the grades of the other evaluations do not show these differences. It can be seen that gamification has a positive effect on grades as [30, 44] suggested. Note that the Impact of a Gamification Learning System on the Academic Performance of Mechanical Engineering Students 1439 Fig. 3. Results of the opinion polls for the 12 questions. (a) Academic year 2016–17, (b) Academic year 2017–18. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 writing group shows no significant improvement with respect to the control group. When the feedback is not gamified, it does not enhance academic results. These results suggest that gamification is the key to the improvement not the feedback itself. No significant differences were detected in the grades of the various groups who did laboratory exam 1. This reveals that it was not the lecturer of the laboratory session who marked the difference but the intervention itself. That suggest that the gamification is the key, not the lecturer. The results of the opinion poll show that the students value the intervention positively, as other studies have pointed out [28]. All of the students stated that the discussion after they had given their responses clarified concepts, and most of them felt that gamification helped them to understand the subject better and motivated them. These results are also in agreement with the literature [15–18]: gamification involves motivation. 5. Conclusions The main goal of this study was to analyse whether a gamification tool could improve academic performance and motivation in the laboratory sessions of the subject Mechanism and Machine Theory. For this purpose, during two consecutive academic years, we divided the students into three groups and each group had different methodological interventions. At the end of the first three sessions, the gamification group (GG) answered a Kahoot! questionnaire; the writing group (WG) answered the same questionnaire but on paper, and the control group (CG) did not take any questionnaire. In the light of the results presented, in general it can be concluded that gamification has provided a (modest) increase in the teaching-learning process in the laboratory sessions of the subject Mechanism and Machine Theory what it is so consistently with the theoretical reasoning that motivated this proposal. However, this study has several limitations and further research will be required. One limitation of the study is that we are not covering how gender differences influenced the effects of gamification. Gender and personality could affect students’ perception toward gamification activities. In addition, we have only use one interface in gamification. In future, similar applications should be tested to compare the obtained results. Additionally, there are different ways of gamified feedback mechanisms such as points, badges, reward, levels, etc. that are not covered in this study. More design investigation is required to further generalize the results. The study has presented a methodology that allow us to validate gamification as a tool to improve academic performance of mechanical engineering students. However, to generalize our results, the purposed methodology should be applied to other mechanical engineering subjects using Kahoot! or similar personal response applications. Acknowledgments – The authors would like to thank the students who participated in the study. This investigation has been partially funded by the Institute of Education Sciences from the Universitat Rovira i Virgili through the project A10/18 (Analysis, broadcast and internationalization of the improvement in the teaching/learning process through gamification).The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. References 1. C. Dichev and D. Dicheva, Gamifying education: what is known, what is believed and what remains uncertain: a critical review, Int. J. Educ. Technol. High. Educ.,14(9), 2017. 2. K. Seaborn and D. I. Fels, Gamification in theory and action: A survey, Int. J. Hum. Comput. Stud.,74, pp. 14–31, 2015. 3. K. M. Kapp, The Gamification of Learning and Instruction: Game-Based Methods and Strategies for Training and Education, San Francisco, California: Pfeiffer, 2012. 4. R. S. Alsawaier, The effect of gamification on motivation and engagement, Int. J. Inf. Learn. Technol.,35(1), pp. 82–93, 2018. 5. L. Ding, C. Kim and M. Orey, Design of gamified asynchronous online discussions, Technol. Pedagog. Educ., pp. 1–17, Aug. 2020. 6. K. Kapp, The Gamification of Learning and Instruction Fieldbook: Ideas into Practice, San Francisco, California: Wiley, 2013. 7. K. M. Kapp, The Gamification of Learning and Instruction Fieldbook: Ideas into Practice, San Francisco, California: Wiley, 2013. 8. G. P. Kusuma, E. K. Wigati, Y. Utomo and L. K. Putera Suryapranata, Analysis of Gamification Models in Education Using MDA Framework, Procedia Comput. Sci.,135, pp. 385–392, 2018. 9. D. Dicheva, C. Dichev, G. Agre and G. Angelova, Gamification in Education?: A Systematic Mapping Study, Educ. Technol. Soc., 18(3), pp. 75–88, 2015. 10. G. Huesca Jua ´rez and M. M. Reyes Carballo, Learning Gains , Motivation and Learning Styles in a Gamified Class, Int. J. Eng. Educ.,32(1), pp. 438–447, 2016. 11. E. Kim, L. Rothrock and A. Freivalds, An Empirical Study on the Impact of Lab Gamification on Engineering Students ’ Satisfaction and Learning, Int. J. Eng. Educ.,34(1), pp. 201–216, 2018. 12. K. A. I. J. U. N. Chew, Interaction Between Learning Assessment and Student Motivation?: A Scoping Review, Int. J. Eng. Educ., 37(5), pp. 1313–1329, 2021. 13. C. E. Holbrey, Kahoot! Using a game-based approach to blended learning to support effective learning environments and student engagement in traditional lecture theatres, Technol. Pedagog. Educ.,29(2), pp. 191–202, Mar. 2020. 14. B. Balakrishnan Nair, Endorsing gamification pedagogy as a helpful strategy to offset the COVID-19 induced disruptions in tourism education, J. Hosp. Leis. Sport Tour. Educ., 30 (September 2021), p. 100362, 2022. Rosa Pa `mies-Vila `et al.1440 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 15. L. De-Marcos, A. Domı ´nguez, J. Saenz-De-Navarrete and C. Page ´s, An empirical study comparing gamification and social networking on e-learning, Comput. Educ.,75, pp. 82–91, 2014. 16. N. Gordon, M. Brayshaw and S. Grey, Maximising gain for minimal pain: Utilising natural game mechanics, Ital. Innov. Teach. Learn. Inf. Comput. Sci.,12(1), pp. 27–38, 2013. 17. C.-H. Chen and H.-C. Yeh, Effects of integrating a questioning strategy with game-based learning on students’ language learning performances in flipped classrooms, Technol. Pedagog. Educ.,28(3), pp. 347–361, May 2019. 18. O. Baydas and M. Cicek, The examination of the gamification process in undergraduate education: a scale development study, Technol. Pedagog. Educ.,28(3), pp. 269–285, May 2019. 19. M. Papastergiou, Digital Game-Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation, Comput. Educ.,52(1), pp. 1–12, 2009. 20. C. Hosseini, O. Humlung, A. Fagerstrøm and M. Haddara, An experimental study on the effects of gamification on task performance, Procedia Comput. Sci.,196(January), pp. 999–1006, 2021. 21. M. D. Hanus and J. Fox, Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance, Comput. Educ.,80, pp. 152–161, 2015. 22. A. Domı ´nguez, J. Saenz-De-Navarrete, L. De-Marcos, L. Ferna ´ndez-Sanz, C. Page ´s and J. J. Martı ´nez-Herra ´iz, Gamifying learning experiences: Practical implications and outcomes, Comput. Educ.,63, pp. 380–392, 2013. 23. M. Bekk, R. Eppmann, K. Klein and F. Vo¨lckner, All that glitters is not gold: An investigation into the undesired effects of gamification and how to mitigate them through gamification design, Int. J. Res. Mark., no. xxxx, 2022. 24. C. Perrotta, G. Featherstone, H. Aston and E. Houghton, Game-based Learning: Latest Evidence and Future Directions (NFER Research Programme: Innovation in Education), Slough: NFER, 2013. 25. J. Hamari, J. Koivisto and H. Sarsa, Does Gamification Work? – A Literature Review of Empirical Studies on Gamification, in 47th Hawaii International Conference on System Science, pp. 3025–3034, 2014. 26. Z. Zainuddin, S. K. W. Chu, M. Shujahat and C. J. Perera, The impact of gamification on learning and instruction: A systematic review of empirical evidence, Educ. Res. Rev.,30, p. 100326, 2020. 27. L. Raitskaya and E. Tikhonova, Gamification as a field landmark in educational research, J. Lang. Educ.,5(3), pp. 4–10, 2019. 28. M. Ranieri, J. E. Raffaghelli and I. Bruni, Game-based student response system: Revisiting its potentials and criticalities in large-size classes, Act. Learn. High. Educ., pp. 1–14, 2018. 29. D. Bruff, Teaching with Classroom Response Systems: Creating Active Learning Environments, San Francisco: Jossey-Bass, 2009. 30. C. Fies and J. Marshall, Classroom Response Systems: A Review of the Literature, J. Sci. Educ. Technol.,15(1), pp. 101–109, 2006. 31. B. Andre ´s, R. Sanchis and R. Poler, Quiz game applications to review the concepts learnt in class: An application at the University context, in Proceedings of INTED2015 Conference, pp. 5654–5662, 2015. 32. V. J. Shute, Focus on Formative Feedback, Rev. Educ. Res.,78(1), pp. 153–189, 2008. 33. J. E. Caldwell, Clickers in the Large Classroom?: Current Research and Best-Practice Tips, CBE – Life Sci. Educ.,6(1), pp. 9–20, 2007. 34. C. B. Paschal, Formative Assessment in Physiology Teaching Using a Wireless Classroom Communication System, Adv. Physiol. Educ.,26(4), pp. 299–308, 2015. 35. S. W. Draper and M. I. Brown, Increasing interactivity in lectures using an electronic voting system, J. Comput. Assist. Learn.,20, pp. 81–94, 2004. 36. J. I. Castillo-Manzano, M. Castro-Nun˜o, L. Lo´pez-Valpuesta, M. T. Sanz-Dı´az and R. Yn˜iguez, Measuring the effect of ARS on academic performance: A global meta-analysis, Comput. Educ.,96, pp. 109–121, 2016. 37. N. J. Hunsu, O. Adesope and D. J. Bayly, A meta-analysis of the effects of audience response systems (clicker-based technologies) on cognition and affect, Comput. Educ.,94, pp. 102–119, 2016. 38. L. A. Van Dijk, G. C. Van Der Berg and H. Van Keulen, Interactive lectures in engineering education, Eur. J. Eng. Educ.,26(1), pp. 15–28, 2001. 39. M. Curto, L. Orcos, P. J. Bla ´zquez and F. J. Molina, Student Assessment of the Use of Kahoot in the Learning Process of Science and Mathematics, Educ. Sci.,9(1), p. 55, 2019. 40. S. G. Bryant, J. M. Correll and B. M. Clarke, Fun with pharmacology: Winning students over with kahoot! game-based learning, J. Nurs. Educ.,57(5), p. 320, 2018. 41. E. Zarzycka-Piskorz, Kahoot it or not? can games be motivating in learning grammar?, Teach. English with Technol.,16(3), pp. 17–36, 2016. 42. D. A. Gentile, The Multiple Dimensions of Video Game Effects, Child Dev. Perspect.,5(2), pp. 75–81, 2011. 43. O. Borras-gene and M. Martinez-nun, New Challenges for the Motivation and Learning in Engineering Education Using Gamification in MOOC, Int. J. Eng. Educ.,32(1(B)), pp. 501–512, 2016. 44. V. Simpson and M. Oliver, Electronic voting systems for lectures then and now: A comparison of research and practice, Australas. J. Educ. Technol.,23(2), pp. 187–208, 2007. Impact of a Gamification Learning System on the Academic Performance of Mechanical Engineering Students 1441 Appendix The 12 pool questions were: Q1. Has Kahoot! helped you to better understand the subject? Q2. Have you become more motivated because of Kahoot!? Q3. Is the time you invested in Kahoot! offset by how much you have learned? Q4. Does the Kahoot! score reflect your understanding? Q5. Did you have enough time to answer the Kahoot! questions? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 Rosa Pa `mies-Vila `et al.1442 Q6. Would you welcome the use of Kahoot! in other subjects? Q7. Did Kahoot! make you more attentive to the class? Q8. If the Kahoot! questions had more weight in the evaluation, would you have been more careful/attentive? Q9. Has Kahoot! improved your relationship with teachers? Q10. Is the competitiveness created by Kahoot! positive? Q11. Can Kahoot! help you acquire knowledge and clarify concepts? Q12. Does the discussion or clarification of your score clarify some concepts? Rosa Pa `mies-Vila `received her PhD. degree in Biomedical Engineering in 2012 from Universitat Polite `cnica de Catalunya, Spain. She is assistant professor at the Mechanical Engineering Department of UPC, where she has been since 2008. Her current research area includes studies on mechanisms and machine theory, specifically in the field of biomechanics of human motion. Moreover, she has expanded her research area to include emerging educational technologies and how gamification can transform traditional teaching methods. Albert Fabregat-Sanjuan is an Assistant Professor in the Department of Mechanical Engineering at the Universitat Rovira i Virgili (URV) since 2009. He has been working in the field of experimental and computational mechanics research and has taught several subjects related to mechanical engineering. As a coordinator of several education innovation projects, his particular interest focuses on implementing innovative methods of teaching, especially the use of gamification and service-learning. He has been awarded for the quality of his teaching methods in the collective (2015) and individual (2018) category from the URV Social Council. Joan Puig Ortiz received his PhD degree in Physics Science at Universitat Polite `cnica de Catalunya, Spain, in 2006. He is Collaborating Lecturer of the Mechanical Engineering Department of UPC. His research interests are in the area of mechanisms and machine theory and the new methods of teaching in STEM. Lluı¨sa Jordi Nebot received her PhD degree in Physics Science in 1999 from Universitat Polite `cnica de Catalunya, Spain. She is university lecturer at the Mechanical Engineering Department of UPC, where she has been since 1988. Her research interests are in the area of mechanisms and machine theory – specially, constant-breadth cam mechanism and bar linkages. She has expanded her research area to include new educational technologies and how transform traditional teaching methods. Antoni Herna ´ndez Ferna ´ndez has been an assistant professor of Physics and Educational Science at the Institute of Education Sciences (ICE–UPC) since 2008. He has also been a part-time teacher in Secondary School and Vocational Training at Terrassa Art and Design School since 1997. He has degrees in Physics (1997) and Linguistics (2003) and received his doctorate in Linguistics from the Universitat de Barcelona (UB) in 2014. His current research interests include education in a broad sense (gamification, project-based learning, technology education, etc.) and quantitative linguistics, i.e. the study of general patterns in communication systems (Zipf’s law, Menzerath-Altmann’s law, etc.). 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57