Understanding student behavior and perceptions toward earning badges in a gamified MOOC

Universal Access in the Information Society manuscript No. (will be inserted by the editor) Understanding Student Behavior and Perceptions toward Earning Badges in a Gamified MOOC Alejandro Ortega-Arranz1 ·Erkan Er1 ·Alejandra Mart´ınez-Mon´es2 · Miguel L. Bote-Lorenzo1 ·Juan I. Asensio-P´erez1 ·Juan A. Mu˜noz-Crist´obal1 Preprint submitted to Universal Access in the Information Society. Lincense: CC BY-NC-ND. Published Article DOI: https://doi.org/10.1007/s10209-019-00677-8 Published Article URL: https://link.springer.com/article/10.1007/s10209-019-00677-8 Abstract Despite the advantages of MOOCs, such as the open and free access to education, these courses are criticized for students’ lack of motivation and their high dropout rates. Gamification is a technique used to increase student motivation and engagement in smallscale educational contexts. However, the effects of gamification on student engagement have been scarcely explored in MOOC environments, and the findings so far are inconsistent. To address this gap, this research work examines the students’ behavior towards earning badges and how it relates to their engagement in a gamified MOOC. According to the results, the behaviors towards badges of the active students were generally positive and significantly correlated with other variables measuring their engagement (e.g., pageviews, submitted tasks, forum posts), although this positive behavior seems to decrease throughout the course. Additionally, students that reported high motivation by badges at the end of the course showed a higher engagement level than those that were not appealed by badges. Keywords Gamification ·MOOC ·Engagement · Perceptions ·Badges ·Rewards 1School of Telecommunications Engineering, Universidad de Valladolid, Paseo de Bel´en 15, 47011 Valladolid, Spain. 2School of Computer Engineering, Universidad de Valladolid, Paseo de Bel´en 15, 47011 Valladolid, Spain. Corr. Aut.: Alejandro Ortega-Arranz (0000-0002-8167-7157) E-mail: [email protected]a.es Tel.: +34 983 42 3698 1 Introduction Massive Open Online Courses (MOOCs) are being established as a form of global education that balances traditional and structured classroom-based environments and tools (e.g., questionnaires) with open resources available on the Internet (e.g., videos, social networks) [14, 55]. MOOCs have brought important benefits to the educational community: open access to learning contents offered by prestigious universities and institutions, the creation of learning communities around a shared topic or interest, etc. [17]. However, despite the substantial growth in the number of MOOCs as well as the number of students enrolling in them every year [53,54], low completion rates still remain as a significant issue [10, 31]. One reason behind the high dropout rates is the high diversity within MOOC learners’ personal goals and interests [1,38]. That is, as a consequence of its massive and open nature, a certain level of dropout can be expected in MOOCs [27,38]. Another important reason for dropouts in MOOCs is the learners’ lack of motivation and engagement, consequently failing to complete the learning activities and the course [32,33]. Failure of such learners could be diminished through effective pedagogical interventions such as those involving active learning strategies [29]. One relevant active learning strategy is gamification. Gamification is defined as the inclusion of elements and structures that frequently appear in games (e.g., leaderboards, badges, narrative) in non-game contexts [11,12]. This technique has been proven to be effective in promoting students’ engagement in different educational contexts (e.g., face to face or blended courses), 2 Alejandro Ortega-Arranz1et al. thus supporting their learning and achievement [11,13, 26]. Among the large amount of game design elements, badges are the most frequent mechanism used in both small-scale online courses and MOOCs to promote student motivation and engagement [13,47]. Badges are optional rewards, represented with graphical icons and issued when users satisfy predefined requirements typically associated with non-compulsory activities [16,24, 25]. Previous research has reported several benefits of badges for student learning in small-scale educational environments. For example, teachers can define and assign badges in a way that enables students to establish their learning goals and to progress in the course by achieving the badges linked with the learning activities [25]. Moreover, badges can help increase students’ sense of recognition based on their learning efforts and achievements, and therefore promoting their motivation and engagement [37]. Badges can potentially offer similar learning benefits in massive learning contexts such as MOOCs. However, the aforementioned benefits of gamification in small scale [15,16,19,21,30,45,52] should not be taken for granted in MOOCs, as they have their own distinct characteristics (e.g., massiveness and heterogeneity of participants, lack of instructor facilitation, reliance on automatic methods, etc.). There have been several efforts in analyzing the effects of badges in MOOCs but the research so far has reported inconclusive results, and needs to be complemented with new empirical studies [4,35,47]. In this regard, analyzing MOOC learners’ behavior and perceptions towards badges, and their relation with student engagement1is important as it may help understand the effects and consequences on the students of using badges in MOOC environments. Attending to the given gap, this research work focuses on MOOC learners’ behaviors towards earning badges and explores the relation of such behavior with their engagement in the course. Also, the research investigates the learners’ perceptions about badges. More specifically, the research question proposed to lead this study is: Which are the students’ behaviors and perceptions towards earning badges in a gamified MOOC? To address this question effectively, we have further subdivided it into two topics: (i) the learners’ behaviors towards earning badges; and (ii) the learners’ perceptions about badges and their relationship with their behavioral engagement. 1Although different authors have proposed multiple definitions of engagement in technology-mediated environments [28], in this study we will focus on the students’ behavioral engagement. According to Fredricks et al., behavioral engagement concerns the observable behaviors that represent the student involvement in learning such as participation, persistence or contributions [20]. In order to explore the aforementioned research question and associated topics, a study was conducted in a MOOC (1031 enrolled students) that incorporated 15 badges associated with different course activities. Differently from previous works, this study investigates badges in a highly heterogeneous set of MOOC learners. Additionally, the badges had to be claimed explicitly by the learners, thus providing new variables to understand their engagement. This paper is structured as follows. The next section provides a brief overview of existing research on badges in MOOC contexts and highlights the factors affecting the gamification design and enactment at massive scale. The subsequent section explains the design of the study including the context, participants, the gamification design, and the research methodology. Then, the findings from the analysis are presented and the results are discussed. The paper ends with some conclusions along with limitations and ideas for future research. 2 Related Work Previous studies in small scale educational contexts have shown the benefits of using reward strategies in improving learners’ motivation [16,19], engagement [15, 21,30,45], learning outcomes [16,30], and enjoyment [19]. However, MOOCs have specific features different from other educational environments (e.g., face to face or blended courses) which may have significant implications in how reward-based gamification strategies affect students [46]. First, the openness and massiveness features of MOOCs lead to a broad variety in participants’ background, knowledge, learning culture and goals as opposed to the limited diversity in formal education settings where the teachers can more easily recognize their students’ characteristics and goals. Therefore, MOOC instructors face challenges to design a variety of badges that could sufficiently challenge and engage a varying learner population without leading to the states of either boredom (over-simple) or anxiety (over-challenging). Adapting from the flow theory2, the badge-related conditions should be neither too easy nor too complicated to keep students inside the flow state and then to maintain their motivation throughout the course [9]. Second, according to Festinger [18], people tend to evaluate their abilities (as the ones that students need to satisfy for the badge conditions) by comparing them 2According to Csikszentmihaly, flow is defined as a state of absorption in one’s work characterized by intense concentration, loss of self-awareness, a feeling of being perfectly challenged and a sense that time is flying [9]. Understanding Student Behavior and Perceptions toward Earning Badges in a Gamified MOOC 3 with the abilities of others. Previous studies incorporating game elements that can be compared by participants (e.g., badges listed in a leaderboard), showed that such comparison usually reduces users’ performance rather than enhance it [58]. Although this drawback was already observed in other gamified educational contexts, the openness and massiveness of MOOCs are likely to increase the heterogeneity (e.g., interest on badges, previous knowledge) and the differences among students’ player profiles [6]. These larger differences can lead to demotivation when comparing others’ achievements in those students avoiding external rewards or with difficulties to earn them. Third, as a result of the massiveness, there is a need for implementing automatic rewarding approaches in MOOCs since the instructors cannot track participant actions individually and they cannot timely issue badges manually [16,21]. Therefore, the predefined conditions under which the rewards are issued are restricted to the students’ actions that can be tracked by the MOOC platforms. As a consequence, the gamification designs are typically limited, hindering the implementation of designs that previously showed positive impact on student learning and engagement. Additionally, in small-scale contexts, teachers can typically cope with the workload of manually assessing the quality of the student actions, thus opening the possibility of designing conditions based on such quality-related aspects of the actions (e.g., correctly answering to a peer question). However, in MOOC contexts, teachers cannot manually assess the quality of learning outcomes due to the massive number of participants. This limitation could be addressed in multiple ways including (1) automatic methods (e.g., natural language processing), or (2) peers taking the role of issuers to evaluate the quality of participant actions. Thus, according to the results of previous research, badges are a promising strategy to be used in MOOCs. However, the common features of this kind of courses could diminish their effectiveness. That is, there is a need for empirical studies regarding the use and effects of badges in MOOC contexts [4,35,47,59]. The most relevant empirical studies so far are described below. Anderson et al. investigated the use of badges to increase the participation in the discussion forums of a MOOC with more than 110,000 enrolled students. Results show that the badge system significantly increased forum participation and engagement compared to a previous run of the same MOOC [3]. Reischer et al. implemented rewards, badges, points, and a leaderboard to explore the effects of gamification on student activity in discussion forums of a MOOC with 605 enrolled students. Badges were awarded for students’ basic actions such as creating an account, receiving “likes”, or marking forum threads as favorite. Although the results showed a high level of user satisfaction, the reading and writing levels in the discussion forums decreased in comparison with the previous nongamified version of the same course [49]. Rizzardini et al. gamified a MOOC with 1,678 enrolled students using badges, leaderboard forums, leagues and redeemable rewards. Badges were used to promote student participation in discussion forums (e.g., receiving “likes” from peers). The gamified strategies used in the course did not lead to an increase in student engagement, although 78% of the students reported that they were more motivated because of the game elements [41, 50]. Kyewski and Kr¨amer performed a between-subjects experimental design about the effects of badges on motivation, performance and the number of days a student is active within a MOOC. A total number of 324 students were enrolled in an online course gamified with 4 different badges associated to 4 different types of activities (forums, peer reviews, quizzes and content resources). Results show that the badge design had no positive impact on students’ motivation and performance for that course [36]. Hakulinen et al. analyzed the effects of using badges in a course about data structures and algorithms offered in a learning online environment. Although this study is not explicitly focused on MOOCs, the number of participants (281) could make gamification have similar effects as in a MOOC environment. Results show a positive impact on students’ behavior such as early task submission and avoiding trial and error submissions [23]. Ruip´erez-Valiente et al. validated a set of indicators to model the student behavior towards badges and analyzed their relationship with other activity indicators. To do so, they gamified three courses that students take before starting their first year of a university degree in Khan Academy3, with 73, 167 and 243 students (most of them between 17-19 years old, enrolled to an engineering degree). Results show a positive correlation between the students intentionality towards badges (main indicator defined by the authors) and different activity metrics such as the time spent in the course, the number of completed exercises or the number of visualized videos [51,52]. Cross et al. analyzed the experiences and attitudes of MOOC participants towards badges based on the number of badges issued and a survey. In their gamification design, the badges had to be requested by 3Khan Academy: https://www.khanacademy.org/, last access: June, 2018. 4 Alejandro Ortega-Arranz1et al. students. Teachers and peers had to manually decide whether badges should be issued to the learners. Most students perceived badges as positive elements of the course with a variety of reasons. Moreover, results show that students’ interest in badges decreased over time [8]. Moreover, there are some research works [5,34,60] which have carried out quantitative and qualitative analysis regarding the effects of game elements on student engagement in MOOCs. However, these studies use other game elements different than badges: duels [60], engagement bars [34], and points, votes and goals [5]. In this context, although the works of Anderson et al. and Reischer et al. were carried out in real MOOC environments (i.e., open, massive, heterogeneous), they were limited to a quantitative analysis of the effects of badges on students’ forum engagement (e.g., voting, posting, receiving likes) without considering the student engagement in other learning activities such as quizzes, peer reviews or group activities (e.g., glossaries, resource sharing) typically implemented in MOOCs. Rizzardini et al. performed a similar work extending the gamification analysis to an overall course engagement. However, the analysis involved a set of game elements including badges, leaderboards, leagues and redeemable rewards without isolating the effects of each element independently. Also, in this study the relationship between the students’ behaviors towards badges and the student engagement is not explored. Conversely, Kyewski and Kr¨amer isolated the effects of badges in a MOOClike context. However, the analysis is limited to the student motivation, performance (grades) and days of activity without considering neither the student engagement nor its relationship with the student behaviors towards badges. Furthermore, Hakulinen et al. and Ruip´erez et al. performed a detailed analysis of the students’ behaviors towards badges and their relationship with the student engagement in a variety of activities. However, both studies were closed to open enrollments, limited to a non-large scale context where the students were homogeneous (i.e., similar background, age and culture), and in which badges were automatically issued even to those students not interested in gamification. Therefore, the aforementioned features of MOOCs that could influence the effects of badges were excluded in both studies. Finally, although the study of Cross et al. was carried out in a real MOOC environment where students had to claim the badges, the analysis is rather limited to the number of badges issued and to the student general opinions and attitudes towards badges without considering the effects of badges on student engagement. As already mentioned, there is a growing number of studies proposing and analyzing the use of badges in MOOCs, however the scarcity of empirical works suggests that gamification in MOOCs is still in its infancy [47]. Differently from the previous research works, this study focuses on the analysis and relationships between the students’ behaviors towards badges, the students’ behavioral engagement and the students’ perceptions about badges. Moreover, this study is performed in a real MOOC environment with a heterogeneous set of students (1031 enrolled students) ranging from younger than 20 to older than 60 years old, from different countries and with different educational background. Finally, students had to claim the badges after fulfilling the requirements, being issued automatically, providing an extra variable to model the student behavior towards earning badges. Thus, this study can help shed some light on understanding the student behavior and perceptions towards earning badges and their relationship with the student engagement in a real MOOC context. 3 Overview of the study The study has been conducted within a MOOC provided by the University of Valladolid in the Canvas Network MOOC platform4from the 6th of February to the 3rd of April, 2017. This section describes the course, the gamification design and implementation, the research methods used, and general information about the course enactment. 3.1 Course Overview The topic of the MOOC is about translation from English to Spanish in the business and economic fields, offered in Spanish. The course was an 8-week instructorled MOOC divided into 7 weekly content modules. One extra week at the end of the course was provided to allow students complete the last activities. Technical and teaching support was offered by the course team (i.e., teachers and researchers) through private messages and posts in the forums. The modules included videos, learning content pages, recommended readings, discussion forums and individual and collaborative activities (e.g., quizzes, term extraction in groups) [48]. Figure 1shows the activities and their relationship with the different badges implemented in the course. The activities can be classified into compulsory and optional. Students had to submit all the compulsory activities in order to receive the course completion certificate. A detailed description of each activity can be found in [48]. For all activities, the submission was due 4Canvas Network: https://www.canvas.net/, last access: June 2018. Understanding Student Behavior and Perceptions toward Earning Badges in a Gamified MOOC 5 Week1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 W. 8 Quiz 0 Quiz 1 Text Analysis Group Term Extraction Text Translation Group Term Extraction Quiz 4-6 Forum Introduction Collab. Glossary External Texts Search Text Translation Text Analysis Peer Review Group Term Extraction Peer Review Text Analysis Peer Review Text Translation Peer Review Group Term Extraction Peer Review Text Translation Text Translation Peer Review Text Translation & Auto Evaluation Special badges can be claimed after earning the 3 badges of the same type Fig. 1 Course activities distributed by release week and their relationship with badges. Dark and light gray cells indicate the compulsory and optional activities respectively. within two weeks after the release of the activity, except in the case of peer reviews, which were required to be completed in a week. Quizzes were set as compulsory activities where students should score at least 5 out of 20 points. Furthermore, the course enrollment was closed in the second week of the course to avoid group management problems in the collaborative activities. 3.2 Gamification Design and Implementation The gamification design was composed of two game elements: a leaderboard and badges. The leaderboard was designed to share students’ badge achievements with all course participants. The intention of including a leaderboard was to allow students to compare their progress with other students. The badges in this course were implemented using the Badgr platform5, a badge recognition and tracking system to store, issue, organize, and share Open Badges6. The Canvas Network platform integrates Badgr by means of IMS LTI7compliant interfaces, allowing the course team to choose among the different goals to be gamified through the Canvas Network user interface. Figure 2illustrates the fifteen badges implemented in the MOOC and the conditions that students had to fulfill to earn each of them. All badges and their conditions were co-designed and configured with the teachers aiming to increase student engagement and to encourage students to participate in the activities throughout 5Badgr: https://info.badgr.io/, last access: June, 2018. 6Open Badges: https://openbadges.org/, last access: June, 2018. 7Learning Tools Interoperability: https://www.imsglobal.org/activity/learning-toolsinteroperability, last access: June, 2018. Fig. 2 List of badges implemented in the course and the conditions to be issued. the course. As Figure 2shows, badges can be classified on suites based on the associated type of activity. For example, the Rookie Reviewer,Intermediate Reviewer, and Advanced Reviewer badges were issued for completing specific peer review activities. In order to make the badge suites clearer to the students, the course team used different colors to identify increasing levels of badges resembling the gold/silver/bronze levels typically employed in games [16,23,57]. Also, there were three badges that could be obtained when students collected all the badges of a specific suite: Quiz Master dependent on Quiz0,Quiz1, and Quiz6 ;Top Colleague dependent on Welcome,Good Colleague, and Awesome Colleague; and Expert Reviewer dependent 6 Alejandro Ortega-Arranz1et al. Fig. 3 Screenshots of the badge tab from the students’ view. on Rookie Reviewer,Intermediate Reviewer, and Advanced Reviewer. All badges were associated to optional activities, except for two that were associated with the group activities (Good Colleague and Awesome Colleague). This design decision was important to ensure that student behavior towards badges could not be attributed to a side effect of the students’ motivation to get the final certificate. A gamification tab was placed in the course panel (see Fig. 3) to allow the students to easily check the badges earned, to read the conditions to earn them, and to track their badge achievements in the leaderboard. In this study, students were requested to claim the badges once they had fulfilled the associated conditions by submitting a summary of the associated task and afterwards visiting the gamification tab. Canvas Network is able to check if the conditions are fulfilled and if so, send the information to the Badgr system in order to issue the badge and display it in the student interface. The information about badge descriptions and the claiming process was provided to the students at the beginning of the course in the course description page. Additionally, a short reminder was included in the descriptions of the course activities. It is important to mention that the course team wanted to use more complex conditions to gamify the course and to engage the participants, but the used platform presents some limitations. By default, Canvas Network allows to set a small number of students’ actions performed within a course as conditions (e.g., posting in a forum or submitting a task). However, the platform hinders the inclusion of more complex conditions that could motivate students such as providing a badge regarding the number of comments submitted by a student in a peer review. Moreover, some activities were implemented through external tools, such as the collaborative glossary with Google Forms, and the terms introduced by the students in the glossary cannot be tracked by Canvas Network. As a consequence, students were requested to copy the terms and provide a summary of the activities in a Canvas submission page in order to gamify them. A similar approach has already been followed in other gamification studies [8,16]. 3.3 Research Question To better answer the proposed research question, Which are the students’ behaviors and perceptions towards earning badges in a gamified MOOC?, we have conducted an anticipatory data reduction process during the evaluation design 8[40]. Thus, an issue [56] has been defined as a conceptual organizer of the evaluation process: Which are the students’ behaviors and perceptions towards earning badges in the MOOC of the study? This issue has been divided into two topics and further subdivided in various informative questions: (topic 1) the learners’ behaviors towards earning badges; and (topic 2) the learners’ personal perceptions about badges and its relationship with their behavioral engagement. Figure 4illustrates this anticipated data reduction approach followed. 3.4 Methods This research employs a mixed-method design [22]. Qualitative and quantitative data were collected and the results of the analysis were triangulated to better understand the relationships between the learners’ behavior and perceptions towards badges with their engagement. The data sources were: –Canvas Network logs: these logs were retrieved from the MOOC platform. Logs contain information about the participants and their interactions with the course activities. They also contain general information about 8According to Miles and Huberman [40], data reduction refers to the process of selecting, focusing, abstracting and transforming the data that appear in written-up fields notes or transcriptions. This data is advised to be divided into topics and subtopics at different levels of analysis deciding the conceptual framework, cases, research questions and data collection approaches to choose [40]. Understanding Student Behavior and Perceptions toward Earning Badges in a Gamified MOOC 7 Research Question (RQ) Issue Topic 1 Students’ Behaviors Topic 2 Students’ Perceptions IQ.1.1.What is the rate of students who satisfied the badges conditions and the students who requested and earned such badges? IQ.1.2. What is the time span between the moment the students satisfy the conditions and the moment they claim the badges? IQ. 3.1. Which are the students’ perceptions towards badges? IQ. 3.2. How do students’ perceptions relate to their behavioral engagement? IQ. 1.3. What is the role of students’ behaviors towards earning badges on their behavioral engagement? RQ: Which are the students’ behaviors and perceptions towards earning badges in a gamified MOOC? Issue: Which are the students’ behaviors and perceptions towards earning badges in the MOOC of the study? Fig. 4 Anticipatory data reduction schema showing the research question, issue, topics and informative questions. the course such as the total number of enrolled students, the active students per week, and the number of students that completed the requirements to earn a badge. These information allowed us to estimate the students’ behavioral engagement based on the page views, tasks submitted, forums posts and activity time in the course; –Badgr log: this log was retrieved from the gamification platform. The log contains information about the students that were issued with badges along with their date stamps; –End-course questionnaire: at the end of the course, students were asked to complete a questionnaire containing 12 items to help understand students’ perceptions about badges (see Section 4.2). The questionnaire also included open-ended questions to qualitatively analyze their perceptions. Before releasing the questionnaire, the items were assessed by four researchers and one student of the course regarding the relevance of the questions for this research and their understandability. 3.5 Course Participants In total, 1031 students were enrolled in the course. A welcoming questionnaire was administered in the first week of the course to obtain information regarding students’ profiles. The questionnaire was completed by 668 students. Most of the participants were women (75.75%), between 20-30 years old (61.23%), with a university degree (53.29%) and were living in Spain (56.89%). Further information about students’ profile regarding their age, gender, background and location is shown in Figure 5. Fig. 5 Course students’ statistics regarding the gender, age, background and location. 4 Results In this section, we present and interpret the results regarding the two topics described in the previous section. 8 Alejandro Ortega-Arranz1et al. Badge Welcome Quiz 0 Quiz 1 Glossary Searcher Translator Good Col. Rookie Rev. Int. Rev. Awes. Col. Quiz 6 Adv. Rev. Top Col. Quiz Master Exp. Rev. Release week 1 1 2 2 3 4 4 5 6 6 7 7 6 7 7 Active stu. 689 689 312 312 194 177 177 171 170 170 161 161 Perform. stu. 302 247 240 122 92 108 147 116 96 144 141 94 102 107 59 Issued stu. 282 227 191 112 84 96 126 103 87 117 117 80 91 94 53 Ratio info (%) Active/issued 40.93 32.95 61.22 35.89 43.30 54.23 71.19 60.23 51.18 68.82 72.67 49.69 Perf./issued 93.38 91.90 79.58 91.80 91.30 88.89 85.71 88.79 90.63 81.25 82.98 85.11 89.22 87.85 89.83 Span info (days) Median 1.00 0.00 0.00 0.00 0.00 1.00 1.00 0.00 1.00 1.00 0.00 0.00 1.00 0.00 0.00 Mode 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Mean 3.18 1.90 2.34 3.00 3.05 4.65 5.34 3.75 3.53 2.29 0.95 0.55 2.32 1.09 0.60 Std. 3.72 4.34 6.03 5.67 5.83 6.83 6.96 6.05 4.55 2.65 1.93 1.35 2.48 2.05 1.52 95% conf. int. 0.43 0.56 0.85 1.05 1.25 1.37 1.22 1.17 0.96 0.48 0.35 0.30 0.51 0.41 0.41 Table 1 Information about the number of students per category, their ratio and descriptive statistics regarding the claiming time span per badge. Fig. 6 Number of active students, students who fulfilled the conditions (performing students), and rewarded students (issued students). 4.1 Topic 1: Students’ Behaviors towards Earning Badges In this study, the student behavior towards earning badges was modeled by two variables: the number of badges issued, and the claiming time span (i.e., the number of days a student waited for claiming a badge after the conditions to earn it were satisfied)9. 9Badges associated to compulsory group activities (i.e., Good colleague and Awesome colleague were included in the analysis but their results could differ from the other badges since: (i) they are compulsory tasks and therefore, students could fulfill the conditions without being motivated by badges, and (ii) the submission of these activities (one 4.1.1 Issued Badges In this subsection we analyze the number of badges issued and the ratio of students who earned badges to those who were active per week in the course and to those who fulfilled the conditions to earn the badges. To do so, the students were classified into three categories: (1) active students (i.e., students who participated in an activity in the current or in an upcoming week of the course), (2) active students who fulfilled the conditions condition of the badges associated to group activities) could be done by any member of the group, a different day that the student log in the MOOC platform affecting to the claiming time span. Understanding Student Behavior and Perceptions toward Earning Badges in a Gamified MOOC 9 to earn a badge (called performing students), and (3) active students who fulfilled the conditions and claimed the badges (called issued students). Table 1shows the number of students at each category per week throughout the course. As illustrated in Figure 6, there was a sharp decrease in the number of active students during the first two weeks, which then slightly decreased in the rest of the course, a trend often observed in MOOC contexts [1]. In the same way, performing students and issued students followed a similar trend. As shown in Table 1, among the active students, those who were issued a badge oscillated throughout the course, ranging from 32.95% (Quiz 0, week 1) to 72.67% (Quiz 6, week 7), even after the dropouts during the initial weeks. The ratio of students who were issued a badge to those who fulfilled the badge conditions was high and stable (i.e., more than 79% for every badge). As different badges may affect student interest on earning them, further analysis was conducted to investigate the influence of badge types on students’ behavior towards badges. As seen in Table 1, the ratio of issued students to active students in badges associated with quiz performance showed an increasing trend throughout the course: 32.95% (Quiz 0, week 1), 61.22% (Quiz 1, week 2) and 72.67% (Quiz 6, week 7). This increase could be attributed to the high number of dropouts in the first weeks. However, badges associated with peerreview participation exhibited a decreasing trend even though they started being issued after the third week (after which dropouts were minimal): 60.23% (Rookie Reviewer, week 5), 51.18% (Intermediate Reviewer, week 6) and 49.69% (Advanced Reviewer, week 7). These results suggest that depending on the badge type and the associated conditions, students behaved differently. Although in this case badges associated with quiz performances appeared to be more popular, further work is needed to analyze if some factors such as the difficulty of the quiz or the time devoted to the peer reviews influenced these results. 4.1.2 Time Span for Claiming Badges In order to better understand the student behavior towards earning badges, we calculated the time span between the moment that the students were eligible for a badge (i.e., when a student fulfilled the conditions) and the moment they claimed it. Results (see Table 1) show that the modal value of claiming time is 0 (the same day) for every badge and the median value varies from 0 to 1 days. Also, results show a high variability in the standard deviation depending on the badge ranging from σ=1.35 (Advanced Reviewer) to σ=6.83 (Translator) days. Furthermore, we calculated the 95% confidence interval (see Fig. 7) to estimate the claiming time span interval that contains the true value for other possible populations [43]. Similar to the sample mean, the confidence interval grows from the beginning (3.18±0.43 days; n=282) to the middle of the course in the fourth week (4.65±1.37 days; n=96), and decreases from the fourth week to the end of the course (0.55±0.30 days; n=80). This initial growth could be attributed to a loss of interest in earning badges. The decrease towards the end of the course might be explained by the end date of the course that creates a shorter time span for claiming the badges. It seems that the growth in the confidence interval would continue if there were no end course date. Further work would be needed to analyze the student claiming time span after the middle of the course. Additionally, ranges are under the threshold of 7 days for every badge. Therefore, most students claimed badges before the release of a new weekly module (and the release of new badges), which suggests a positive attitude towards them. 4.1.3 The Role of Student Behavior towards Badges on Behavioral Engagement As already stated, behavioral engagement concerns the observable behaviors that represent the student involvement in learning such as participation, persistence or contributions [20]. This way, the variables considered to measure the student behaviors’ towards earning badges (i.e., the number of badges earned and the claiming time span) can also be considered as additional variables for modeling such behavioral engagement. Apart from the variables measuring the behavior towards earning badges, the behavioral engagement was also determined by four more variables typically used to this end [28]: (a) the number of pageviews, (b) the number of tasks completed, (c) the number of forum posts and (d) the activity time10. In this subsection, we analyze the relationship between the variables modeling the student behavior towards earning badges and the variables that model behavioral engagement in the course. This analysis will allow us explore whether students with high levels of engagement, as measured by “traditional variables”, are also the students that tend to claim more badges in a shorter time (or not). To do so, a Bivariate Pearson correlation analysis [39] was performed based on the con10 Activity time counts the time the student had the course open in the browser. Although this measure is different than the total time a student was working in the course, it can help us to understand the relationship with other parameters measuring the engagement. 16 Alejandro Ortega-Arranz1et al. Finally, although this study was carried out in a real MOOC environment, a set of similar studies would be needed to increase the transferability of these findings to other MOOC contexts. Acknowledgements This research has been partially funded by the Spanish State Research Agency (AEI) together with the European Regional Development Fund, under project grants TIN2014-53199-C3-2-R and TIN2017-85179-C3-2-R, the Regional Goverment of Castilla y Le´on together with the European Regional Development Fund, under project grants VA082U16 and VA257P18, and the European Commission, under project grant 588438-EPP-1-2017-1-EL-EPPKA2-KA. The authors would like to thank the rest of the GSIC-EMIC research team for their valuable ideas, and the Canvas Network team for the support received conducting this research. References 1. Alario-Hoyos, C., P´erez-Sanagust´ın, M., Cormier, D., Delgado Kloos, C.: Proposal for a Conceptual Framework for Educators to Describe and Design MOOCs. J. UCS 20(1), 6–23 (2014) 2. Alario-Hoyos, C., P´erez-Sanagust´ın, M., Delgado Kloos, C., Parada G., H.A., Mu˜noz-Organero, M.: Delving into Participants’ Profiles and Use of Social Tools in MOOCs. IEEE Transactions on Learning Technologies 3(7), 260– 266 (2014) 3. Anderson, A., Huttenlocher, D., Kleinberg, J., Leskovec, J.: Engaging with Massive Online Courses. In: Proceedings of the 23rd International Conference on World Wide Web, pp. 687–698. ACM (2014) 4. Antonaci, A., Klemke, R., Stracke, C.M., Specht, M.: Gamification in MOOCs to enhance users’ goal achievement. In: Proceedings of the 2017 Global Engineering Education Conference, pp. 1654–1662. IEEE (2017) 5. Aparicio, M., Oliveira, T., Bacao, F., Painho, M.: Gamification: A key determinant of massive open online course (MOOC) success. Information & Management 56(1), 39– 54 (2019) 6. Bartle, R.: Hearts, clubs, diamonds, spades: Players who suit MUDs. Journal of MUD research 1(1) (1996) 7. Bethlehem, J.: Selection Bias in Web Surveys. International Statistical Review 78(2), 161–188 (2010) 8. Cross, S., Whitelock, D., Galley, R.: The use, role and reception of open badges as a method for formative and summative reward in two Massive Open Online Courses. International Journal of e-Assessment 4(1), 1–16 (2014) 9. Csikszentmihalyi, M.: Beyond boredom and anxiety. Jossey-Bass (2000) 10. Daniel, J.: Making Sense of MOOCs: Musings in a Maze of Myth, Paradox and Possibility. Journal of Interactive Media in Education 2012(3) (2012) 11. De Sousa Borges, S., Durelli, V., Reis, H., Isotani, S.: A Systematic Mapping on Gamification Applied to Education. In: Proceedings of the 29th Annual ACM Symposium on Applied Computing, pp. 216–222. ACM (2014) 12. Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From Game Design Elements to Gamefulness: Defining Gamification. In: Proceedings of the 15th ACM International Academic MindTrek Conference: Envisioning Future Media Environments, pp. 9–15 (2011) 13. Dicheva, D., Dichev, C., Agre, G., Angelova, G.: Gamification in Education: A Systematic Mapping Study. Journal of Educational Technology & Society 18(3), 75–88 (2015) 14. Dillenbourg, P., Fox, A., Kirchner, C., Mitchell, J., Wirsing, M.: Massive Open Online Courses: Current State and Perspectives (Dagstuhl Perspectives Workshop 14112). Dagstuhl Manifestos 4, 1–27 (2014) 15. Ding, L., Kim, C., Orey, M.: Studies of student engagement in gamified online discussions. Computers & Education 115, 126–142 (2017) 16. Dom´ınguez, A., Saenz-de Navarrete, J., De-Marcos, L., Fern´andez-Sanz, L., Pag´es, C., Mart´ınez-Herr´aiz, J.J.: Gamifying learning experiences: Practical implications and outcomes. Computers & Education 63, 380–392 (2013) 17. Ferguson, R., Sharples, M.: Innovative pedagogy at massive scale: teaching and learning in MOOCs. In: Open Learning and Teaching in Educational Communities, pp. 98–111. Springer (2014) 18. Festinger, L.: A theory of social comparison processes. Human relations 7(2), 117–140 (1954) 19. Fitz-Walter, Z.: Achievement Unlocked: Investigating the Design of Effective Gamification Experiences for Mobile Applications and Devices. Ph.D. thesis, Queensland University of Technology, Australia (2015) 20. Fredricks, J.A., Blumenfeld, P.C., Paris, A.H.: School Engagement: Potential of the Concept, State of the Evidence. Review of educational research 74(1), 59–109 (2004) 21. Fresno, J., Ortega-Arranz, H., Ortega-Arranz, A., Gonz´alez-Escribano, A., Llanos, D.R.: Applying Gamification in a Parallel Programming Course. GamificationBased E-Learning Strategies for Computer Programming Education, IGI Global (2016) 22. Greene, J.C.: Mixed methods in social inquiry, vol. 9. John Wiley & Sons (2007) 23. Hakulinen, L., Auvinen, T., Korhonen, A.: Empirical Study on the Effect of Achievement Badges in TRAKLA2 Online Learning Environment. In: Proceedings of the 2013 International Conference on Learning and Teaching in Computing and Engineering, pp. 47–54. IEEE (2013) 24. Hamari, J.: Transforming homo economicus into homo ludens: A field experiment on gamification in a utilitarian peer-to-peer trading service. Electronic commerce research and applications 12(4), 236–245 (2013) 25. Hamari, J.: Do badges increase user activity? A field experiment on the effects of gamification. Computers in human behavior 71, 469–478 (2017) 26. Hamari, J., Koivisto, J., Sarsa, H.: Does Gamification Work? - A Literature Review of Empirical Studies on Gamification. In: Proceedings of the 47th Hawaii International Conference on System Sciences, pp. 3025–3034. IEEE (2014) 27. Henderikx, M.A., Kreijns, K., Kalz, M.: Refining success and dropout in massive open online courses based on the intention–behavior gap. Distance Education 38(3), 353– 368 (2017) 28. Henrie, C.R., Halverson, L.R., Graham, C.R.: Measuring student engagement in technology-mediated learning: A review. Computers & Education 90, 36–53 (2015) 29. Hew, K.F.: Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology 47(2), 320–341 (2016) 30. Ib´a˜nez, M.B., Di-Serio, A., Delgado Kloos, C.: Gamification for Engaging Computer Science Students in Learning Understanding Student Behavior and Perceptions toward Earning Badges in a Gamified MOOC 17 Activities: A Case Study. IEEE Transactions on Learning Technologies 7(3), 291–301 (2014) 31. Jordan, K.: Initial trends in enrolment and completion of massive open online courses. The International Review of Research in Open and Distributed Learning 15(1), 133– 160 (2014) 32. Khalil, H., Ebner, M.: MOOCs Completion Rates and Possible Methods to Improve Retention - A Literature Review. In: Proceedings of the 2014 World Conference on Educational Multimedia, Hypermedia and Telecommunications, pp. 1305–1313 (2014) 33. Khalil, M., Ebner, M.: Learning Analytics in MOOCs: Can Data Improve Students Retention and Learning? In: Proceedings of the 2016 World Conference on Educational Media and Technology, pp. 581–588. Association for the Advancement of Computing in Education (AACE) (2016) 34. Khalil, M., Ebner, M., Admiraal, W.: How can Gamification Improve MOOC Student Engagement? In: Proceedings of the 11th European Conference on Games Based Learning, pp. 819–828 (2017) 35. Khalil, M., Wong, J., de Koning, B., Ebner, M., Paas, F.: Gamification in MOOCs: A Review of the State of the Art. In: Proceedings of the 2018 Global Engineering Education Conference, pp. 1629–1638. IEEE (2018) 36. Kyewski, E., Kr¨amer, N.C.: To gamify or not to gamify? an experimental field study of the influence of badges on motivation, activity, and performance in an online learning course. Computers & Education 118, 25–37 (2018) 37. Lee, J.J., Hammer, J.: Gamification in Education: What, How, Why Bother? Academic Exchange Quarterly 15(2), 146 (2011) 38. Liyanagunawardena, T.R., Lundqvist, K.Ø., Williams, S.A.: Who are with us: Mooc learners on a futurelearn course. British Journal of Educational Technology 46(3), 557–569 (2015) 39. Mertler, C.A., Reinhart, R.V.: Advanced and multivariate statistical methods: Practical application and interpretation. Routledge (2016) 40. Miles, M.B., Huberman, A.M.: Qualitative data analysis: An expanded sourcebook. SAGE Publications Inc. (1994) 41. Morales, M., Amado-Salvatierra, H.R., Hern´andez, R., Pirker, J., G¨utl, C.: A Practical Experience on the Use of Gamification in MOOC Courses as a Strategy to Increase Motivation. In: Proceeding of the 2016 International Workshop on Learning Technology for Education in Cloud, pp. 139–149. Springer (2016) 42. Mukaka, M.M.: A guide to appropriate use of correlation coefficient in medical research. Malawi Medical Journal 24(3), 69–71 (2012) 43. Nakagawa, S., Cuthill, I.C.: Effect size, confidence interval and statistical significance: a practical guide for biologists. Biological reviews 82(4), 591–605 (2007) 44. Navidi, W.C.: Statistics for engineers and scientists. McGraw-Hill Higher Education New York, NY, USA (2008) 45. O’Donovan, S., Gain, J., Marais, P.: A case study in the gamification of a university-level games development course. In: Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference, pp. 242–251. ACM (2013) 46. Ortega-Arranz, A., Kalz, M., Mart´ınez-Mon´es, A.: Creating Engaging Experiences in MOOC through In-Course Redeemable Rewards. In: Proceedings of the 2018 Global Engineering Education Conference, pp. 1875–1882. IEEE (2018) 47. Ortega-Arranz, A., Mu˜noz-Crist´obal, J.A., Mart´ınezMon´es, A., Bote-Lorenzo, M.L., Asensio-P´erez, J.I.: How Gamification is Being Implemented in MOOCs? A Systematic Literature Review. In: In Proceedings of the 12th European Conference on Technology Enhanced Learning, pp. 441–447. Springer (2017) 48. Ortega-Arranz, A., Sanz-Mart´ınez, L., ´ Alvarez- ´ Alvarez, S., Mu˜noz-Crist´obal, J.A., Bote-Lorenzo, M.L., Mart´ınez-Mon´es, A., Dimitriadis, Y.: From LowScale to Collaborative, Gamified and Massive-Scale Courses: Redesigning a MOOC. In: Proceedings of the 5th European Conference on Massive Open Online Courses, pp. 77–87. Springer (2017) 49. Reischer, M., Khalil, M., Ebner, M.: Does Gamification in MOOC Discussion Forums Work? In: Proceedings of the 5th European Conference on Massive Open Online Courses, pp. 95–101. Springer (2017) 50. Rizzardini, R., Chan, M., Guetl, C.: An Attrition Model for MOOCs: Evaluating the Learning Strategies of Gamification. Formative Assessment, Learning Data Analytics and Gamification: In ICT Education. Elsevier (2016) 51. Ruip´erez-Valiente, J.A., Mu˜noz-Merino, P.J., Delgado Kloos, C.: An analysis of the use of badges in an educational experiment. In: Frontiers in Education Conference (FIE), 2016 IEEE, pp. 1–8. IEEE (2016) 52. Ruip´erez-Valiente, J.A., Mu˜noz-Merino, P.J., Delgado Kloos, C.: Detecting and Clustering Students by their Gamification Behavior with Badges: A Case Study in Engineering Education. International Journal of Engineering Education 33(2), 816–830 (2017) 53. Shah, D.: Monetization over Massiveness: A Review of MOOC stats and Trends in 2016. Retrieved from: https://www.class-central.com/report/moocs-stats-andtrends-2016/ last access: April 2017 (2016) 54. Shah, D.: By The Numbers: MOOCS in 2017. How has the MOOC space grown this year? Get the facts, figures, and pie charts. Retrieved from: https://www.classcentral.com/report/mooc-stats-2017/ last access: June 2018 (2017) 55. Siemens, G.: Massive open online courses: Innovation in education. Open educational resources: Innovation, research and practice 5, 5–15 (2013) 56. Stake, R.E.: The Art of Case Study Research. SAGE Publications Inc. (1995) 57. Staubitz, T., Willems, C., Hagedorn, C., Meinel, C.: The Gamification of a MOOC Platform. In: Proceedings of the 2017 Global Engineering Education Conference, pp. 883–892. IEEE (2017) 58. Werbach, K., Hunter, D.: For the win: How game thinking can revolutionize your business. Wharton Digital Press (2012) 59. Xu, X., Zhi, H.: Study Of Game Elements Impacting On SE Course Completion Rate In MOOCs. Master’s thesis, Faculty of Computing, Blekinge Institute of Technology, SE-371 79, Karlskrona, Sweden (2017) 60. Zufferey, J.D., Alavi, H., Kidzinski, L., Dillenbourg, P.: Gamified Competition Features for Corporate MOOCs: The Battle Mode. In: Proceedings of the 4th European MOOCs Stakeholder Summit, p. 223 (2016)