The rise of motivational information systems : A review of gamification research

Contents lists available at ScienceDirect International Journal of Information Management journal homepage: www.elsevier.com/locate/ijinfomgt The rise of motivational information systems: A review of gamification research Jonna Koivisto a,⁎ , Juho Hamari a,b a Gamification Group, Faculty of Information Technology and Communications, Tampere University, Finland b Gamification Group, Faculty of Humanities, University of Turku, Finland ARTICLE INFO Keywords: Gamification Games Motivational information system Affordance Literature review ABSTRACT Today, our reality and lives are increasingly game-like, not only because games have become a pervasive part of our lives, but also because activities, systems and services are increasingly gamified. Gamification refers to designing information systems to afford similar experiences and motivations as games do, and consequently, attempting to affect user behavior. In recent years, popularity of gamification has skyrocketed and manifested in growing numbers of gamified applications, as well as a rapidly increasing amount of research. However, this vein of research has mainly advanced without an agenda, theoretical guidance or a clear picture of the field. To make the picture more coherent, we provide a comprehensive review of the gamification research (N = 819 studies) and analyze the research models and results in empirical studies on gamification. While the results in general lean towards positive findings about the effectiveness of gamification, the amount of mixed results is remarkable. Furthermore, education, health and crowdsourcing as well as points, badges and leaderboards persist as the most common contexts and ways of implementing gamification. Concurrently, gamification research still lacks coherence in research models, and a consistency in the variables and theoretical foundations. As a final contribution of the review, we provide a comprehensive discussion, consisting of 15 future research trajectories, on future agenda for the growing vein of literature on gamification and gameful systems within the information system science field. 1. Introduction During recent decades, we have witnessed glimpses of a fascinating emerging development where utilitarian and hedonic systems are in a state of spiraling convergence. Today, the spiral has made a full revolution, and we now see hedonic or entertainment-oriented technologies being re-appropriated for productive use. This development has been titled “gamification” and the phenomenon has quickly cemented itself as being one of the major developments in the information systems (IS) field and other domains. Hedonic information systems initially came about through the re-appropriation of instrumental information technology. Most notably, the first video games emerged from a playful re-appropriation of oscilloscopes – a seemingly utilitarian system (“Tennis for Two” developed by Higinbotham in 1958 – see e.g. Tavinor, 2009). Since then, we have witnessed a wide diffusion of game consoles (e.g. Pong in 1972, Atari 2600 in 1977, Nintendo in 1983, Xbox in 2002 etc.) and other video game applications. Forwarding to today, hedonic systems and software are everywhere, and are developed for the sole purpose of promoting user enjoyment. Furthermore, digital games have penetrated our everyday lives at an increasing pace and have now become a mainstream form of entertainment, enjoyed by people from all demographic groups (see e.g. Williams, Yee, & Caplan, 2008;Williams, Consalvo, Caplan, & Yee, 2009). However, especially during the last ten years, we have come a full circle, and hedonic systems (and especially game designs) are currently merging back into utilitarian systems and even perhaps new strains of utilitarian systems are emerging from hedonic systems. Games are especially known for their ability to engage and excite, and when playing games, people commonly experience e.g. mastery, competence, enjoyment, immersion, or flow, all of which are characteristic of intrinsically motivated human behavior (e.g. Huotari & Hamari, 2017;Ryan, Rigby, & Przybylski, 2006;Deci & Ryan, 2000; Ryan & Deci, 2000;Agarwal & Karahanna, 2000;Venkatesh, 1999; Webster & Martocchio, 1992;Csíkszentmihályi, 1975,1990). An essential aspect of playing games is the self-purposeful nature of the activity, as well as the engagement and enjoyment of the activity. It is this nature of playing games that gamification technology attempts to capture, harness and implement into contexts that commonly have a https://doi.org/10.1016/j.ijinfomgt.2018.10.013 Received 27 June 2017; Received in revised form 13 July 2018; Accepted 15 October 2018 ⁎ Corresponding author at: Gamification Group, Faculty of Information Technology and Communications, FI-33014 Tampere University, Finland. E-mail address: [email protected] (J. Koivisto). International Journal of Information Management 45 (2019) 191–210 Available online 11 December 2018 0268-4012/ © 2018 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/BY-NC-ND/4.0/). T more instrumental purpose (Hamari & Koivisto, 2015b;Huotari & Hamari, 2017;Liu, Xun, & Santhanam, 2013;Santhanam, Liu, & Shen, 2016;Vesa, Hamari, Harviainen, & Warmelink, 2017;Liu, Santhanam, & Webster, 2017). When starting a game, a player accepts the contingency of the end result, however, the process is often enjoyable regardless of the outcome (see e.g. Malaby, 2007). Incorporating the engagement and enjoyment of the gameful process into activities outside games is at the core of what commonly is titled gamification; a design approach of employing game elements into different types of systems and services, with the goal of affording gameful experiences (Huotari & Hamari, 2017). Since its conceptual inception around 2010, gamification has increasingly drawn the attention of academics and practitioners (see Hamari, Koivisto, & Sarsa, 2014). In addition to gaining popular proponents, the approach has gained traction from positive prospects published in business analyses by Gartner (2011) and IEEE (2014) which predict that most companies and organizations will implement gamification in the near future. Consequently, operators in various fields have been attracted by the potential of gamification for inducing motivation and engagement for a diverse range of activities. This has led to gamification being implemented in domains such as enterprise resource planning (Alcivar & Abad, 2016;Herzig, Strahringer, & Ameling, 2012), intra-organizational communication and activity (Farzan et al., 2008b,Farzan et al., 2008a;Thom, Millen, & DiMicco, 2012), science (Sørensen et al., 2016), government services and public engagement (Bista, Nepal, Paris, & Colineau, 2014;Tolmie, Chamberlain, & Benford, 2014; see also Hassan & Hamari, 2019 for a review), work (see Warmelink, Koivisto, Mayer, Vesa, & Hamari, 2018 for a review) and crowdsourcing (Eickhoff, Harris, de Vries, & Srinivasan, 2012;Lee, Ceyhan, Jordan-Cooley, & Sung, 2013;Ipeirotis & Gabrilovich, 2014; see also Morschheuser, Hamari, Koivisto, & Maedche, 2017 for a review), commerce (Hamari, 2015;Hamari, 2013), exercise (Hamari & Koivisto, 2015a;Koivisto & Hamari, 2014), health (Jones, Madden, & Wengreen, 2014; see also Alahäivälä & OinasKukkonen, 2016 for a review), education (e.g. Bonde et al., 2014; Christy & Fox, 2014;de-Marcos, Domínguez, Saenz-de-Navarrete, & Pagés, 2014;Denny, 2013;Domínguez et al., 2013;Farzan & Brusilovsky, 2011;Filsecker & Hickey, 2014;Hakulinen, Auvinen, & Korhonen, 2013;Simões, Díaz Redondo, & Fernández Vilas, 2013; see also Majuri, Koivisto, & Hamari, 2018 for a review), environmental behavior (Lee et al., 2013;Lounis, Pramatari, & Theotokis, 2014), as well as marketing and advertising (Cechanowicz, Gutwin, Brownell, & Goodfellow, 2013;Terlutter & Capella, 2013;Xi & Hamari, 2019), to name a few. The literature on gamification is rapidly increasing and spreading in many directions, but this is similar to any development that has great potential and which is surrounded by a crowd of hyped enthusiasts. In order to control and take advantage of this development, concerted efforts are needed to harness the literature and existing knowledge to productive use, and to provide the field with an agenda for further research. Gamification is still in its infancy and rapidly developing, but what is actually known of the phenomenon tends to stem from fragmented pieces of knowledge, and from a variety of perspectives. While some attempts have been made to synthesize the literature on gamification, previous reviews have been very focused in their scope. In order to provide both academics and practitioners with a more widespread view on the gamification phenomenon, a larger scale review of the phenomenon should help to map its development and progress, as well as aid in steering future literature and agendas. We firmly believe that gamification is especially an IS/IT phenomenon, since it has at its core the use of leisure information systems (more specifically (video) games) and their design in a variety of utilitarian information system contexts. However, if we consider the host of literature on gamification that has been produced thus far, it appears to be relatively under-represented in IS literature, regardless of it clearly being an IS phenomenon. This suggests that other fields (especially those of education and humancomputer interaction) have perhaps shown more innovation and openness in their approach to this prominent technological development. Therefore, it is also important to more broadly initiate a discussion about gamification in IS literature. In this study, we aim to, firstly, comprehensively review and synthesize the extant literature on the concept of gamification; and secondly, to theorize and delineate a further research agenda for the research of gamification and motivational information systems within the information systems research field. The review draws together the existent knowledge on the topic and presents it in a structured manner. The review process mainly follows the guidelines described by Webster and Watson (2002) and Paré, Trudel, Jaana, and Kitsiou, (2015). Over 800 papers have been categorized, and 273 empirical studies are analyzed in detail to outline the domains in which gamification is being implemented, how it is being implemented, how it has been studied, as well as identifying the kinds of results that have been produced thus far. The findings of the review indicate where research knowledge is already abundant, where further research is needed, and what steps should be taken in future research to develop knowledge on the topic. 2. Background Information systems discipline has traditionally been characterized as the pursuit of knowledge pertaining to productivity and efficiency (see e.g. Hirschheim & Klein, 2012), and improvement of these. A substantial body of knowledge has sprung from this rational, utilityseeking premise of aiding in the development and construction of efficiently managed and operated organizations and information systems within them. However, this utility-driven lens of information systems was not geared towards capturing the use of varied non-utilitarian information systems that started to heavily appear when information technology had advanced enough in its graphics and calculation power. Information systems seeking to fulfill entertainment-oriented needs challenged the utilitarian premise previously dominating the research and understanding in the IS field (e.g. van der Heijden, 2004). Therefore, the scope of information systems science was expanded by introducing the study of hedonic information systems that deviates from the utility/rational-core of the IS discipline. The first wave of literature started to widen the perspective of IS research into intrinsic and hedonic motivations in the early 1990s by studying the concepts of playfulness and enjoyment in relation to technology acceptance and use (see e.g. Webster & Martocchio, 1992;Davis, Bagozzi, & Warshaw, 1992), and later in 2004 by e.g. van der Heijden (2004) via the development of models that addressed the acceptance and use of hedonic information systems. During the same period, the marketing research field and literature also witnessed a surge of research on hedonic aspects of consumption (see Hirschman & Holbrook, 1982 for an early account). Since then, however, a disconnect between the rational and hedonic veins of IS literature has existed until very recent literature on dual and multipurpose systems (as can be seen e.g. in Wu & Lu, 2013). When considering these various types of information systems, it seems that gamification has a rather interesting and peculiar role. Traditionally, the information systems field has distinguished between two system types that are designed to address different needs, either utilitarian or hedonic (see e.g. van der Heijden, 2004). Systems defined as utilitarian information systems are commonly designed to serve purposes related to productivity. They seek to increase the efficiency of a given task, and therefore, at their core they serve mainly instrumental purposes. From a motivational perspective, the use of utilitarian systems can be considered as being extrinsically motivated (see e.g. Deci & Ryan, 1985;van der Heijden, 2004); i.e. the system aids the user in reaching a goal that is separate from the system use itself (Davis et al., 1992). The usefulness and benefits of the system thus rise from reaching the external goal more efficiently. Conversely, the use of hedonic information systems is mostly entertainment-driven (van der Heijden, 2004). These systems aim at creating experiences of enjoyment and are J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 192 used for the purposes of recreation and entertainment, and for the sake of using the system itself. Therefore, the use of hedonic systems is considered to be autotelic and intrinsically motivated (Deci & Ryan, 1985;Ryan & Deci, 2000), as opposed to systems with utilitarian use objectives. Examples of hedonic systems are entertainment-oriented websites and services, video games, blogs, and social networking sites. In recent years, information systems have increasingly been designed to meet more varied user motivations and orientations (Gerow, Ayyagari, Thatcher, & Roth, 2013;Sun & Zhang, 2006). The research field has started to acknowledge that many systems serve both utilitarian as well as hedonic needs (Hamari & Keronen, 2017), and increasingly, systems are fundamentally being designed to cater for these needs, i.e. as mixed systems (Gerow et al., 2013). One such type of mixed systems are those with the objective of motivating users toward different individually and collectively beneficial behaviors (Hamari & Koivisto, 2015b;Hamari et al., 2014). These systems are an intriguing combination of both utilitarian (Davis, 1989) and hedonic aspects (van der Heijden, 2004): the goals of the systems’ use are related to productivity, however, the means and the design by which the systems promote productivity are hedonic in nature. Whereas traditional utilitarian systems aim for productivity through efficiency and traditional hedonic systems aim for creating fun experiences, these Motivational Information Systems can be characterized to aim for “productivity through fun”. Therefore, Motivational Information Systems differ from the utilitarian and hedonic information systems in one important, crucial way: the acceptance is mainly driven by usefulness as in utilitarian systems, but the usefulness is determined by the enjoyment of the use. One of the most prominent solutions for addressing motivational challenges has been to draw from one of the pinnacle forms of hedonic information systems, i.e. digital games. This approach is commonly referred to as gamification. Gamification refers to a design approach of enhancing services and systems with affordances for experiences similar to those created by games (Deterding, Dixon, Khaled, & Nacke, 2011;Hamari, Huotari, & Tolvanen, 2015;Huotari & Hamari, 2012;Liu et al., 2017;Santhanam et al., 2016;Vesa et al., 2017). These “gameful” affordances aim at supporting and motivating the user toward the behavior that the gamified system is targeting, such as healthy behaviors and exercise, participation in learning activities etc. The experiences created by games refer e.g. to senses of enjoyment, flow, autonomy, mastery, accomplishment etc., that are considered to be induced by games and game play (e.g. Ryan et al., 2006). In the context of games, these experiences are often considered to be what make games intrinsically motivating, so that the user wishes to engage with the system simply for the sake of using it. Harnessing similar experiences by implementing gameful affordances in the contexts of the utilitarian functions described above aims at transferring similar motivational effects into the new environment. What makes motivational information systems such as gamification interesting is the fact that the systems at their core motivate and support the user toward a given activity or behavior. This conveys that the system should increase the efficiency and productivity regarding the target behavior. Thus, their usefulness is determined on the basis of whether they manage to do so. Furthermore, in many of the commonly gamified contexts, such as learning or healthy behavior, the activities require long-term commitment and persistence for the results to actualize. On the other hand, the systems features hedonic design, and therefore aim at making the process of using the system enjoyable. When the system use is enjoyable, the chances of engaging with it in long-term may be increased (see e.g. van der Heijden, 2004;Atkinson & Kydd, 1997;Moon & Kim, 2001;Venkatesh, 1999;Mäntymäki, Merikivi, Verhagen, Feldberg, & Rajala, 2014;Mäntymäki & Riemer, 2014;Hamari, 2015;Mäntymäki & Salo, 2013;Mäntymäki & Salo, 2015). The potential of gamification lies in the restructuring of tasks and activities with game elements and gameful affordances. This may be by dividing a larger whole into sub-tasks with clear goals and providing direct feedback for accomplishments, reframing an activity by establishing a meaningful narrative, or by gathering a social community to provide support. A commonly used theoretical frame for understanding the motivational potential of games is that of self-determination theory (SDT) and its sub-theories (Deci & Ryan, 2000;Deci & Ryan, 1985;Ryan et al., 2006) that consider human motivation to be either intrinsically or extrinsically motivated, depending on whether the activity is performed for the sake of the activity itself or for reasons external to the activity. For a behavior to be intrinsically motivated, it is likely that it results from motivational needs for competence, autonomy and relatedness (Deci & Ryan, 2000;Ryan & Deci, 2000). Competence refers to feelings of mastering the challenge at hand. Autonomy refers to the freedom of choosing what challenges to undertake, and relatedness refers to experiences of recognition and acceptance (Deci & Ryan, 2000). All of these motivational needs are well-documented as being commonly satisfied by playing games (Ryan et al., 2006), which is generally considered to be a highly intrinsically motivated behavior (Rigby & Ryan, 2011). Playing games is usually a voluntary behavior, conducted at one’s own instigation, and thus a behavior that promotes autonomy (Ryan et al., 2006). Furthermore, encountering and overcoming challenges that are often adjusted to the optimal level for the player (see e.g. Csíkszentmihályi, 1975,1990) is essential to gameplay, and is often considered to be a core component of games (Deterding, 2015;Ryan et al., 2006). Thus, playing games commonly provides experiences of competence as the player tackles the challenges of the game. Moreover, the player’s relatedness is often catered for by social environments created either within the game or around it (Huang, Cheng, Huang, & Teng, 2018;Ryan et al., 2006). On an overarching level, gamification can be seen to comprise of three main elements (Hamari et al., 2014): the affordances implemented to a system or service lead to psychological outcomes, and these gameful experiences further lead to behavioral outcomes, i.e. the activities and behaviors that the gamification aims to support and motivate (see Fig. 1). Furthermore, all these elements and the supported activity are situated within a certain context (Hamari et al., 2014; Huotari & Hamari, 2017;Deterding, 2015). The affordances refer to the various elements and mechanics that structure games and aid in inducing gameful experiences within the systems. The psychological outcomes refer to psychological experiences such as competence, autonomy and relatedness, or for example enjoyment and engagement, which games and gamification are commonly considered to promote. The behavioral outcomes of gamification refer to behaviors and activities that are supported through use of the gamification system, such as continued or increased physical activity in the context of exercise gamification, or better learning results in the context of education gamification. The topic of motivational information systems and especially gamification has gained significant popularity in recent years, which has Fig. 1. Overall conceptualization of gamification following Hamari et al. (2014);Huotari and Hamari (2017) and Deterding (2015). J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 193 manifested in a great amount of literature being produced on the topic from both academic and non-academic sources. The first attempts to map the literature under the flag of gamification were conducted in early 2014, and formed a review examining the then-current body of 24 peer-reviewed internationally published research papers (Hamari et al., 2014). Since then, the popularity of the topic has increased significantly and the concept has been trending among practitioners and academics. As confirmation of the growth of the topic, academic databases provide proof of the expanding volume of literature found with the keyword ‘gamification’. In their review, Hamari et al. (2014) reported that a literature search in the Scopus database with the search terms ‘gamification’, ‘gamif*’, ‘gameful’ and ‘motivational affordance’ resulted in 330 hits. In June 2015, a similar literature search using only the search term ‘gamif*’ provided 807 hits. At the time of writing, in June 2016, the latter search provides 1767 hits. In order to maintain currency with the research area, an overview of how the field of gamification is developing is desperately needed. While some literature mapping has been conducted since the Hamari et al. (2014) review (see e.g. Seaborn & Fels, 2015), a general overview of how the phenomenon has been researched and developed is lacking. 3. The Review 3.1. Review procedure The literature searches were conducted in the Scopus database and the Association for Information Systems (AIS) Electronic Library (AISeL), which were chosen for the reason that they index all of the other potentially relevant databases, for example ACM, IEEE, Springer, and the DBLP Computer Science Bibliography. Conducting the searches in as few comprehensive databases as possible instead of several ones is seen as preferable for purposes of rigor and clarity (see Paré et al., 2015). The search for literature in the Scopus database was conducted using the search query: TITLE-ABS-KEY (gamif*). The search was limited to include conference papers, articles, articles in press, reviews and book chapters, in order to exclude non-academic publications. In the AISeL database, the search query ‘abstract:gamif* OR subject:gamif* OR title:gamif*’ was used. The search term gamif* was chosen as it takes into account all possible forms derived from the root, i.e. the noun gamification, and the verb gamify in all its forms. The search fields were defined in Scopus as title, abstract, and keywords, and in AISeL as title, abstract, and subject. These search parameters were used as it is considered that inclusion of the search term in the metadata would indicate that the term is actually relevant for the given paper. With these search queries, we aimed to reach all the academic literature (within the used databases) that is published under the flag of gamification. Inevitably, this strategy also led to hits where the term gamification was not of actual relevance to the paper, but where, for example, the abstract contained a mention of gamification as a future prospect. These hits were included in the review, but highlighted as not directly relevant and therefore not included in the full analyses. The literature search was conducted in June 2015 and resulted in 807 hits from the Scopus database and 35 from the AISeL database. All of the hits were collected into reference management software, where Fig. 2. Flowchart describing the literature review process. J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 194 the references were managed, organized and categorized. The review process was conducted by the first author of this paper. Any cases and categorizations that were unclear were discussed amongst the research team. On further examination, 2 hits were removed as duplicate entries in the Scopus database and 18 hits were identified as overlapping between the Scopus and AISeL search results. Of the remaining papers, 3 studies were identified as false hits where the paper concerned used the letter combination ‘gamif’ to refer to something other than gamification. The final body of literature amounted to 819 papers. The full versions of the papers were retrieved. Where papers were initially not available for us, we contacted the authors either by email or through academic social network tools such as ResearchGate and requested the full version. Around 30 papers were obtained in this way. The final number of unavailable papers was 16. Additionally, 16 of the papers were identified as not being written in English, and were therefore excluded from further analyses. The remaining 787 papers were categorized based on the type of publication, whether the papers were full research papers, and whether they contained empirical data. Fig. 2 outlines the literature review process. 286 full papers were identified as containing empirical data. Of these, 13 were further identified as explicitly stating that the studied system contained no gamification, that gamification was part of future plans to develop the featured solution, or the paper contained a mention of the term solely as a keyword. These papers were removed from the further analyses which focused on empirical studies. The final amount of analyzed empirical papers was 273. In the literature analysis, the studies were coded. As mentioned above, the amount of information coded from the retrieved papers depended on whether they were categorized as empirical or non-empirical. Full analyses were conducted only on empirical full papers. Table 1 outlines the details that were retrieved during the coding process from both empirical and non-empirical papers. Following the guidelines of Webster and Watson (2002), all of the full papers were analyzed, firstly author-centrically, and then conceptcentrically. The units of analysis were defined prior to the analytical process. Author-centric coding was conducted, where the pre-defined units of analysis were checked and coded for each paper as it was read. Through this procedure, a matrix of the coded literature was produced. Continuing the analysis process, a concept-centric approach was then taken where coded literature was then organized based on further units of analysis. As suggested by Webster and Watson (2002), the coded concepts were comprised into frequency tables, which form the core of this review. 3.2. Bibliometric descriptors Since 2011, there has been rapidly increasing general interest in gamification, and this has also been reflected by the academic interest shown in the topic. As the literature search hits by year indicate (Table 2), the amount of academic literature on gamification has consistently risen over the past few years. From the 819 total hits, 260 papers were identified as not being full papers (Table 3). In this category we included all the papers that, e.g. reported research-in-progress or other types of incomplete research, were not peer-reviewed, or were in other formats than research reports. Many of these papers were in fact short conference publications and workshop papers. The full research papers were categorized based on the form of Table 1 Details retrieved from the full papers in the reviewed body of literature. Coded information Non-empirical, full papers Empirical full papers Empirical full papers - Quantitative experimental studies Bibliometric data: Author(s), publication year, publication venue X X X Type of study: Empirical, (preliminary) description of a study/system, conceptual papers; frameworks; think pieces, review papers, simulations; modelings, heuristics, definition papers X X X Domain: E.g. education, health and exercise, business and management etc. X X X Data gathering method: E.g. Implementation/prototype, interviews, observation etc. X X Data type used in analyses: E.g. Log data, interview data, observation data etc. X X Analysis method: Qualitative, quantitative (descriptive, inferential) X X Affordances: E.g. points, narratives, social features etc. X X Outcomes: Psychological outcomes, behavioral outcomes X X Sample sizes X Results: Positive, mixed with mostly positive, null or equal positive and negative, mixed with mainly negative, negative; categorized by domain and affordances X Table 2 All Scopus and AISeL hits for gamif* by year at the time of data gathering (6/2015). False hits, duplicates and overlapping hits removed. Year Number of hits 2011 26 2012 92 2013 263 2014 368 06/2015 70* Total 819 *It is noteworthy that due to the natural delay in addition of entries to the repositories, a proportionately smaller number of studies had yet appeared during 2015 at the time of the data gathering. Table 3 Types of not full papers (N = 260). Types of not full papers Frequency Short papers, research notes 69 Workshop papers 60 Posters 28 Magazine articles, journal paper articles, newsletters, commentaries 19 Extended abstracts 17 Research-in-progress 19 Demonstrations, tutorials 11 Technical reports, industry track papers 8 Panel or track proposals/descriptions 8 PhD papers 3 Editorials 2 Business reports 1 Books 1 Other not full 14 J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 195 research reported, e.g. empirical research, descriptive non-empirical research, conceptual work, or literature reviews (see Table 4). After the removal of duplicate studies, 514 full research papers were examined to determine the type of the research conducted and reported. Of the 514 papers, 286 were identified as empirical. However, 13 of these explicitly stated that either gamification was not examined in the paper, that gamification was considered as a future possibility, or the paper simply contained gamification as a keyword without any further mentions. These studies were therefore not included in further analyses. Thus, the final analysis included 273 studies reporting empirical data that was analyzed in the paper. These publications varied greatly in terms of the extent of data gathering and analysis, ranging from largescale or multi-year experiments to preliminary trials with only a few participants. However, all of the studies were categorized as empirical research papers for the purposes of this review, and for the purpose of gaining the widest possible overview of the research field. Papers describing a future study or a system under development were prominent (105 studies), but as these papers were identified not to contain any actual empirical data, they were excluded from further content analysis. Conceptual papers, frameworks and think pieces (84 studies) mostly considered ideas and perspectives for developing the understanding of gamification or the study of the phenomenon. The remainder of the body of literature consisted of reviews, simulations on gamification, heuristic papers, as well as papers concentrating on defining the concept of gamification. Of the full research papers, a clear majority (352 papers) were published in conference proceedings, and only 155 were published in a journal (see Table 5). What potentially follows from the fact that conference publications are the main venue for publication of gamification research is related to the comprehensiveness of the research work. Conference publications are usually limited in length and often not as comprehensive in their discussion of various aspects of the conducted research when compared to journal publications. Thus, the great number of conference proceedings has a potential effect on the theoretical depth of the research on gamification. In their review of gamification literature, Seaborn and Fels (2015) found that the majority of the reviewed papers did not address the theoretical foundations which guided the research, and the lack of theoretical discussions has been acknowledged as a problem in the field. Seaborn and Fels (2015) also observe a disconnection between theory and applied research in their reviewed literature. While the theoretical stances and foundations of gamification were not explicitly mapped in this review, similar observations can be made from the body of current literature; a large proportion of the research papers addressed the theoretical foundations of the work only in passing, via a definition of gamification or presenting related prior research. Nevertheless, the bibliometric descriptors of the current body of literature demonstrate the increasing amount of research that has been published on the topic of gamification. While there is no possibility of knowing whether this trend will continue, the bibliometric results may provide some perspectives which could be applied to future developments. As conference papers often present early research or developing ideas, the abundance of conference publications in the body of literature can reflect how interest in the concept of gamification will likely develop in the future. Additionally, as a large part of the literature reported work-in-progress or preliminary results, this suggests that the amount of literature will also continue to increase in the future. 3.3. Domains Table 6 outlines the domains of empirical studies, non-empirical descriptive, and conceptual papers. Altogether, 462 research papers were analyzed for their domain. Reviews, definition papers, heuristic and simulations/modelings were excluded, since their contexts were often not clearly definable due to the type of research presented. While domains may evidently overlap (e.g. an empirical study about gamification in education may actually be more concentrated on the workload it creates for the teachers, instead of the effects it has on students), for the purposes of the review, all of the studies were assigned to a single domain which describes the general context where the gamification has Table 4 Types of full papers. Types of papers (duplicates removed) Frequency % Empirical (also unclear methods/data) 273 53.1 Explicitly not about gamification 13 2.5 (Preliminary) description of a study/system, no empiricism 105 20.4 Conceptual papers; frameworks; think pieces 84 16.3 Review papers 19 3.7 Simulations; modelings 11 2.1 Heuristics 5 1.0 Definition papers 4 0.8 Total 514 100 Table 5 Publication venues of full papers. Publication venue Frequency % Conference 352 68.5 Journal 155 30.2 Book chapters 7 1.4 Total 514 100 Table 6 Domains in empirical, non-empirical descriptive and conceptual papers (N = 462). Empirical papers Nonempirical papers Domain Frequency % Frequency % Education/Learning 129 46.7 67 35.4 Health/Exercise 40 14.5 15 7.9 Crowdsourcing (includes information gathering, knowledge sharing and citizen science) 25 9.1 7 3.7 Social behavior/networking/ sharing 14 5.1 2 1.1 Software development/design * 11 4.0 25 13.2 Business/Management 10 3.6 19 10.1 Ecological/environmental behavior 9 3.3 9 4.8 eCommerce/eServices 8 2.9 1 0.5 Software engineering ** 7 2.5 5 2.6 Marketing/Consumer behavior 4 1.4 5 2.6 Citizen/public engagement/activity 3 1.1 2 1.1 Entertainment (includes gaming, watching TV, media capturing) 3 1.1 1 0.5 Innovation 3 1.1 2 1.1 Transportation/Mobility 3 1.1 4 2.1 Culture/Tourism 2 0.7 4 2.1 Architecture 1 0.4 0 0.0 Communication 1 0.4 0 0.0 Emergency planning 1 0.4 0 0.0 Politics 1 0.4 3 1.6 Welfare/city/human/public services 1 0.4 4 2.1 Work 1 0.4 2 1.1 Theory *** 0 0.0 12 6.3 Total 276**** 189 * Studies related to design and development of gamification services. ** Software engineering as a field of work/industry. *** Studies where gamification or an aspect related to it is discussed on an abstract/theoretical level. **** One study contains 4 different cases in different domains. All of the 4 domains are included in the categorizations. J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 196 been implemented or considered. The analysis of domains in the body of literature shows that a clear majority of the empirical research on gamification is conducted in the domain of education and learning. Second largest category of empirical studies is health and exercise, followed by research papers relating to crowdsourcing. These three categories comprise over 70% of the empirical research in the current body of literature. The fourth largest category consists of various social behavior and networking domains, followed by empirical studies which are related to the design and development of gamification services, as well as papers in the business and management domain. The remainder of the domain categories comprise less than ten empirical research papers. Interestingly, in the non-empirical research papers, education and learning is again the most common domain, but the second largest domain is formed by studies related to the design and development of gamification services. The third most common domain of non-empirical research is business and management. Thus it can be seen that in nonempirical research which potentially contains studies at a preliminary phase of the research process, there is the promise of certain domains gaining more empirical research in the future. In more detail, the domain of education and learning constitutes nearly half of the published empirical research on gamification, with 129 published empirical papers. In the non-empirical research, more than one third of the publications also stem from the education and learning domain. The education domain has previously been noted as the most popular gamification context for empirical research (Hamari et al., 2014;Seaborn & Fels, 2015). This review stands as further confirmation of these findings indicating that in the expanding field of online learning and training (Panigrahi, Srivastava, & Sharma, 2018) gamification has its share. The attractiveness of gamification in the education domain is rather intuitive, as games in general commonly promote learning and the developing of skills in an inherent manner, and often provide a structured environment where these skills can be practiced (see e.g. Landers & Armstrong, 2015). However, the educational domain has been somewhat riddled with conceptual unclarity as the terms of gamification, game-based learning and serious games are all commonly used to refer to the use of games in education contexts (Landers, 2015). In this review, no distinctions between the concepts were made, and any paper labeled by the authors as gamification has been considered, as specified above. In the empirical research, health and exercise forms the second largest category for studies. Especially with activities such as healthy eating and physical exercise, extra support and a motivational push may be needed in order to maintain routines (Hamari & Koivisto, 2015a). The gamification approach has been shown to have positive results in this domain in several of the featured studies (Allam, Kostova, Nakamoto, & Schulz, 2015;Cafazzo, Casselman, Hamming, Katzman, & Palmert, 2012;Chen & Pu, 2014;Chen, Zhang, & Pu, 2014;Hamari & Koivisto, 2015a;Jones et al., 2014;Koivisto & Hamari, 2014;Riva, Camerini, Allam, & Schulz, 2014;Watson, Mandryk, & Stanley, 2013). Furthermore, an interesting development is the large number of research papers on gamification in crowdsourcing. Research examining the combinations of these two fields has previously been mapped by Morschheuser, Hamari, and Koivisto, 2016;Morschheuser et al., 2017, who suggest that the addition of game elements for increasing the motivations of volunteer crowdsourcees is an efficient approach based on empirical research findings. Regarding the largest domains of empirical gamification research, it can be concluded that the findings of this review support those of Seaborn and Fels (2015), who found health, crowdsourcing and social networking to be the biggest domains for empirical research of gamification, in addition to education. As a further conclusion, based on the domains of the empirical research, it is evident that gamification seems to be implemented especially in domains where long-term commitment and perseverance are needed for gaining results. These may include learning or the development of healthy or beneficial habits. This idea is supported by the fact that education, health and e.g. domains relating to ecological behavior are among the most popular categories. Among the non-empirical research, the prevalence of the domains differs from the pattern seen in empirical works, with the exception of education and learning. Non-empirical work has been conducted more in domains such as general software development and the design of gamification services. This is rather expected, as within a field that is still very much in development, plenty of work is required on the conceptualization and design of solutions, before it is feasible to produce empirically testable prototypes or systems. This is evident also from the noticeable number of theoretical papers, present among the non-empirical works. 3.4. Methods and data As shown in Table 7, the empirical research papers most frequently employed quantitative research methods. Mixed methods studies were also numerous, with qualitative research papers featuring in the minority. The category labeled as ‘mixed’ refers to empirical papers which combined any forms of qualitative and quantitative approach. The quantitative research papers were further categorized based on the approach taken in their analyses, as either descriptive or inferential. The majority of the empirical quantitative research on gamification is descriptive, meaning that the analyses of the data are often reported as percentages and means drawn from the numerical data. In the inferential studies, some relationship between variables has been examined, and the results are reported. For studies using a mixed methodology, the type of quantitative analyses was not recorded. However, if studies combining qualitative and quantitative research approaches were considered, then the amounts of both descriptive and inferential quantitative research would be higher. The prevalence of quantitative research in the empirical research on gamification is potentially due to a large part of the research being published in technical and computer science related venues, which commonly employ quantitative approaches. The small number of fully qualitative research papers (16.8% of the empirical studies) is also noteworthy, and may again be a consequence of the disciplines and research fields where the research is published. However, if the mixed methods studies are accounted for, then qualitative research approaches feature in nearly 40% of the empirical studies. Table 8 reports the data gathering methods employed in the empirical research papers, the types of data used for their analyses, and the analysis methods that were used. The most commonly used method of gathering data (and consequently the most common form of data) was by surveys and questionnaires, with the data being analyzed either qualitatively or quantitatively. The second most common data gathering method used a system implementation or prototype, from which some form of usage or log data was gathered. Also popular were experimental settings and interviews methods. For the data types, various forms of feedback and observation data formed also large categories. Regarding the analysis methods employed, quantitative descriptive analyses as well as qualitative analyses were reported in most studies. While there was a wide variety of data gathering methods and data types used in the analyses, the research approaches tended to Table 7 Type of empirical research. Empirical type Frequency Frequency % % Quantitative 165 60.4 descriptive 85 51.5 inferential 80 48.5 Qualitative 46 16.8 Mixed 62 22.7 Total 273 100 J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 197 concentrate on certain methods and data types. A popular structure for data gathering was where a gamification implementation or a prototype had been built, then a group of study participants used the implementation and some kind of usage data was gathered. The participants also tended to take part in a quantitative or a qualitative survey that was either numerical or with open-ended questions. 3.5. Affordances Altogether, 47 different affordances were identified in the studies (see Table 9). These were subsequently grouped based on their type, into achievement/progression-oriented (10 affordances), social-oriented (7), immersion-oriented (5), real world-related (8) and miscellaneous (17) elements. On average, the 273 empirical studies examined implementations or systems with 3.5 affordances. The affordance identifications were formed, depending on how the authors referred to the elements in their paper, without further analysis on similarities or differences between the affordances featured in different studies. This decision was taken due to the amount of the reviewed papers, which would have made a more detailed analysis of the implemented affordances more challenging. Although some overlap between the individual elements may exist (e.g. displaying a score might be very tightly connected to a progress bar in a system), for the purposes of this overview, each element was assigned to only one category. The most commonly implemented gameful affordances involve various forms of points and scoring. Also different forms of challenges, clear goals, achievements and leaderboards were among the most frequent elements. Generally in game design, points, achievements and leaderboards have been categorized as goal metrics that provide performance feedback to the player (Zagal, Mateas, Fernandez-Vara, Hochhalter, & Lichti, 2005). The prevalence of these affordances in gamification implementations is potentially due to them being easily applicable to various types of existing system (Mekler, Brühlmann, Tuch, & Opwis, 2015). In this study, the frequency of various challenge/clear goal implementations differs from the findings of previous reviews (Hamari et al., 2014), where such affordances were not as common. In this review, all of the studies which explicitly stated to contain some form of challenge, quest, mission, task or clear goal were categorized into this group. However, it is clear that challenges and goals overlap heavily for example with badge-type affordances, which generally provide a goal to work towards (Hamari, 2017). Similarly for instance, leaderboards and levels may act as goals that a user aims to reach. Therefore, the difference between this and prior reviews with regards to the prevalence of the challenge/clear goal affordances might be due to differences in the coding of the literature. In the reviewed literature, the application of various achievement or progression signaling affordances is generally the most common way to gamify activities. The second most frequent provision of affordances involved social elements in various forms. Especially, different features common to social networking services, e.g. friending, liking, status updates, commenting and profile pages are often implemented as gamification features. Cooperation and team-based activities also featured frequently among social affordances. Only 25 papers clearly articulated competition as an element, however, gamification implementations with leaderboards or other means of social comparison also promote a sense of competition among users. Therefore, competition as a concept is most likely to be more prevalent in actual implementations, but not necessarily specified in the research reports. Various immersion-oriented affordances such as the use of stories and narratives, avatars, virtual worlds etc. were featured, but these were not as commonly implemented as achievement and social affordances. The analysis of the affordances employed in the reviewed empirical literature indicates that the triad of points, badges and leaderboards continues to dominate the landscape of gamification. Several critical Table 8 Data gathering methods employed in the reviewed body of literature. Data gathering method Frequency Data type Frequency Survey, questionnaire (qualitative and quantitative) 179 Survey data 167 Implementation, prototype 161 Use data, log data 128 Experiment, trial 78 Interview data 48 Interviews 53 Feedback 25 Course, learning session 18 Observation data 21 Observation 16 Test scores, assignment scores 15 Focus group 9 Audio, video recordings, photos 6 Diaries 6 Diary entries, daily reports 5 Course assignments, education related assignments, tests 6 Focus group data, discussion data 5 Data from a system or a platform (existing systems; commercial products) 5 Course attendance data, other course data 3 Workshop 4 Field notes, experiential data (phenomenological data) 3 Data mining 3 Blog texts, forum discussions 2 Video data gathering 2 Delphi method data 1 Case study 2 Psychophysiological data 1 Ethnography 2 Brainstorming data 1 Google blog search 1 Analysis method Frequency Audio data gathering 1 Quantitative descriptive 136 Phenomenological assessment 1 Qualitative 100 Photography 1 Quantitative modeling (T-tests, Regressions, Mann-Whitney U-test, Wilcoxon rank-sum test, Structural equation modeling, Wilcoxon signed rank test, Welch’s t-test) 70 Delphi method 1 Quantitative comparisons (ANOVA/ANCOVA/MANOVA, Kruskal-Wallis) 44 Work session 1 Quantitative association-based (Correlations, Chi squares, Factor analysis, Crosstabs, Spearman’s rho) 30 Contest 1 Statistical quantitative (Binomial tests, Social network analysis, Growth curve analysis, Cluster analysis, Logistic models, Fuzzy AHP, Granger causality test, Hierarchical linear modeling, Z-test for proportions) 11 Cognitive ability test 1 Forum discussions 1 J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 198 views regarding the prevalence of these elements have been voiced, suggesting that employing such affordances without further consideration of the context or the users results in mere “pointsification” of the activities (see Deterding, 2015). One reason for the continued popularity of these elements may be that inserting them as an additional layer to an existing system can be achieved without undue effort (Mekler et al., 2015). Another potential explanation for the popularity of the blueprint design of points, badges and leaderboards may be that many gamification design guides and frameworks approach gamification design from a pattern-based perspective (Seffah & Taleb, 2012), and suggest designing gamification implementations by basically choosing the affordances from a given list of elements (Deterding, 2015). In reality, gamification is difficult to design for three main reasons: 1) games are complex, multifaceted, and therefore difficult to holistically transfer to other environments; 2) gamification involves motivational information system design (Hamari, 2015) which entails an understanding of (motivational) psychology; and 3) the goal of gamification is often to affect behavior, and this adds yet another layer to the scope of gamification design. Moreover, gamification design is targeted to a variety of audiences and activities, as well as serving a range of motivational needs which individuals may have in the varying gamified contexts. Deterding (2015) highlights the tendency of gamification guides and frameworks to promote the idea of a certain design pattern leading to a specific effect. However, as Deterding (2015) notes, from a psychological perspective, the motivational effect of every given situation is the result of “situated, active interpretation”. In other words, how an individual perceives the gamification is highly dependent on the nature of the activity, the contextual factors related to it, as well as the specific situation where the system is being used - all in addition to the individual’s own personal and demographic characteristics. As called for by Hamari et al. (2014), some studies looking at individual elements and their effects have started to appear (see e.g. Mekler et al., 2015;Christy & Fox, 2014). These studies have provided valuable information for research and practice by examining a specific relationship between an affordance and an outcome. Given their context dependency, the results of such studies are not universally applicable to all gamification settings, but they still provide much needed support for making design choices. 3.6. Outcomes The empirical studies were examined for any psychological and behavioral outcomes that were featured in the papers (reported in Tables 10 and 11 respectively). Psychological outcomes were quantitatively studied in 138 of the 273 empirical studies. The analysis of psychological outcomes indicates that the empirical research on gamification is mostly interested in how gamification implementations are perceived and experienced as systems, whether they are enjoyable or useful, and whether the users feel motivated by the systems. Most commonly, the empirical research papers examined perceptions of using a system, some specific system features, or some other assessments of use experiences. In a large portion of the studies, an implementation or a prototype had been developed and was subsequently studied with its users, so it is not surprising that inquiries regarding user experiences and perceptions are commonplace. Other frequently studied psychological outcomes reflect the most common reasons for implementing gamification. As games are generally associated with experiences of enjoyment, the application of gamification is often laden with an intention of creating enjoyment for the user. Enjoyment and experiences of “fun” were the second most frequent psychological outcome featured in the empirical research papers. Similarly, gamification is commonly framed as a method for increasing motivation towards various activities and tasks. Consequently, many of the empirical studies examine motivation as a psychological outcome. Further aspects such as perceived usefulness or effectiveness, or the ease of use or effort required to use a system were frequently examined as psychological outcomes. According to theories on technology acceptance and adoption, these aspects are considered to be important determinants for the continued use of various systems. Social aspects were also studied in many of the empirical research Table 9 Affordances studied in the empirical research papers. Affordances Affordances Achievement/progression Non-digital elements Points, score, XP 138 Real world/financial reward 16 Challenges, quests, missions, tasks, clear goals 91 Check-ins, location data 16 Badges, achievements, medals, trophies 85 Motion tracking 10 Leaderboards, rankings 82 Physical cards 4 Levels 59 Physical playboard 2 Performance stats (includes visualization of agreement in crowdsourcing), performance feedback 46 Real world interactive objects 1 Progress, status bars, skill trees 32 Physical objects as game resources 1 Quizzes, questions 32 Physical dice 1 Timer, speed 23 Miscellaneous Increasing difficulty 11 Full game (also board games), also commercial gamification systems not described 17 Social Assistance, virtual helpers 15 Social networking features 49 Virtual currency 10 Cooperation, teams 47 Reminders (to create engagement), cues, notifications, annotations 9 Competition 25 Retries, health, health points 7 Peer-rating, also betting to review work of others 17 Onboarding (safe environment to practice the rules), benefits for beginners 3 Customization, personalization 7 Adaptive difficulty 3 Multiplayer 3 Game rounds 2 Collective voting 1 Warnings 1 Immersion Penalties 1 Avatar, character, virtual identity 29 Game slogans 1 Narrative, narration, storytelling, dialogues, theme 22 Funny movies 1 Virtual world, 3D world, game world 14 Virtual pets 1 In-game rewards 13 Trading 1 Role play 6 Making suggestions 1 Virtual objects as augmented reality 1 J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 199 Research investigating the different determinants of why people play different types of games could prove useful in these endeavours (e.g Hamari & Keronen, 2017). Theoretical agenda point 1) Future gamification research should pay more attention to the pre-determinants/requirements of gamification success, instead of only the effectiveness of gamification for those users who have already chosen to adopt it. Moreover, gamification applications are inherently motivational information systems that attempt to support people in their goals and tasks related to the system use (Hamari & Koivisto, 2015b). In more specific terms, gamification can be seen to make goals more SMART (Burke, 2014;Hamari, 2013,2015); that is, more Specific, Measurable, Attainable, Realistic and Time-bound. According to goal-setting theories and decades of research, such goals assist individuals towards the attainment of their objective (Locke & Latham, 2002;Mann, De Ridder, & Fujita, 2013). This phenomenon has been observed and postulated in the context of gamification in various works (Burke, 2014;Hamari, 2013,2015;Landers, Bauer, & Callan, 2015). Although several studies have discussed the relationship between gamification and goal-setting (see e.g. Hamari et al. 2018), there is a current dearth of literature relating to goals themselves in gamification. Therefore, for a future agenda on gamification, we suggest an investigation into the relationship of the effects of gamification, depending on what kinds of goals users have, how goal-oriented they are, and what kinds of tasks they look to accomplish. Users do not share the same types of goals, nor do they have the same orientations toward them. For example, some users may be more oriented towards the outcomes of goals, whereas other users are more concerned about the process of reaching their goals (Locke & Latham, 2002;Mann et al., 2013). Therefore, a pertinent question is what kinds of gamification initiatives might be better suited to users, depending on their goals and the orientation they have towards them? Goals differ regarding their defining characteristics, for example in their difficulty or specificity (Elliot & Harackiewicz, 1994; Freund, Hennecke, & Riediger, 2010;Mann et al., 2013), hence they differ in their attainability and goal seeking outcomes (Freund et al., 2010;Hackel, Jones, Carbonneau, & Mueller, 2016;Landers et al., 2015;Lunenburg, 2011;Mann et al., 2013). Therefore, the design principles will differ depending upon the goals they address, and a single design solution cannot be expected to fit every situation (Koivisto & Hamari, 2014;Mann et al., 2013;Wang, Schneider, & Valacich, 2015). Theoretical agenda point 2) Future research into the effectiveness and adoption of gamification should take into account the role of the user, their goals (within the information system), and their individual attributes. In addition to the factors related to users, the usage context as well as the nature of the gamified service need to be given more attention. How the users perceive the gamification is highly dependent not only on the users’ characteristics, but also on how they perceive the context for the gamification (the domain or the environment where the gamification takes place), as well as the specific activity they are encouraged to perform (see e.g. Hamari, 2013). Gamification has been heavily implemented in the education context where it seems to fit rather intuitively: learning new skills, especially in an institutional setting such as a school, has traditionally been imbued with a similar structure, i.e. progressing in steps and receiving feedback for each step. Thus, the gamification of such a context does not feel particularly inappropriate or awkward. However, when gamifying areas such as health or social services, the context is evidently a lot more sensitive. For example, the gamification of an eCommerce service might be perceived differently than the gamification of a social networking service. The former is potentially perceived as a very utilitarian context, and playful or gameful elements might lessen the perception of seriousness and divert the customers away from the service. However, gamifying a social networking service would not cause such a reaction, due to the context being more casual in the first place. The lack of acknowledging the contextual factors in research suggests a lack of the theoretical understanding surrounding the phenomenon; the factors affecting human behavior, which in the case of gamification is more often than not the focus of the systems and the research conducted on them. When not acknowledging, for example, the environment of the gamification as well as the specific characteristics of the gamified activity, we risk producing research results which in reality are not applicable outside the very specific setting of a given study. Moreover, we ignore the chance for developing more comprehensive theoretical understanding of the phenomenon. Theoretical agenda point 3) Future gamification research should incorporate the contexts in which the gamification is deployed and investigated more strongly into research models. By way of its many motivational affordances, gamification can primarily be seen to attempt to provide users with feedback. Gamification provides three types of feedback: 1) cognitive, 2) affective, and 3) social. Cognitively, gamification commonly uses data on a user’s behavior to derive points and other indicators of progress, thus providing instrumental cognitive data about the users’ actions. From this perspective, gamification can be seen as a decision support system for the self, quantifying individual rather than organizational activities. Affective, motivational feedback is at the core of many gameful affordances as game design elements often aim for positive emotional arousal, such as enjoyment, excitement or interest. Finally, many gamification affordances are inherently social; e.g. high score lists afford social comparison (Festinger, 1954), and mutual goals can afford sense of community and strengthen ‘we-intentions’ (Tuomela, 1995). However, while gamification functions through these feedback systems and loops, neither the current theoretical understanding of gamification nor the empirical literature have made any serious inquiries into the different types of feedback that gamification affords. It appears that feedback functions as an essential mediator between the interaction with a gamification and the resulting psychological outcomes. Whilst prior literature has investigated the resultant psychological states and experiences (such as e.g. usefulness, enjoyment and perceived competition), it appears that the link between gamification affordances and resulting psychological states is still unexplored. Thus, it is unknown through which mechanisms gamification produces the psychological effects it aims to achieve. Theoretical agenda point 4) Future gamification research should pay more attention to the different types of feedback, which kinds of gamification affordances are best equipped to deliver them, and the effect that the feedback has on system users. The current understanding of gamification highlights that its effects proceed in a linear chain of events. This is reflected in both the theoretical and empirical literature on gamification. For example, the most cited definitions of gamification by Deterding et al. (2011) and Huotari and Hamari (2012);Huotari & Hamari, 2017, both conceptualize gamification as a process within which the implemented elements linearly proceed to affecting psychological states and experiences, and eventually user behavior. Similarly, the main body of empirical literature on gamification (as seen in this and previous reviews) treats gamification as following a similar process. Even though this is understandable from the perspective of economizing and simplifying empirical research design, it offers a rather limited view of the multifaceted complex systems and processes that gamification entails. Gamification is a dynamic, cyclical, two-way process in which the technology, the users, and the contextual factors of the system all contribute to the outcomes which are achieved. Gamification affects the behavior of the users, who continue the behavior, but not as the same “clean slates” as when they first adopted the system. So, it is clear that the behavior is altered directly due to the effect of the gamification, and that new patterns of behavior result in new responses to the gamification itself. Evidently, studying the process of gamification empirically is very challenging, and addressing these questions most likely calls for the adoption of qualitative research approaches that can capture the many facets of the complex phenomenon. However, simply acknowledging J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 206 the multifaceted nature of the gamification beyond the current linear conceptualizations that are seen in literature and practice would be a step forward. Theoretical agenda point 5) Gamification research and its theoretical conceptualizations should make an acknowledgement of the dynamic, cyclical nature of gamification. 4.3. Methodological agenda While the body of literature on gamification has been growing steadily, some common issues still hinder the development of the field. Firstly, the research field is very scattered in terms of the research models and variables which are used, especially when studying experiences and attitudes. While variables such as enjoyment are often included in the research models examining gamification, there are considerably few studies using similar or validated measurement instruments for a particular variable. Therefore, comparing research results or conducting any comprehensive meta-analyses poses a significant challenge. In order for the research field to develop, a consistency in measurement instruments and research models is needed across studies, in order to gain comparable results. Furthermore, many empirical studies rely on reporting only descriptive statistics, even though conducting some form of inferential analyses could also be feasible in many cases. Methodological agenda point 1) Future gamification research should aim for consistency in measurement instruments and research models, as well as developing the depth of analyses to go beyond a mere description of data. Secondly, most of the empirical research on gamification is conducted without control groups. While many of the studies are motivated by the question of whether the gamification approach is effective or not, in many cases, the studies eventually fail to accurately answer the question due to a lack of control groups. Furthermore, even if empirical approaches are used, a large proportion of the studies rely simply on user evaluations. Previous literature reviews have already pointed out these shortcomings (Hamari et al., 2014), and the amount of controlled, experimental research settings has increased as the field has matured. However, as noted in this review that out of 227 studies using quantitative methods (165 quantitative studies and 62 mixed methods studies), only 66 studies were identified as controlled experiments, there are still considerable steps to be taken in order to strengthen the research field. Methodological agenda point 2) Future gamification research should increasingly employ controlled experimental research methods, in order to gain knowledge on the actual effects of gamification. Thirdly, a common methodological problem within the gamification research field has been the study designs, which more often than not do not control between the various affordances implemented in the studied systems. Many studies examine the effects or perceptions of gamified systems containing several different elements, but as a whole. In these types of research settings, identifying which element actually causes the observed effects becomes impossible. Furthermore, investigating how much the different affordances contribute to the results is similarly very challenging, if none of the elements are controlled for. As reported in the analyses, only 11 of the 66 controlled experimental studies examined the effects of individual affordances. Thus, both research design and the overall research field would benefit from more work which identifies the effects of different affordances, as more information could be gained on the kinds of gamification elements that actually work. However, it must be remembered that even when controlling for the effects of a certain affordance, the contextual factors as well as individual user characteristics are likely to affect the results. For example, positive findings regarding the effectiveness of leaderboards for a class of schoolchildren does not guarantee that a similar leaderboard design would produce similar results in a work place setting. Methodological agenda point 3) Future gamification research should seek to control for the effects of the individual affordances used in a given gamification implementation. In addition, when investigating the effects of the affordances, the contextual characteristics of the setting should also be taken into account. Fourthly, as previously mentioned, the empirical research on gamification has been limited in terms of sample sizes, as well as experimental timeframes. Developments on this front can be seen to have taken place, when comparing the results of this review with the findings of the earlier review by Hamari et al. (2014), and the sample sizes of quantitative experimental studies have increased considerably. Yet, there are still several studies with only small groups of study participants. One explanation for these small samples is the nature of the studies, as many papers report preliminary exploratory research by testing a prototype or a concept. However, in order to advance the field in general, research must eventually move beyond isolated works with prototypes and aim for more theory-informed confirmatory studies. Furthermore, short timeframes pose an evident threat to the validity of study findings. Especially, novelty has been shown to have an effect on users of gamification services (Farzan et al., 2008b;Koivisto & Hamari, 2014), and with short time periods for data gathering, the risk of findings being skewed by the novelty of the implementation is elevated. Methodological agenda point 4) In future gamification research, the sample sizes of studies should be large enough to increase methodological rigor, as well as to amplify the transferability and explanatory power of the results. Furthermore, the time spans of studies should be long enough to enable novelty effects in the data to be minimized. Fifthly, in many research papers the reporting of the methods, data, analysis and results is unclear. Part of this problem is potentially caused by the abundance of conference publications which duly limits the space available for research papers, and therefore, limits the details that can be included. While this is naturally not an excuse for poor reporting, in some cases it may have contributed to the quality and clarity of the reports. In any case, an encouragement of precise and thorough reporting would enable much more efficient diffusion of research results. Methodological agenda point 5) In future gamification research, attention should be paid to clear and comprehensive reporting of research. 4.4. Limitations of the review In this review, we have followed the suggestions by Paré et al. (2015) to ensure its quality, in terms of both rigor and relevance. The review procedure has been described in detail to ensure the clarity of the process, and to enable replication of the procedure. Furthermore, the goals of the review have been explicitly stated in order to ensure the suitability of the chosen methods for the expressed goals. However, the chosen perspective and methods limit the review in different ways. The present review focuses on the phenomenon of gamification on an overview level. There is evidently variation, for example in how gamification has been defined in the different publications, or how the various affordances have been defined and implemented. Due to the goal of comprehensively overviewing a substantial body of literature, there has been no possibility to go into further detail of individual studies. In the coding and analysis processes, some abstraction has obviously been necessary, which has consequently caused some specifics of the studies to be lost. Furthermore, the literature search was limited to the Scopus and AISeL databases. While we are confident of the comprehensiveness of our literature search, it is nevertheless possible that some publications have been missed due to either not being among venues indexed in these databases, or due to indexing errors within the databases (as is the case with any review study). In any case, the potential number of missed publications is likely to be meager, and their inclusion would not foreseeably affect the results of the review to any notable degree. J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 207 Declaration of interest No conflicts of interest exist. Acknowledgements This work has been supported by the Business Finland (5479/31/ 2017, 40111/14, 40107/14 and 40009/16) and participating partners, Satakunnan korkeakoulusäätiö and its collaborators, and Academy of Finland (Center of Excellence - GameCult). References Agarwal, R., & Karahanna, E. (2000). Time flies when you’re having fun: Cognitive absorption and beliefs about information technology usage. MIS Quarterly, 24(4), 665–694. Ajzen, I. (1988). Attitudes, personality, and behaviour. Chicago, IL: Dorsey Press. Ajzen, I. (1991). The theory of planned behaviour. Organizational Behavior and Human Decision Processes, 50(2), 179–211. Alahäivälä, T., & Oinas-Kukkonen, H. (2016). Understanding persuasion contexts in health gamification: A systematic analysis of gamified health behavior change support systems literature. International Journal of Medical Informatics, 96, 62–70. Alcivar, I., & Abad, A. G. (2016). Design and evaluation of a gamified system for ERP training. Computers in Human Behavior, 58, 109–118. Allam, A., Kostova, Z., Nakamoto, K., & Schulz, P. J. (2015). The effect of social support features and gamification on a web based intervention for rheumatoid arthritis patients: Randomized controlled trial. Journal of Medical Internet Research, 17(1), e14. Araújo, J., & Pestana, G. (2017). A framework for social well-being and skills management at the workplace. International Journal of Information Management, 37(6), 718–725. Atkinson, M. A., & Kydd, C. (1997). Individual characteristics associated with world wide web use: An empirical study of playfulness and motivation. Data Base for Advances in Information Systems, 28(2), 53–62. Attali, Y., & Arieli-Attali, M. (2014). Gamification in assessment: Do points affect test performance? Computers & Education, 83, 57–63. Bista, S. K., Nepal, S., Paris, C., & Colineau, N. (2014). Gamification for online communities: A case study for delivering government services. International Journal of Cooperative Information Systems, 23(2). Bittner, J. V., & Schipper, J. (2014). Motivational effects and age differences of gamification in product advertising. The Journal of Consumer Marketing, 31(5), 391–400. Blohm, I., & Leimeister, J. M. (2013). Gamification - design of IT-Based enhancing services for motivational support and behavioral change. Business & Information Systems Engineering, 5(4), 275–278. Bonde, M. T., Makransky, G., Wandall, J., Larsen, M. V., Morsing, M., Jarmer, H., et al. (2014). Improving biotech education through gamified laboratory simulations. Nature Biotechnology, 32(7), 694–697. Buisman, A. L. D., & van Eekelen, M. C. J. D. (2014). Gamification in educational software development. Proceedings of the Computer Science Education Research Conference (pp. 9–20). Burke, B. (2014). Gamify: How gamification motivates people to do extraordinary things. Bibliomotion, Inc. Butler, C. (2014). A framework for evaluating the effectiveness of gamification techniques by personality type. International Conference on HCI in Business (Heraklion, Crete, Greece, June 22-27, 2014)381–389. Cafazzo, J. A., Casselman, M., Hamming, N., Katzman, D. K., & Palmert, M. R. (2012). Design of an mHealth app for the self-management of adolescent type 1 diabetes: A pilot study. Journal of Medical Internet Research, 14(3), e70. Caillois, R. (1961). Man, play and games. Urbana, IL: University of Illinois Press. Cechanowicz, J., Gutwin, C., Brownell, B., & Goodfellow, L. (2013). Effects of gamification on participation and data quality in a real-world market research domain. Proceedings of Gamification’ 13 Stratford (Ontario, Canada, October 2–4, 2013)58–65. Chen, Y., & Pu, P. (2014). HealthyTogether: Exploring social incentives for mobile fitness applications. Proceedings of Chinese CHI’ 14 (Toronto, Ontario, Canada, April 26–27) 25–34. Chen, C. H., Sun, C. T., & Hsieh, J. (2008). Player guild dynamics and evolution in massively multiplayer online games. CyberPsychology & Behavior, 11(3), 293–301. Chen, Y., Zhang, J., & Pu, P. (2014). Exploring social accountability in pervasive fitness apps. Proceeding of the UBICOMM2014 (Rome, Italy, August 24–28, 2014)221–226. Choi, J., Choi, H., So, W., Lee, J., & You, J. (2014). A study about designing reward for gamified crowdsourcing system. Proceedings of 3rd International Conference on Design, User Experience, and Usability: User Experience Design for Diverse Interaction Platforms and Environments (pp. 678–687). Christy, K. R., & Fox, J. (2014). Leaderboards in a virtual classroom: A test of stereotype threat and social comparison explanations for women’s math performance. Computers & Education, 78, 66–77. Cole, H., & Griffiths, M. D. (2007). Social interactions in massively multiplayer online role-playing gamers. CyberPsychology & Behavior, 10(4), 575–583. Csíkszentmihályi, M. (1975). Beyond boredom and anxiety: Experiencing flow in work and play. San Francisco: Jossey-Bass. Csíkszentmihályi, M. (1990). Flow: The psychology of optimal experience. New York: Harper and Row. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1992). Extrinsic and intrinsic motivation to use computers in the workplace. Journal of Applied Social Psychology, 22, 1111–1132. Deci, E. L., & Ryan, R. M. (1985). Self-determination. New Jersey: John Wiley and Sons Inc. Deci, E. L., & Ryan, R. M. (2000). The ‘what’ and ‘why’ of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11(4), 227–268. de-Marcos, L., Domínguez, A., Saenz-de-Navarrete, J., & Pagés, C. (2014). An empirical study comparing gamification and social networking on e-learning. Computers & Education, 75, 82–91. Denny, P. (2013). The effect of virtual achievements on student engagement, Proceedings of CHI 2013: Changing perspectives (Paris, France, April 27–May 2, 2013). 763–772. Deterding, S. (2015). The lens of intrinsic skill atoms: A method for gameful design. Human-Computer Interaction, 30(3-4), 294–335. Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: Defining gamification. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future media Environments (Tampere, Finland, September 28–30, 2011)9–15. Domínguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fernández-Sanz, L., Pagés, C., & Martínez-Herráiz, J.-J. (2013). Gamifying learning experiences: Practical implications and outcomes. Computers & Education, 63, 380–392. Eickhoff, C., Harris, C. G., de Vries, A. P., & Srinivasan, P. (2012). Quality through flow and immersion: Gamifying crowdsourced relevance assessments. Proceedings of the 35th International ACM SIGIR Conference on Research and Development in InformaTion Retrieval (Portland, Oregon, USA, August 12–16, 2012)871–880. Elliot, A. J., & Harackiewicz, J. M. (1994). Goal setting, achievement orientation, and intrinsic motivation: A mediational analysis. Journal of Personality and Social Psychology, 66(5), 968–980. Farzan, R., & Brusilovsky, P. (2011). Encouraging user participation in a course recommender system: An impact on user behavior. Computers in Human Behavior, 27(1), 276–284. Farzan, R., DiMicco, J. M., Millen, D. R., Brownholtz, B., Geyer, W., & Dugan, C. (2008a). When the experiment is over: Deploying an incentive system to all the users. Symposium on persuasive technology (Aberdeen, Scotland, April, 2008). Farzan, R., DiMicco, J. M., Millen, D. R., Brownholtz, B., Geyer, W., & Dugan, C. (2008b). Results from deploying a participation incentive mechanism within the enterprise. Proceedings of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 5–10, 2008)563–572. Festinger, L. (1954). A theory of social comparison processes. Human Relations, 7(2), 117–140. Filsecker, M., & Hickey, D. T. (2014). A multilevel analysis of the effects of external rewards on elementary students’ motivation, engagement and learning in an educational game. Computers & Education, 75, 136–148. Fishbein, M. (1979). A theory of reasoned action: Some applications and implications. Nebraska Symposium on Motivation Nebraska Symposium on Motivation, 27, 65–116. Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention, and behavior: An introduction to the theory and research. Reading, MA: Addison-Wesley. Freund, A. M., Hennecke, M., & Riediger, M. (2010). Age-related differences in outcome and process goal focus. The European Journal of Developmental Psychology, 7(2), 198–222. Gartner (2011). Gartner says by 2015, more than 50 percent of organizations that manage innovation processes will gamify those processes [WWW document]. (Accessed 14th of February 2017) http://www.gartner.com/it/page.jsp?id=1629214. Gerow, J. E., Ayyagari, R., Thatcher, J. B., & Roth, P. L. (2013). Can we have fun @ work? The role of intrinsic motivation for utilitarian systems. European Journal of Information Systems, 22(3), 360–380. Hackel, T. S., Jones, M. H., Carbonneau, K. J., & Mueller, C. E. (2016). Re-examining achievement goal instrumentation: Convergent validity of AGQ and PALS. Contemporary Educational Psychology, 46, 73–80. Hakulinen, L., Auvinen, T., & Korhonen, A. (2013). Empirical study on the effect of achievement badges in TRAKLA2 online learning environment. Proceedings of Learning and Teaching in Computing and Engineering (LaTiCE) Conference (Macau, March 21–24, 2013)47–54. Hakulinen, L., Auvinen, T., & Korhonen, A. (2015). The effect of achievement badges on students’ behavior: An empirical study in a university-level computer science course. International Journal of Emerging Technologies in Learning (iJET), 10(1), 18–29. Hall, M., Glanz, S., Caton, S., & Weinhardt, C. (2013). Measuring your Best you: A gamification framework for Well-being measurement. Proceedings of 2013 International Conference on Cloud and Green Computing (Karlsruhe, Germany, September 30-October 2, 2013)277–282. Hamari, J., Hassan, L., & Dias, A. (2018). Gamification, quantified-self or social networking? Matching users’ goals with motivational technology. User Modelling and User-Adapted Interaction, 28(1), 35–74. Hamari, J. (2013). Transforming homo economicus into homo ludens: A field experiment on gamification in a utilitarian peer-to-peer trading service. Electronic Commerce Research and Applications, 12(4), 236–245. Hamari, J. (2017). Do badges increase user activity? A field experiment on effects of gamification. Computers in Human Behavior, 71, 469–478. Hamari, J. (2015). Why do people buy virtual goods? Attitude towards virtual good purchases versus game enjoyment. International Journal of Information Management, 35(3), 299–308. Hamari, J., Huotari, K., & Tolvanen, J. (2015). Gamification and economics. In S. P. Walz, & S. Deterding (Eds.). The gameful world: Approaches, issues, applications (pp. 139– 161). Cambridge, MA: MIT Press. Hamari, J., & Keronen, L. (2017). Why do people play games? A meta-analysis. International Journal of Information Management, 37(3), 125–141. Hamari, J., & Koivisto, J. (2015a). Working out for likes”: An empirical study on social J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 208 influence in exercise gamification. Computers in Human Behavior, 50, 333–347. Hamari, J., & Koivisto, J. (2015b). Why do people use gamification services? International Journal of Information Management, 35(4), 419–431. Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? – A literature review of empirical studies on gamification. Proceedings of the 47th Hawaii International Conference on System Sciences (Hawaii, USA, JanUary 6–9, 2014)3025–3034. Hamari, J., & Tuunanen, J. (2014). Player types: A meta-synthesis. Transactions of the Digital Games Research Association, 1(2), 29–53. Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers & Education, 80, 152–161. Hassan, L., & Hamari, J. (2019). Gamification of e-participation: A literature review. Proceedings of the 52nd Annual Hawaii International Conference on System Sciences (HICSS). Herzig, P., Strahringer, S., & Ameling, M. (2012). Gamification of ERP systems - exploring gamification effects on user acceptance constructs, in Multikonferenz Wirtschaftsinformatik 2012: Tagungsband der MKWI 2012. Berlin: Gito Verlag599–612. Hirschheim, R., & Klein, H. K. (2012). A glorious and not-so-Short history of the information systems field. Journal of the Association for Information Systems, 13(4) Article 5. Hirschman, E. C., & Holbrook, M. B. (1982). Hedonic consumption: Emerging concepts, methods and propositions. Journal of Marketing, 46, 92–101. Huang, H.-C., Cheng, T. C. E., Huang, W.-F., & Teng, C.-I. (2018). Impact of online gamers’ personality traits on interdependence, network convergence, and continuance intention: Perspective of social exchange theory. International Journal of Information Management, 38(1), 232–242. Huizinga, J. (1955). Homo ludens: A study of the play element in culture. Boston, MA: Beacon Press. Huotari, K., & Hamari, J. (2012). Defining gamification – A service marketing perspective. Proceedings of the 16th International Academic MindTrek Conference (Tampere, Finland, 3–5 October, 2012)17–22. Huotari, K., & Hamari, J. (2017). A definition for gamification: Anchoring gamification in the service marketing literature. Electronic Markets, 27(1), 21–31. IEEE (2014). Everyone’s a gamer – IEEE experts predict gaming will be integrated into more than 85 percent of daily tasks by 2020 [WWW document]. (Accessed 14th of February 2017) http://www.ieee.org/about/news/2014/25_feb_2014.html. Ipeirotis, P. G., & Gabrilovich, E. (2014). Quizz: Targeted crowdsourcing with a billion (potential) users. Proceedings of WWW’ 14 (Seoul, Korea, April 7–11, 2014)143–154. Jones, B. A., Madden, G. J., & Wengreen, H. J. (2014). The FIT game: Preliminary evaluation of a gamification approach to increasing fruit and vegetable consumption in school. Preventive Medicine, 68, 76–79. Kallio, K. P., Mäyrä, F., & Kaipainen, K. (2010). At least nine ways to play: Approaching gamer mentalities. Games and Culture 1555412010391089. Koivisto, J., & Hamari, J. (2014). Demographic differences in perceived benefits from gamification. Computers in Human Behavior, 35, 179–188. Landers, R. N. (2015). Developing a theory of gamified learning: Linking serious games and gamification of learning. Simulation & Gaming, 45, 752–768. Landers, R. N., & Armstrong, M. B. (2015). Enhancing instructional outcomes with gamification: An empirical test of the Technology-Enhanced Training Effectiveness Model. Computers in Human Behavior. Landers, R. N., Bauer, K. N., & Callan, R. C. (2015). Gamification of task performance with leaderboards: A goal setting experiment. Computers in Human Behavior. Landers, R. N., & Landers, A. K. (2015). An empirical test of the theory of gamified learning: The effect of leaderboards on time-on-task and academic performance. Simulation & Gaming, 45(6), 769–785. Laureyssens, T., Coenen, T., Claeys, L., Mechant, P., Criel, J., & Vande Moere, A. (2014). ZWERM: A modular component network approach for an urban participation game. Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems - CHI’ 14 (Toronto, ON, Canada, April 26-May 1, 2014)3259–3268. Lee, J. J., Ceyhan, P., Jordan-Cooley, W., & Sung, W. (2013). GREENIFY: A real world action game for climate change education. Simulation & Gaming, 44(2–3), 349–365. Liu, D., Santhanam, R., & Webster, J. (2017). Towards meaningful engagement: A framework for design and research of gamified information systems. MIS Quarterly, 41(4), 1011–1034. Liu, D., Xun, L., & Santhanam, R. (2013). Digital games and beyond? What happens when player compete? MIS Quarterly, 37(1), 111–124. Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. The American Psychologist, 57(9), 705–717. Long, Y., & Aleven, V. (2014). Gamification of joint student/system control over problem selection in a linear equation tutor. Proceedings of the 12th International Conference on Intelligent Tutoring Systems (pp. 378–387). Lounis, S., Pramatari, K., & Theotokis, A. (2014). Gamification is all about fun: The role of incentive type and community collaboration. Proceedings of ECIS 2014 (Tel Aviv, Israel, June 9–11, 2014)1–14. Lunenburg, F. C. (2011). Goal-setting theory of motivation. International Journal of Management, Business, and Administration, 15(1). Majuri, J., Koivisto, J., & Hamari, J. (2018). Gamification of education and learning: A review of empirical literature. Proceedings of the 2nd International GamiFIN conference (pp. 11–19). Malaby, T. M. (2007). Beyond play: A new approach to games. Games and Culture, 2(2), 95–113. Mann, T., De Ridder, D., & Fujita, K. (2013). Self-regulation of health behavior: Social psychological approaches to goal setting and goal striving. Health Psychology, 32(5), 487–498. Mekler, E. D., Brühlmann, F., Opwis, K., & Tuch, A. N. (2013). Do points, levels and leaderboards harm intrinsic motivation? Proceedings of the First International Conference on Gameful Design, Research, and Applications - Gamification ’13 (pp. 66–73). Mekler, E. D., Brühlmann, F., Tuch, A. N., & Opwis, K. (2015). Towards understanding the effects of individual gamification elements on intrinsic motivation and performance. Computers in Human Behavior. Moon, J.-W., & Kim, Y.-G. (2001). Extending the TAM for a world-wide-Web context. Information & Management, 38(4), 217–230. Morschheuser, B., Hamari, J., & Koivisto, J. (2016). Gamification in crowdsourcing: A review. Proceedings of the 49th Annual Hawaii International Conference on System Sciences (HICSS) (Hawaii, USA, January 5-8, 2016)4375–4384. Morschheuser, B., Hamari, J., & Maedche, A. (2018). Cooperation or competition–When do people contribute more? A field experiment on gamification of crowdsourcing. International Journal of Human-Computer Studies. Morschheuser, B., Hassan, L., Werder, K., & Hamari, J. (2018). How to design gamification? A method for engineering gamified software. Information & Software Technology, 95, 219–237. Morschheuser, B., Hamari, J., Koivisto, J., & Maedche, A. (2017). Gamified crowdsourcing: Conceptualization, literature review, and future agenda. International Journal of Human-computer Studies, 106, 26–43. Mäntymäki, M., Merikivi, J., Verhagen, T., Feldberg, F., & Rajala, R. (2014). Does a contextualized theory of planned behavior explain why teenagers stay in virtual worlds? International Journal of Information Management, 34(5), 567–576. Mäntymäki, M., & Riemer, K. (2014). Digital natives in social virtual worlds: A multimethod study of gratifications and social influences in Habbo Hotel. International Journal of Information Management, 34(2), 210–220. Mäntymäki, M., & Salo, J. (2013). Purchasing behavior in social virtual worlds: An examination of Habbo Hotel. International Journal of Information Management, 33(2), 282–290. Mäntymäki, M., & Salo, J. (2015). Why do teens spend real money in virtual worlds? A consumption values and developmental psychology perspective on virtual consumption. International Journal of Information Management, 35(1), 124–134. Mäyrä, F. (2008). Introduction to game studies: Games in culture. London & NY: Sage Publications. Panigrahi, R., Srivastava, P. R., & Sharma, D. (2018). Online learning: Adoption, continuance, and learning outcome—A review of literature. International Journal of Information Management, 43, 1–14. Paré, G., Trudel, M. C., Jaana, M., & Kitsiou, S. (2015). Synthesizing information systems knowledge: A typology of literature reviews. Information & Management, 52(2), 183–199. Reitz, L., Sohny, A., & Lochmann, G. (2016). VR-based gamification of communication training and oral examination in a second language. International Journal of GameBased Learning, 6(2), 46–61. Rigby, S., & Ryan, R. M. (2011). Glued to games: How video games draw us in and hold us spellbound. Santa Barbara, CA: Praeger. Riva, S., Camerini, A. L., Allam, A., & Schulz, P. J. (2014). Interactive sections of an internet-based intervention increase empowerment of chronic back pain patients: Randomized controlled trial. Journal of Medical Internet Research, 16(8). Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. The American Psychologist, 55(1), 68–78. Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006). The motivational pull of video games: A self-determination theory approach. Motivation and Emotion, 30(4), 344–360. Sanchez-Nielsen, E., & Lee, D. (2013). eParticipation in practice in Europe: The case of "puzzled by policy: Helping you Be part of EU". Proceedings of the 46th Annual Hawaii International Conference on System Sciences (HICSS) (Hawaii, USA, January 7-10, 20131870–1879. Santhanam, R., Liu, D., & Shen, W. C. M. (2016). Research note - gamification of technology-mediated training: Not all competitions are the same. Information Systems Research, 27(2), 453–465. Scharkow, M., Festl, R., Vogelgesang, J., & Quandt, T. (2015). Beyond the “core-gamer”: Genre preferences and gratifications in computer games. Computers in Human Behavior, 44, 293–298. Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International Journal of Human-computer Studies, 74, 14–31. Seffah, A., & Taleb, M. (2012). Tracing the evolution of HCI patterns as an interaction design tool. Innovations in Systems and Software Engineering, 8(2), 93–109. Simões, J., Díaz Redondo, R., & Fernández Vilas, A. (2013). A social gamification framework for a K-6 learning platform. Computers in Human Behavior, 29(2), 345–353. Sun, H., & Zhang, P. (2006). Causal relationships between perceived enjoyment and perceived ease of use: An alternative approach. Journal of the Association for Information Systems, 7(9), 618–645. Sørensen, J. J. W., Pedersen, M. K., Munch, M., Haikka, P., Jensen, J. H., Planke, T., Andreasen, M. G., Gajdacz, M., Mølmer, K., Lieberoth, A., & Sherson, J. F. (2016). Exploring the quantum speed limit with computer games. Nature, 532, 210–213. Talaei-Khoei, A., & Daniel, J. (2018). How younger elderly realize usefulness of cognitive training video games to maintain their independent living. International Journal of Information Management, 42, 1–12. Tavinor, G. (2009). The art of videogames. Malden, MA: Wiley-Blackwell. Teng, C.-I. (2017). Impact of avatar identification on online gamer loyalty: Perspectives of social identity and social capital theories. International Journal of Information Management, 37(6), 601–610. Teng, C. I., & Chen, W. W. (2014). Team participation and online gamer loyalty. Electronic Commerce Research and Applications, 13(1), 24–31. Terlutter, R., & Capella, M. L. (2013). The gamification of advertising: Analysis and research directions of in-game advertising, advergames, and advertising in social network games. Journal of Advertising, 42(2-3), 95–112. J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 209 Thom, J., Millen, D., & DiMicco, J. (2012). Removing gamification from an enterprise SNS, in proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (Seattle, Washington, USA, February 11–15, 2012). 1067–1070. Tolmie, P., Chamberlain, A., & Benford, S. (2014). Designing for reportability: Sustainable gamification, public engagement, and promoting environmental debate. Personal and Ubiquitous Computing, 18(7), 1763–1774. Tuomela, R. (1995). The importance of us: A philosophy study of basic social notions. Stanford, CA: Stanford University Press. van der Heijden, H. (2004). User acceptance of hedonic information systems. MIS Quarterly, 28(4), 695–704. Venkatesh, V. (1999). Creation of favorable user perceptions: Exploring the role of intrinsic motivation. MIS Quarterly, 23(2), 239–260. Vesa, M., Hamari, J., Harviainen, J. T., & Warmelink, H. (2017). Computer games and organization studies. Organization Studies, 38(2), 273–284. Wang, X., Schneider, C., & Valacich, J. S. (2015). Enhancing creativity in group collaboration: How performance targets and feedback shape perceptions and idea generation performance. Computers in Human Behavior, 42, 187–195. Warmelink, H., Koivisto, J., Mayer, I., Vesa, M., & Hamari, J. (2018). Gamification of production and logistics operations: Status quo and future directions. Journal of Business Research.https://www.sciencedirect.com/science/article/pii/ S0148296318304594. Watson, D., Mandryk, R. L., & Stanley, K. G. (2013). The design and evaluation of a classroom exergame. Proceedings of Gamification’ 13 (Stratford, ON, Canada, October 02–04, 2013)34–41. Webster, J., & Martocchio, J. J. (1992). Microcomputer playfulness: Development of a measure with workplace implications. MIS Quarterly, 16(2), 201–226. Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2), xiii–xxiii. Williams, D., Consalvo, M., Caplan, S., & Yee, N. (2009). Looking for gender: Gender roles and behaviors among online gamers. The Journal of Communication, 59(4), 700–725. Williams, D., Yee, N., & Caplan, S. E. (2008). Who plays, how much, and why? Debunking the stereotypical gamer profile. Journal of Computer-Mediated Communication, 13(4), 993–1018. Wu, J., & Lu, X. (2013). Effects of extrinsic and intrinsic motivators on using utilitarian, hedonic, and dual-purposed information systems: A meta-analysis. Journal of the Association for Information Systems, 14(3), 153–191. Xi, N., & Hamari, J. (2019). The relationship between gamification, brand engagement and brand equity. Proceedings of the 52nd Annual Hawaii International Conference on System Sciences (HICSS). Yates, M., Kelemen, A., & Sik Lanyi, C. (2016). Virtual reality gaming in the rehabilitation of the upper extremities post-stroke. Brain Injury, 30(7), 855–863. Yee, N. (2006). Motivations for play in online games. CyberPsychology & Behavior, 9(6), 772–775. Zagal, J. P., Mateas, M., Fernandez-Vara, C., Hochhalter, B., & Lichti, N. (2005). Towards an ontological language for game analysis. Proceedings of International DiGRA Conference: Changing Views - Worlds in Play (Vancouver, Canada, June 16–20, 2005) 3e14. J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210 210