The effectiveness of gamification in the online and offline qualitative marketing research

Ścibor-Rylski, Michał Article The effectiveness of gamification in the online and offline qualitative marketing research Journal of Marketing and Consumer Behaviour in Emerging Markets (JMCBEM) Provided in Cooperation with: Faculty of Management, University of Warsaw Suggested Citation: Ścibor-Rylski, Michał (2021) : The effectiveness of gamification in the online and offline qualitative marketing research, Journal of Marketing and Consumer Behaviour in Emerging Markets (JMCBEM), ISSN 2449-6634, University of Warsaw, Faculty of Management, Warsaw, Iss. 2, pp. 57-64, https://doi.org/10.7172/2449-6634.jmcbem.2021.2.4 This Version is available at: https://hdl.handle.net/10419/311523 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ DOI: 10.7172/2449-6634.jmcbem.2021.2.4 Journal of Marketing and Consumer Behaviour in Emerging Markets 2(13)2021 56 (56–64) © 2021 Authors. This is an open access journal distributed under the Creative Commons BY 4.0 license (https://creativecommons.org/licenses/by/4.0/) The Effectiveness of Gamifi cation in the Online and Offl ine Qualitative Marketing Research Michał Ścibor-Rylski Wydział Zarządzania, Uniwersytet Warszawski, Szturmowa 1/3, 02-678 Warszawa mscibor-r[email protected].edu.pl https://orcid.org/0000-0002-0594-492X Received: 9 November 2021/ Revised: 11 December 2021/ Accepted: 14 December 2021/ Published online: 30 December 2021 ABSTRACT Gamifi ed market research tools help to increase respondents’ engagement and obtain more indepth results. Up till now the eff ects of gamifi cation have been tested in the offl ine environment. The COVID-19 pandemic changed the world of the qualitative research and also triggered a need to replicate some previously presented eff ects. The article shows the results of two experiments proving the eff ectiveness of gamifi ed approach to the qualitative advertisement and product concept testing in an online environment. The experimental groups with a narrative context added to a question regarding the fi rst impression after viewing an advertisement or reading a product concept provided more elaborated answers to the open-ended questions than the control groups with a standard task. What’s important, for the advertisement test the eff ect was signifi cant for both online and offl ine conditions, whereas in the case of the product concept test only the online variant proved the superiority of the gamifi ed task. JEL classifi cation: M310, M370, M300 Keywords: gamifi cation, marketing research, online research 1. INTRODUCTION The use of game elements in the marketing research helps to acquire and retain the respondents’ attention making them more engaged in the process. There are numerous results proving the eff ectiveness of gamifi cation used in brand, communication and customer experience studies (Ścibor-Rylski, 2019, 2020). Previous research was conducted mainly offl ine using a questionnaire with regular vs. gamifi ed questions. The time of the COVID-19 pandemic forced a rapid change in how the marketing research was conducted – especially in the qualitative domain, where face to face contact dominated until 2020. Market research agencies have been running focus group and individual interviews online for over 1.5 years now and they needed to adapt proven tools to the new reality. A wooden table has been replaced with an online virtual co-working space provided by Miro or Mural platforms, paper questionnaires checking the fi rst impressions in the process of evaluating the concepts of advertisements and products have been digitalized. These new challenges in adapting the research methods to online requirements were an inspiration to replicate in a new, online environment some of the previous results showing the DOI: 10.7172/2449-6634.jmcbem.2021.2.4 Journal of Marketing and Consumer Behaviour in Emerging Markets 2(13)2021 Michał Ścibor-Rylski 57 (56–64) © 2021 Authors. This is an open access journal distributed under the Creative Commons BY 4.0 license (https://creativecommons.org/licenses/by/4.0/) superiority of gamifi ed tasks in the advertisement evaluation as well as to compare online and offl ine eff ects of the use of gamifi cation in the new research fi eld: product concept testing. The goal of this paper is to present the results of two experiments designed to recreate a natural market research situation of the advertisement and product concept evaluation. The main research focus was to compare the participants’ eff ectiveness in two conditions: regular question and gamifi ed one in offl ine vs. online formula. The fi rst experiment was designed as an online replication of an eff ect published last year (Ścibor-Rylski, 2020a). The second experiment used the same methodology and gamifi cation approach but the marketing research area has been changed. The main research question referred to the comparison of the eff ect sizes of the use of gamifi cation in an online and offl ine variant of the study. 2. LITERATURE REVIEW 2.1. Gamifi cation in the Marketing Research Gorączka and Protasiuk (2020) distinguish three levels of gamifi cation in the marketing research: – Surveytainent: not related to any game mechanics, mostly graphical improvement of a questionnaire enhancing the respondent’s experience of taking part in research. – Soft gamifi cation: the use of the game mechanics elements in research. The essential components of this level are: feedback, narration, challenge and competition. – Hard gamifi cation: the ultimate level of gamifi cation in marketing research – it is basically running a study as a game. Specially designed board and narrative games are used in qualitative research and appealing online games increase the respondents’ engagement in the quantitative approach. All the levels of gamifi cation mentioned above help researchers to increase respondents’ engagement in the process – more appealing tasks make people more motivated. Harrison (2011) shows that the use of gamifi cation leads to higher involvement and openness to discussion and sharing thoughts. Better motivation results in an increased completion rate and more positive experience – research participants consider the process more enjoyable (Triantoro, Gopal, Benbunan-Fich, & Lang, 2020; Zichermann & Cunningham, 2011). ‘Soft gamifi cation’ of questionnaires results in more elaborated responses – adding a narrative context to a research question signifi cantly increases the average number of generated items (e.g. brand associations or image traits) when compared to the regular approach (Puleston & Rintoul, 2012; Ścibor-Rylski, 2018, 2019). Next crucial benefi t of gamifying the marketing research processes is the depth of collected data (Bailey, Pritchard, & Kernohan, 2015). Applying ‘hard gamifi cation’ makes respondents immerse in the research and induces a ‘hot’ behavioural state facilitating the process of recreation of the motivations and the reasons behind consumers’ choices as well as emotional states. Playing a game makes people more effi cient in recalling some elements from the past – e.g. detailed memories of their customer journeys (Ścibor-Rylski, 2020b). 2.2. Gamifi cation Online Gamifi cation in the marketing research serves similar goals as in education – it is a tool to increase participants’ engagement and motivation, transforming the fl ow of a not very exciting activity into a rewarding task that activates a dopamine loop: “challenge-achievement-reward loop DOI: 10.7172/2449-6634.jmcbem.2021.2.4 Journal of Marketing and Consumer Behaviour in Emerging Markets 2(13)2021 Michał Ścibor-Rylski 58 (56–64) © 2021 Authors. This is an open access journal distributed under the Creative Commons BY 4.0 license (https://creativecommons.org/licenses/by/4.0/) promotes the production of dopamine in the brain, reinforcing our desire to play” (Zichermann & Cunningham, 2011, p. 4). The research in the fi eld of education proves that turning the process of learning into a game makes students more engaged and motivated (da Rocha Seixas, Gomes, & de Melo Filho, 2016; Homer, Jew, & Tan, 2018). The COVID-19 pandemic had a great impact on the learning process and also the use of a gamifi ed approach to education. Nieto-Escamez & Roldán-Tapia (2021) present a review of the research aimed at enhancing online learning by improving participants’ motivation and engagement. In most cases, a gamifi ed approach was considered eff ective and engaging, and also fun. Year 2020 changed also the way in which companies run marketing research – especially in the qualitative approach. Online methods became everyday practice. The raising popularity of online focus groups and marketing research online communities (MROCs – Baldus, 2013) puts new challenges to the application of gamifi cation in the qualitative fi eld of the market research and also requires careful scientifi c verifi cation of the eff ects of the use of gamifi ed tasks in such an environment. The use of game elements in quantitative online surveys is already a subject of researchers’ focus (Downes-Le Guin, Baker, Mechling, & Ruyle, 2012; Bailey, Pritchard, & Kernohan, 2015; Puleston, 2011), but it is still an unexplored territory in the online qualitative research. The research presented in this paper may be considered as a fi rst step to verify the eff ectiveness of gamifi cation in this fi eld. 3. DATA AND METHODS 3.1. The Research Goal and Operationalization of the Dependent Variable The results of two experiments will be presented in this part: 1. Online replication of a study on the eff ectiveness of gamifi cation in the communication research (Ścibor-Rylski, 2020a), 2. A study on the eff ectiveness of gamifi cation in the product concept testing research both in online and offl ine conditions. In both experiments, the same method of gamifying the task was used – a narrative context was included in the question about the fi rst impression regarding an advertisement or a product concept. The participants in the offl ine condition were gathered in a lecture room. The online version was sent to the respondents as a Google Forms link. Both versions of the questionnaire were identical. The dependent variable was defi ned as the research participants’ engagement refl ected in their eff ectiveness. The indicator of the dependent variable was the number of words used by the participants in their statement about the fi rst impression. 3.2. The Hypothesis and the Research Question As a result of the literature review presented above, a general hypothesis was formulated: Hypothesis: employing the gamifi cation technique based on adding a context to a question improves engagement of the research participants, resulting in better eff ectiveness. Additionally, due to a lack of previous studies regarding the use of online gamifi cation in the qualitative marketing research, a research question was asked: Research question: Is there any moderating eff ect of the condition (online and offl ine) of running marketing research on the eff ectiveness of gamifi cation? DOI: 10.7172/2449-6634.jmcbem.2021.2.4 Journal of Marketing and Consumer Behaviour in Emerging Markets 2(13)2021 Michał Ścibor-Rylski 59 (56–64) © 2021 Authors. This is an open access journal distributed under the Creative Commons BY 4.0 license (https://creativecommons.org/licenses/by/4.0/) 3.3. Research Methodology – Communication Research The results of the original experiment proved the eff ectiveness of a gamifi ed approach in qualitative advertisement testing in an offl ine condition (Ścibor-Rylski, 2020a). The experimental group with a narrative context added to a question regarding the fi rst impression was more eff ective than the control group with a standard task. The number of words written by the participants was used as an indicator of the dependent variable. The average word count for the control group was M = 37.77, while the participants from the experimental group wrote ten more words on average (M = 47.79). The diff erence was statistically signifi cant: t(57) = 2.05; p < 0.05; d = 0.53. The online replication of this experiment was conducted via Google Forms among 67 students of the Faculty of Management of the University of Warsaw. The participants were split into two groups: – control (no gamifi ed techniques were used) – experimental (with a gamifi ed question – a narrative context added). As in the original research, the participants were asked to watch a commercial – a TV advertisement of Castorama – a DiY retailer (Castorama Polska, 2018). The ad lasted 45 seconds and its plot was focused on a story of two neighbours falling in love. The brand inspired the male character to accomplish a project – create a roof garden that helped him charm his female neighbour. After watching the commercial, the participants were asked to fi ll out a form with only one question. Its content was diff erent in the control and in the experimental group. The control group was simply asked about their fi rst impressions. They received the following instruction: “Watch the commercial and write down what you think about it.” In the experimental group, a narrative context was introduced to the question. It is considered as one of the components of ‘soft gamifi cation’ (Gorączka & Protasiuk, 2020). They received the following instruction: “Imagine you work for an advertisement agency and you are working on a new campaign. Your biggest rival working for a competitor’s brand has just created a new ad – somehow you managed to watch it before the offi cial premiere. You need to react as soon as possible to design a relevant commercial as a response and not to fall behind. You want to share your thoughts with the creative team and your management. Watch the commercial and write down what do you think about it.” No time limit was imposed. The next subchapter presents the results of the comparison between two groups. As in the original experiment, the indicator of the eff ectiveness of gamifi cation was the word count in each participant’s statement. 3.4. Results – Communication Research The average word count of the statements was calculated in each group. The experimental group (N = 28) used more words: M = 38.97; SD = 30.61 compared to the control group (N = 31): M = 96.82; SD = 76.77. T-test was used to analyse the signifi cance of the diff erence. It revealed a statistically signifi cant diff erence and an average eff ect size: t(51) = 4.27; p < 0.001; d = 1.08. The results are presented in Chart 1. DOI: 10.7172/2449-6634.jmcbem.2021.2.4 Journal of Marketing and Consumer Behaviour in Emerging Markets 2(13)2021 Michał Ścibor-Rylski 60 (56–64) © 2021 Authors. This is an open access journal distributed under the Creative Commons BY 4.0 license (https://creativecommons.org/licenses/by/4.0/) Chart 1 Diff erences in the aver age number of words used by the control and experimental groups number of used words 100.00 120.00 38.97 96.82 control group experimental group 80.00 60.00 40.00 20.00 0.00 The group with the narrative gamifi cation technique using an extended context generated signifi cantly longer statements than the control group. The eff ect was replicated but what is surprising is the size of the eff ect. The results of the control groups in both online and offl ine conditions are similar, whereas there is a big diff erence in the case of the performance of the gamifi ed groups. Also the size of the standard deviation in the experimental group in an online condition shows that there was no coherence in the length of the statements: some participants wrote very long fi rst impressions (over 100 words, in some cases even 200 words) and some could be considered as standard observed in an offl ine experiment (40–50 words). Such a surprising result was an inspiration to run another experiment focused on the comparison of the eff ectiveness of gamifi cation used in two conditions: online and offl ine. This time, the research was focused on the product concept testing and gathering the consumers’ fi rst impressions. 3.5. Research Methodology – Product Concept Research The experiment was conducted among 134 students of the Faculty of Management of the University of Warsaw. Two independent variables were defi ned: – research condition: online via Google Forms vs offl ine in a lecture room – gamifi cation: control group (no gamifi ed techniques were used) vs experimental (with a gamifi ed question – a narrative context added). Table 1 summarizes the number of participants in each research condition. Table 1 The number of participants in each research condition Control group Experimental (gamifi ed) group Total Online N = 43 N = 32 N = 75 Offl ine N = 28 N = 31 N = 59 total N = 71 N = 63 N = 134 The participants were asked to read the concept – a detailed description of a fi ctional product: smartphone case with a built-in charger. The concept was built in a classic way: with consumer insight, benefi t and reason to believe: Consumer insight: You are a busy person who constantly uses the phone in your work. Unfortunately, the battery cannot withstand a full day of intensive use. Power banks are a solution, but they are a tangle of cables and another item that you have to carry and remember to charge. DOI: 10.7172/2449-6634.jmcbem.2021.2.4 Journal of Marketing and Consumer Behaviour in Emerging Markets 2(13)2021 Michał Ścibor-Rylski 61 (56–64) © 2021 Authors. This is an open access journal distributed under the Creative Commons BY 4.0 license (https://creativecommons.org/licenses/by/4.0/) Benefi t: ChargeCase will appear on the market – a special case that extends the battery life of your phone. It is fully compatible with most phone models, and is also light and aesthetic. It can be personalized by selecting any graphic theme or photo. ChargeCase does not block the charging input, so you do not have to remove it when charging the phone and it charges simultaneously with it. Extends battery life by up to 100%. RtB: Modern cells used by ChargeCase allow you to store large amounts of energy in a very small, completely fl at battery, so that the phone does not increase its size and you do not have to bother with the charging process. After reading the concept, the participants were asked to fi ll out a form with only one question. Its content was diff erent in the control and in the experimental group. The control group was simply asked about their fi rst impressions. They received the following instruction: “Read the product description and write down what you think about it.” In the experimental group, a narrative context was introduced to the question. The participants received the following instruction: “Imagine you are a researcher participating in the design thinking process. You work in two teams that develop prototypes that are independent of each other. The participants of the second group have just fi nished the design session and sent you a ready idea for the product – your task is to provide them with your feedback, which will be used to optimize the concept. Read the product description and write down what you think about it.” No time limit was imposed. The next subsection presents the results of the comparison between two groups. 3.6. Results – Product Concept Research The average word count of the statements was calculated in each group. In the offl ine condition, the average number of words in the control group (M = 39.25; SD = 16.83) was slightly higher than in the experimental group (M = 34.81; SD = 17.46). In the online condition, the average number of words in the control group (M = 20.51; SD = 17.99) was lower than in the experimental group (M = 53.81; SD = 54.64). Two-way ANOVA was used to analyse the results. It revealed a statistically signifi cant interaction eff ect with a medium eff ect size: F(1, 130) = 12.30; p < 0,001; eta2 = 0.09. The results are presented in Chart 2. The arrows indicate signifi cant simple eff ects. Chart 2 The interaction eff ect number of used words 50.00 60.00 39.25 34.81 20.51 53.81 control group exp. group offline online 40.00 30.00 20.00 10.00 0.00 The analysis of the simple eff ects: DOI: 10.7172/2449-6634.jmcbem.2021.2.4 Journal of Marketing and Consumer Behaviour in Emerging Markets 2(13)2021 Michał Ścibor-Rylski 62 (56–64) © 2021 Authors. This is an open access journal distributed under the Creative Commons BY 4.0 license (https://creativecommons.org/licenses/by/4.0/) The simple eff ect of the offl ine condition is not signifi cant: F(1, 130) = 0.31; p = 0.58. There is no signifi cant diff erence in the words used by the control and the experimental group in the offl ine task. The simple eff ect of the online condition is signifi cant and strong: F(1, 130) = 21.52; p < 0.001; eta 2 = 0.14. In the online task, the experimental group used signifi cantly more words than the control group. The simple eff ect of the control group is signifi cant and rather weak: F(1, 130) = 6.30; p = 0.013; eta2 = 0.05. In the control group, the participants used signifi cantly more words in the offl ine than in the online condition. The simple eff ect of the experimental group is signifi cant and rather weak: F(1, 130) = 6.02; p = 0.016; eta2 = 0.04. In the experimental group, the participants used signifi cantly more words in the online than in the offl ine condition. Summing up – the gamifi cation of the task was eff ective only in the online condition. What is surprising is that the control group performed better in the offl ine than in the online condition. The results pattern in the experimental group was reversed – the participants were more eff ective in the online task. 3.7. Discussion of the Results The analysis of the results of both experiments partially confi rmed the hypothesis. Gamifi ed tasks make people more engaged and thus more eff ective, but there is evidence that this eff ect is moderated by the condition of the task, which answers the research question. In the case of online experiments, both in advertisement and in product concept research, the experimental group achieved higher word counts than the control group. When done offl ine, the eff ect was maintained but also impaired only for the advertisement test and there were no diff erences between the groups in the case of the product concept test. Another interesting observation is the huge diff erence between the control and experimental group in the online variant of advertisement research. Comparing both experiments, the regular, not gamifi ed groups both achieved a similar length of performance (average around 38 words), but in case of the experimental groups the diff erences are vast – online participants reacted to the gamifi ed task with the average of almost 97 words compared to over 47 in the offl ine mode. Such a fi nding needs a thorough examination and replication, but one of the possible explanations might be the fact that people in the online variant were fi lling out the form in their free time without any mental limitations. Perhaps the gamifi ed task was so involving as a role-playing experience that they allowed themselves to be carried away with the fl ow. The participants of the offl ine variant were asked to fi ll out the form during a lecture in a lecture room. Such an environment, the presence of other people and observation of their performance might limit the facilitation of the role-playing, narrative experience or even eliminate it in some types of tasks – like in the case of the evaluation of the product concept. The reason for the failure of the gamifi ed task in this kind of assignment might be the specifi city of concept testing and its verbal form. It is possible that in a more rigid offl ine condition, the participants were able to empathize with the role only when confronted with non-verbal material in a form of an appealing love story presented in the advertisement. Reading a product concept might not be such a strong trigger of narrative fl ow. This hypothesis needs further examination. The presence of other people increases vigilance and arousal which might facilitate the performance of simple, psychomotor tasks using well-trained skills (Zajonc & Sales, 1966). A complex, narrative and individual assignment requiring full concentration and the use of imagination might result in the impairment of the performance of the experimental offl ine groups DOI: 10.7172/2449-6634.jmcbem.2021.2.4 Journal of Marketing and Consumer Behaviour in Emerging Markets 2(13)2021 Michał Ścibor-Rylski 63 (56–64) © 2021 Authors. This is an open access journal distributed under the Creative Commons BY 4.0 license (https://creativecommons.org/licenses/by/4.0/) in both commercial and product concept tasks (Baron, 1986). This result might have a big impact on the use of gamifi ed techniques in focus groups – both offl ine and online. The results of the control group in the product concept test experiment shows that in the case of online tasks the performance declines compared to the offl ine condition. It is a proof that the use of gamifi ed techniques might be a solution to increase respondents’ engagement, which is signifi cantly lower when standard marketing research tasks are applied. 4. CONCLUSIONS The new results in the fi eld of the use of gamifi cation in marketing research confi rm that it is an eff ective tool to increase the participant’s engagement. Online replication of the eff ect provided a result of great importance – the COVID-19 pandemic accelerated the process of digitalization of qualitative research. Proving that gamifi cation works and is even more eff ective in online tasks opens new possibilities for market researchers. The next research step is to test gamifi ed tasks in online, qualitative consumer communities. A safe environment, no time pressure and natural activity of writing opinions on the Internet make it a perfect environment for the use of gamifi cation. Such communities usually exist several days, which does not create time pressure and allows the researcher to test diff erent, more complicated and sequential gamifi ed tasks – not only narrative but also those using challenge, feedback and competition. Another important research direction would be an experiment designed to verify the hypothesis regarding the social inhibition eff ect – is online gamifi cation eff ective only for individual tasks? How about online focus groups where the participants are aware of the presence of other people? These questions need to be answered in future research. References Bailey, P., Pritchard, G., & Kernohan, H. (2015). Gamifi cation in market research: Increasing enjoyment, participant Engagement and richness of data, but what of data validity?. International Journal of Market Research, 57(1), 17–28. https://doi.org/10.2501/IJMR-2015-003. Baldus, B.J. (2013). Engagement in online brand communities and marketing research online communities (MROCs) (A dissertation). Michigan State University. Baron, R.S. (1986). Distraction-confl ict theory: Progress and problems. Advances in Experimental Social Psychology, 19, 1–40. Castorama Polska. (2018, March 19). Potrzeby inspirują do zmian – wiosenny ogród na dachu [Video]. YouTube. Retrieved from https://www.youtube.com/watch?v=uzpPuOSfaqk. da Rocha Seixas, L., Gomes, A.S., & de Melo Filho, I.J. (2016). Eff ectiveness of gamifi cation in the engagement of students. Computers in Human Behavior, 58, 48–63. https://doi.org/10.1016/j.chb.2015.11.021. Downes-Le Guin, T., Baker, R., Mechling, J., & Ruyle, E. (2012). Myths and realities of respondent engagement in online surveys. International Journal of Market Research, 54(5), 613–633. Gorączka, A., & Protasiuk, M. (2020). Gamifi cation. Jak wygrać zaangażowanie respondenta?. Warszawa: PWN. Harrison, P. (2011). The researchifi cation of games: Adopting a game designer’s approach to market research. ESOMAR Congress Miami. Homer, R., Hew, K.F., & Tan, C.Y. (2018). Comparing digital badges-and-points with classroom token systems: Eff ects on elementary school ESL students’ classroom behavior and English learning. Educational Technology and Society, 21(1), 137–151. Nieto-Escamez, F.A., & Roldán-Tapia, M.D. (2021). Gamifi cation as online teaching strategy during COVID-19: A mini review. Frontiers in Psychology, 12, 648552. https://doi.org/10.3389/fpsyg.2021.648552.