Capture the Flag as a Learning Tool to Improve Cybersecurity Education

Research Paper Recommended citation: Schafeitel-Tähtinen, T., & Lazarov, W. (2025). Capture the Flag as a Learning Tool to Improve Cybersecurity Education. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631277. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. CAPTURE THE FLAG AS A LEARNING TOOL TO IMPROVE CYBERSECURITY EDUCATION T. Schafeitel-Tähtinen a, 1 , W. Lazarov b a Tampere University, Finland, 0000-0002-1286-8427 b Brno University of Technology, Czechia, 0000-0001-6820-8391 Conference Key Areas: 15. Improving higher engineering education through researching engineering education, 4. Digital tools and AI in engineering education Keywords: Capture the Flag, cyber range, cybersecurity learning ABSTRACT Recent research has shown promising results in using gamified Capture the Flag (CTF) scenarios for cybersecurity education. This paper examines the effectiveness of a gamified CTF scenario among 83 university students. We analysed differences in effectiveness based on students' gender and field of study, including information security, computer science, engineering, and other study programs. The results show that participation in the CTF increased the knowledge related to the CTF content in all groups. Skills and self-efficacy also increased, except among information security students, for whom the difficulty level of the CTF was too low. The students perceived the CTF meaningful for learning and were satisfied with the experience. However, the findings suggest that gamified teaching may appeal more to men than women. Based on the study findings, we find that CTF is a suitable approach to offering practical cybersecurity exercises to students. With the appropriate difficulty level, it can be especially important for students in computer science programs to help them assess their cybersecurity skills more accurately. However, to increase the effectiveness of cybersecurity education, it is important to appropriately set the difficulty of CTF in relation to learning objectives and students' skill levels. 1 Corresponding Author T. Schafeitel-Tähtinen [email protected] 1 INTRODUCTION There is a global shortage of cybersecurity experts as societies and services digitalize (ISC2, 2024). Cybersecurity education should provide graduates with practical and theoretical knowledge, along with confidence and skills to apply knowledge in ever evolving security threat landscape. Important part of effective cybersecurity teaching are practical exercises and learning-by-doing as they enhance problem-solving skills. These skills are especially critical in real-world security incidents, where defenders must counter attacks and mitigate damage. Recent research in security education has shown promising results in utilizing gamified teaching approaches, such as Capture the Flag (CTF) scenarios (Beltran et al., 2018; Chothia & Novakovic, 2015; Cole, 2022; Karagiannis & Magkos, 2021; Leune & Petrilli, 2017; Vykopal et al., 2020; Wolfenden, 2019). CTFs are hands-on exercises, where students look for “flags” to proceed with the exercise. Flags are ”captured” by successfully completing a cybersecurity task, such as scanning a vulnerable system or decrypting a message. CTF exercises can be further gamified by incorporating a scoring system, rewards, or an engaging storyline. In addition, CTFs often provide students with a space to practice their skills in a sandbox environment that represents real-life scenarios. According to previous research, CTF exercises can increase students’ motivation (Beltran et al., 2018; Cole, 2022), confidence (Karagiannis & Magkos, 2021; Leune & Petrilli, 2017), as well as improve their knowledge and skills (Chothia & Novakovic, 2015; Karagiannis & Magkos, 2021). Another important finding is the increase in engagement and enjoyment in learning (Chothia et al., 2019; Karagiannis & Magkos, 2021; Vykopal et al., 2020). However, some studies also indicate that achieving a deep understanding of the topics requires additional learning approaches alongside CTFs (Chothia & Novakovic, 2015; Leune & Petrilli, 2017). Less is known regarding whether CTFs are effective in improving cybersecurity learning regardless of students’ field of study or gender. Another interesting question is how CTF scenario elements, such as instructions, tasks, and gamification elements, impact students’ satisfaction or perceived meaningfulness for learning, and whether there exist study field or gender related differences. In this study, we examine the effectiveness of the gamified CTF scenario among university students. We analyse differences in effectiveness according to the students' gender and their field of study, such as information security, computer sciences, engineering, and other study programs. Additionally, we assess student satisfaction and the perceived meaningfulness of learning for different CTF scenario elements, such as instructions, tasks, and gamification elements. 2 METHODOLOGY 2.1 Scenario Design For our study, we designed a gamified scenario titled "(Un)usual Monday Morning", which is focused on the fundamentals of cybersecurity and covers diverse topics such as encryption, password security, and encoding techniques. To develop this scenario and prepare an interactive learning environment, we utilized the Brno University of Technology Cyber Arena (BUTCA) platform (Lazarov et al., 2023). Cyber Arena enables the creation of CTF scenarios and the deployment of a sandbox environment, allowing remote hands-on cybersecurity training. The CTF scenario was divided into seven practical tasks, and its structure was as follows: pre-survey, prologue (P), seven tasks (T1, T2, …, T7), epilogue (E), final test (FT), and two post-surveys (see Fig. 1). Fig. 1. Phases of the presented study. The gamified element is the background story about a student who wakes up with short-term memory loss and must determine what happened the previous night, where he is, and how to get home. Throughout the story, the student completes several practical challenges focused on cybersecurity topics, which are listed in Table 1. Table 1. Tasks of the CTF scenario. Task Difficulty Challenge T1 Easy Substitution cipher decryption T2 Easy Security settings (phone PIN) T3 Medium Extracting and cracking SHA-256 hash T4 Medium Weaknesses of LM hash in Windows OS T5 Easy Identification and encoding of Base64 string T6 Beginner Geolocation in the interactive map system T7 Easy Extracting a hidden message from a file 2.2 Measurements We measured CTF effectiveness using preand post-surveys. Pre-survey was completed before participating in the CTF and the post-surveys afterwards. Measurements are based on the model from Schafeitel-Tähtinen et al. (2024), where theoretical background and description of the variables can be found. We assessed teaching, knowledge, self-efficacy, attitude, behaviour, and interest-related variables. Whenever possible, survey scales were adapted from previous research, while some were constructed based on the tasks and content of the CTF. Career, research, and further study interest were measured using a self-developed scale. Teaching-related variables and changes in general interest in learning cybersecurity were measured only in the post-survey. The first post-survey was conducted immediately after the CTF. Behaviour-related variables were assessed in the second post-survey two weeks later. But participation in the second post-survey was lower than in the first one. All measurements were self-reported and measured students’ own experiences. For analysis, we organized students into the following groups: all students (N=83), female students (N=29), male students (N=54), InfoSec students (N=13), ICT students (N=45), Engineering students (N=12), and Mixed students (N=13). InfoSec students are students whose primary focus of studies is information security. ICT students are enrolled in various computing science programs, while Engineering students are in different engineering programs. The Mixed students group includes those from fields such as medicine, humanities, social sciences, and education. Furthermore, we examined the reliability of the scales with Guttmann’s lambda 2. The lambdas were above the threshold of 0.7 for all scales except for last week’s behaviour (pre: 0.553, post: 0.519). To investigate differences between preand post-surveys, we first compared variables in preand post-surveys among the same students. Since the data were not normally distributed, we used the nonparametric Wilcoxon signedrank test. To compare the differences between male and female students, we used the nonparametric Mann-Whitney U test. We examined pre-survey value differences, post-survey value differences, and gain, which was calculated by subtracting the presurvey value from the post-survey value. To compare differences between major groups, we used the Kruskal-Wallis test, and to analyse sample pairs for dominance, we used the pairwise Mann-Whitney test with Bonferroni correction. All statistical analyses were conducted using SPSS 29.0.1.0 (171). 2.3 Participants A total of 83 students, 29% of whom were women, participated in the study by playing the CTF scenario. Details of the study participants are listed in Table 2. Table 2. Overview of research participants. All InfoSec ICT Eng. Other Total 83 13 45 12 13 Men 59 11 34 7 7 Women 24 2 11 5 6 Age under 20 1 1 20-29 60 10 36 9 5 30-39 15 2 4 2 7 40-49 6 1 4 0 1 50-59 1 0 0 1 0 Prev. education Primary school (9 years) 1 0 1 0 0 Upper secondary school 45 5 27 8 5 Vocational education 2 0 2 0 0 Bachelor's degree 23 4 13 2 4 Master's degree 11 4 1 2 4 PhD 1 0 1 0 0 3 RESULTS 3.1 Effectiveness In terms of effectiveness, students reported statistically significant higher postthan pre-values in CTF content knowledge, CTF task skills, CTF task self-efficacy, CTF content interest, further study interest, and attitude (see Table 3). Post values were also higher for all other variables except general self-efficacy, which remained stable. However, these differences were non-significant. Furthermore, female students reported significantly higher postthan pre-values in CTF content knowledge, CTF task skills, CTF task self-efficacy, and CTF content interest. Male students reported significantly higher postthan pre-values in CTF content knowledge, CTF task skills, CTF task self-efficacy, CTF content interest, further study interest, and attitude. Table 3. Comparison of post and pre values (statistically significant differences highlighted). All students N=83 Men N=59 Women N=24 z p eff. size z p eff. size z p eff. size CTF content knowledge -7.44 <.001 0.82 -6.33 <.001 0.82 -4.05 <.001 0.83 CTF task skills -6.54 <.001 0.72 -5.34 <.001 0.70 -3.79 <.001 0.77 General selfefficacy -0.23 0.819 0.03 -0.50 0.619 0.06 -0.57 0.567 0.12 Cybersecurity self-efficacy -1.46 0.145 0.16 -1.05 0.295 0.14 -0.98 0.325 0.20 CTF task selfefficacy -4.92 <.001 0.54 -3.61 <.001 0.47 -3.38 <.001 0.69 CTF content interest -3.31 <.001 0.36 -2.33 0.020 0.30 -2.51 0.012 0.51 Further study interest -2.75 0.006 0.30 -2.73 0.006 0.35 -0.81 0.416 0.17 Career interest -1.20 0.229 0.13 -0.94 0.346 0.12 -0.74 0.458 0.15 Research interest -1.64 0.101 0.18 -1.61 0.107 0.21 -0.54 0.589 0.11 Attitude -2.42 0.015 0.27 -2.22 0.027 0.29 -0.99 0.323 0.20 Post-survey 2 N=50 N=36 N=14 Security behaviour intentions -0.92 0.36 0.13 -0.15 0.88 0.02 -1.72 0.09 0.46 Last week's behaviour -0.20 0.84 0.03 -0.65 0.52 0.11 -0.97 0.33 0.26 The InfoSec students reported significantly higher postthan pre-values in CTF content knowledge (see Tables 4 and 5). ICT students reported significantly higher postthan pre-values in CTF content knowledge, CTF task skills, CTF task selfefficacy, CTF content interest, and further study interest. They also reported the only statistically significant lower postthan pre-value in cybersecurity self-efficacy. Engineering and Mixed students reported significantly higher postthan pre-values in CTF content knowledge, CTF task skills, and CTF task self-efficacy. 3.2 Differences between groups In the pre-survey, male students had statistically significantly higher values in security behaviour intentions and last week's behaviour than female students. In the postsurvey, male students had significantly higher values in attitude than female students. Regarding gain, no significant differences were found between male and female students. Table 4. Comparison of post and pre values (statistically significant increases are highlighted with green and decreases with orange colour). InfoSec N=13 ICT N=45 z p eff. size z p eff. size CTF content knowledge -2.53 0.011 0.70 -5.66 <.001 0.84 CTF task skills -1.61 0.107 0.45 -5.24 <.001 0.78 General self-efficacy -0.75 0.453 0.21 -0.82 0.414 0.12 Cybersecurity self-efficacy -0.14 0.885 0.04 -2.20 0.028 0.33 CTF task self-efficacy -0.51 0.610 0.14 -3.70 <.001 0.55 CTF content interest -0.59 0.557 0.16 -3.78 <.001 0.56 Further study interest -1.54 0.123 0.43 -3.16 0.002 0.47 Career interest -0.81 0.420 0.22 -0.25 0.799 0.04 Research interest -0.63 0.527 0.18 -1.44 0.151 0.21 Attitude -0.98 0.327 0.27 -1.84 0.067 0.27 Post-survey 2 N=9 N=26 Security behaviour intentions -1.34 0.180 0.37 -1.84 0.434 0.27 Last week's behaviour -0.48 0.655 0.13 -0.44 0.660 0.07 Table 5. Comparison of post and pre values. Engineering N=12 Mixed N=13 z p eff. size z p eff. size CTF content knowledge -2.89 0.004 0.83 -3.36 <.001 0.93 CTF task skills -2.52 0.012 0.73 -2.59 0.010 0.72 General self-efficacy -0.94 0.348 0.27 -1.72 0.086 0.48 Cybersecurity self-efficacy -1.51 0.131 0.44 -0.28 0.783 0.08 CTF task self-efficacy -2.46 0.014 0.71 -2.81 0.005 0.78 CTF content interest -0.43 0.666 0.12 -0.54 0.589 0.15 Further study interest -0.41 0.680 0.12 -0.27 0.785 0.08 Career interest -1.89 0.059 0.55 0.00 1.000 0.00 Research interest -0.28 0.783 0.08 -1.00 0.317 0.28 Attitude -0.36 0.719 0.10 -1.38 0.167 0.38 Post-survey 2 N=7 N=8 Security behavior intentions -1.13 0.257 0.33 -0.76 0.450 0.21 Last week's behavior -1.34 0.180 0.37 -1.00 0.317 0.28 Table 6 shows the differences in statistically significant values in preand post-survey between groups. For example, in the pre-survey, the InfoSec group had statistically significantly higher values in CTF content knowledge than any other group, but in postsurvey, the knowledge difference between InfoSec group and ICT group has flattened. Similar flattening can be seen in CTF task self-efficacy between ICT and Mixed groups, as in the post-survey, the difference is not anymore significant. However, some differences have also steepened, for example, InfoSec and ITC groups have significantly higher cybersecurity self-efficacy values compared to Engineering and Mixed groups in the post-survey. Additionally, the InfoSec group reported significantly higher values in their general interest in learning cybersecurity compared to the ICT and Mixed groups. However, in terms of gain, there were no statistically significant differences between groups. Table 6. Statistically significant differences in preand post-survey between groups. Group Variable Statistically significantly higher value compared to ... Pre-survey Post-survey InfoSec CTF content knowledge ICT Eng. Mixed Eng. Mixed ″ CTF task skills Eng. Mixed Mixed ″ Cybersecurity self-efficacy Eng. Mixed ″ CTF task self-efficacy Mixed Eng. Mixed ″ Further study interest Mixed Eng. Mixed ″ CTF content interest Eng. Mixed ″ Career interest ICT Eng. Mixed ICT Eng. Mixed ″ Research interest ICT Eng. Mixed ICT Mixed ″ Attitude Eng. Eng. Mixed ICT CTF task skills Mixed ″ CTF task self-efficacy Mixed ″ Cybersecurity self-efficacy Eng. Mixed 3.3 Satisfaction, perceived meaningfulness, and general interest in learning In the post-survey, students rated the CTF tasks and various other CTF features in terms of satisfaction, perceived meaningfulness for learning, and how the tasks and features affected their general interest in learning cybersecurity: • Satisfaction with CTF scenario: 46 students (55%) were very or extremely satisfied, out of them 59% male and 46% female students; 34 students (41%) were satisfied. • Meaningfulness for learning: 40 students (48%) perceived the CTF as very or extremely meaningful, out of them 51% male and 42% female students; 40 students (48%) perceived the CTF as meaningful. • Interest in learning cybersecurity: 58 students (71%) reported increased interest, out of them 75% male and 62% female students; remaining 29% of students reported no change. 3.4 Tasks, instructions and gamification elements Students were significantly more satisfied with CTF tasks than with instructions. CTF tasks were perceived as significantly more meaningful for learning than gamification elements, which were also perceived less meaningful than instructions. Furthermore, CTF tasks were regarded as significantly more important for general interest in learning cybersecurity compared to both instructions and gamification elements, and instructions were considered significantly more important than gamification elements. There was a significant difference between male and female students in gamification elements' perceived meaningfulness for learning and its importance for general interest in cybersecurity learning. In both cases, male students considered gamification elements significantly higher than female students. Additionally, the InfoSec and ICT groups were more satisfied with instructions than the Mixed group. Lastly, InfoSec students perceived instructions significantly more meaningful for learning than the ICT and Mixed groups. 4 DISCUSSION AND CONCLUSIONS CTF participation increased knowledge across all groups with large effect sizes, indicating improvement. For all groups except InfoSec students, task skills also increased with large effect sizes. Additionally, CTF participation improved task skill self-efficacy in all groups except InfoSec students. The highest effect sizes were observed in the Mixed, Engineering, and female student groups, while the lowest in the male student and ICT groups. Increase in knowledge and skills has been observed, e.g., by Chothia & Novakovic (2015) and Karagiannis & Magkos (2021), however, this study found variation based on the students’ major. The likely reason for the lack of improvement in skills and task skill self-efficacy among InfoSec students was the beginner level CTF. These students probably already had considerable practice with task-related skills, leaving little room for further development. Therefore, teachers should consider participants’ skill levels when selecting the CTF difficulty to ensure that even skilled students can improve. While in InfoSec group knowledge increased, there were no significant changes in any other variables. In contrast, the CTF had more benefits for ICT students. Their knowledge, skills, task-related self-efficacy, content interest, and even interest in further study significantly increased. Findings among ICT students align with the earlier CTF research, e.g., (Cole, 2022; Karagiannis & Magkos, 2021). However, ICT students’ cybersecurity-specific self-efficacy significantly decreased, although the effect size was small. This may suggest that the CTF scenario helped to correct an overestimation of security skills, allowing ICT students to assess their confidence in these skills more appropriately. If this interpretation is correct, and the result can be achieved with such a simple arrangement, it directs to recommending security-related practical exercises for ICT students. It is crucial, for example, from secure software production perspective that engineers are not overconfident in their security skills. However, regarding CTFs’ behaviour related variables, no effectiveness was observed in this study. The only statistically significant increase in attitude was found among male students; however, the effect size was small. Since the CTF tasks were not designed to promote behavioural changes, this outcome was expected. However, for CTFs aiming for increasing e.g., cybersecurity awareness, also behavioural impacts should be examined when assessing the effectiveness of the CTF. Regarding CTF properties, students were significantly more satisfied with the CTF tasks than with the instructions. Additionally, CTF tasks were perceived significantly more meaningful for learning than gamification elements. Male students perceived gamification elements more meaningful for learning and more important for increasing interest in learning cybersecurity than female students. Realizing how important it is that the tasks are perceived meaningful by students may help teachers to design the CTFs in a motivating and instructive way. Guaranteeing meaningfulness for everyone might not be easy but when successful it may also mitigate the challenge that gamified learning seems to be more appealing to men than to women. The participants were satisfied with the CTF and perceived it meaningful for learning. General interest in learning cybersecurity increased, without any significant differences in gains between groups. Thus, CTF is a suitable approach for providing practical cybersecurity exercises to students. However, it is important to set the CTF difficulty level appropriately in relation to the learning objectives to ensure its effectiveness.