Curricula Designer with Enhanced ECSF Analysis Jan Hajny [email protected] Brno University of Technology Brno, Czech Republic Marek Sikora
[email protected] Brno University of Technology Brno, Czech Republic Konstantinos Adamos [email protected] University of the Aegean Samos, Greece Fabio Di Franco [email protected]opa.eu ENISA Athens, Greece ABSTRACT In late 2022, the novel European Cybersecurity Skills Framework (ECSF) was officially released by the European Union Agency for Cybersecurity (ENISA). It aims to connect cybersecurity education and training with practical needs of the job market. In particular, it maps role profiles, that reflect jobs, to the knowledge and skills they require. One of the first tools that demonstrated ECSF is the Curricula Designer web application that guides cybersecurity study program administrators in designing and analyzing their curricula. In this paper, we present a major update of the Curricula Designer tool. We develop a novel method for course scoring and quantitative analysis of curricula based on the European Credit Transfer and Accumulation System (ECTS) credits. We describe the underlying methods and show their practical implementation into the publiclyavailable web application. Furthermore, we update the definitions of the Skills, Knowledge and Role Profiles according to the latest ECSF definition and present the mappings in comprehensive matrices in the appendices. CCS CONCEPTS •Social and professional topics → Computing education programs;Employment issues;•Applied computing → Education;•Software and its engineering →Designing software. KEYWORDS Education, training, tools, cybersecurity, profiles, skills, knowledge ACM Reference Format: Jan Hajny, Marek Sikora, Konstantinos Adamos, and Fabio Di Franco. 2023. Curricula Designer with Enhanced ECSF Analysis. In The 18th International Conference on Availability, Reliability and Security (ARES 2023), August 29– September 01, 2023, Benevento, Italy. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3600160.3604987 1 INTRODUCTION Cybersecurity is a topic that is discussed intensively in all relevant domains, including academia, business and public sectors. The need This work is licensed under a Creative Commons Attribution International 4.0 License. ARES 2023, August 29–September 01, 2023, Benevento, Italy ©2023 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-0772-8/23/08. https://doi.org/10.1145/3600160.3604987 for more cybersecurity experts is evident in various studies focused on the current job market and education [ 4 , 5 ]. This demand is reflected by the increasing number of higher-education and professional study programs that are focused directly on this field. Specialized study programs are provided by all types of education providers, particularly by universities and professional training providers. One of the leading databases that certify this increasing number of programs is, among others, the ENISA CyberHead database [ 6 ] that covers over 130 programmes from 26 European countries. Despite this dynamic activity in cybersecurity education and training, there is still no universal guidance and very few goodpractice examples on how to build a good cybersecurity-focused program. In this paper, we show tools that were designed to help education providers to build cybersecurity study programs that reflect the needs of the job market and that provide education and training in topics that are highly relevant to the cybersecurity field. In particular, we show the latest version of the ENISA’s European Cybresecurity Skills Framework [ 7 ] and present a software tool that makes use of ECSF’s features for study program analysis and design. We present novel advanced features for curricula evaluation according to requirements of ECSF and describe analytical tools that may be used for rigorous evaluation of study programs’ content with respect to the requirements of job profiles. We also present features that may be used to directly compare study programs to each other using ETCS credit-based scoring. The intended audience of this paper are study program administrators, course supervisors and education provider managers that will learn about basic principles of program design and will get access to tools for the evaluation of their programs so that they can reflect current needs and are compatible with other programs provided in different countries across EU. 1.1 State of the Art In this paper, we work with two tools that were released relatively recently: The European Cybersecurity Skills Framework (ECSF) released in Sept. 2022 [ 7 ] and the Curricula Designer tool released in 2021 [13]. There are already numerous frameworks and recommendations for cybersecurity skill and knowledge development. Besides ECSF, the NIST NICE [ 10 ], CyBOK [ 11 ], REWIRE European Cybersecurity Blueprint [ 12 ] or SecNumedu [ 2 ] belong to the most well-known. There also exist national frameworks, such as the CyQUAL [ 14 ], that reflect specifics of particular countries. However, the ENISA
ARES 2023, August 29–September 01, 2023, Benevento, Italy Hajny et al. Figure 1: User Interface of Curricula Designer ECSF has the unique ambition to be the first universal framework that is usable across the Europe, not only on national levels, with reflections of the EU cybersecurity job market. Although other frameworks, such as the NIST NICE, are also very useful for curricula design, they may not fully reflect the EU’s landscape, in particular the legal and market aspects. Therefore, the ENISA ECSF has been selected as the core engine to evaluate study programs in our tools. Currently, there are very few tools for cybersecurity study program design. In the US, some program administrators use CyberSeek [ 1 ] to explore the supply and demand on the job market. However, direct guidance on the study program design and the analytical tools that may help to evaluate the resulting curricula are not available. Some education providers, usually universities, provide tools for students to help them design their own personalized curricula, but these tools have a different purpose than we seek here. We aim predominantly at educators that design completely new study programs, rather than at students who only compose their curricula from available courses. Currently, we are not aware of any software tool that is directly targeting the curricula design and analysis as does the Curricula Designer [ 13 ] by the SPARTA project. 1.2 Our Contribution We present major updates to the methodology of course evaluation and to the Curricula Designer software tool in this paper. First, we show a method for a precise evaluation of the content of study courses and introduce a scoring system based on the European Credit Transfer and Accumulation System (ECTS) [ 3 ]. We also demonstrate how to use the individual course scores to evaluate the whole cybersecurity curriculum and how to compare it to others with respect to compliance with ECSF Role Profiles’ requirements. We implement our evaluation methodology in the Curricula Designer web application and show how to use it in practice. Furthermore, we update the previous Curricula Designer with the latest ECSF Skills and Knowledge database so that the latest version is reflected in the tool. By introducing these updates, we make a quantitative analysis of cybersecurity study programs possible. Compared to only binary evaluations (supported/unsupported ECSF Role) used in the previous version of the Curricula Designer, these updates make a big difference in the precision of the analytical component of the tool. In particular, we can evaluate not only if some ECSF Role is addressed, but how much it is addressed by a given curriculum. 2 EUROPEAN CYBERSECURITY SKILLS FRAMEWORK 22.9 The European Cybersecurity Skills Framework (ECSF) was publicly released by ENISA on Sept. 19th 2022. It is described in two main documents, the definition of twelve ECSF Role Profiles [ 9 ] and the User Manual [ 8 ]. According to ENISA, the main goals of ECSF are: • Introduce common terminology and understanding between employers and education providers across EU. • Identify critical skills required from the workforce perspective. • Explain the leading cybersecurity professional roles and the skills and knowledge they require. • Contribute to the harmonisation in cybersecurity education and training. • Generally increase the protection against cyber attacks and help to provide security of IT systems. The ECSF framework specifically defines the following Role Profiles: •Chief Information Security Officer (CISO) •Cyber Incident Responder •Cyber Legal, Policy & Compliance Officer •Cyber Threat Intelligence Specialist
Curricula Designer with Enhanced ECSF Analysis ARES 2023, August 29–September 01, 2023, Benevento, Italy Course A 6 ECTS Credits P1: 1.5 ECTS P7: 1.5 ECTS P10: 2.1 ECTS Study Load to Get Knowledge Study Load to Get Skills 0 % 100 % 6 ECTS 50 %30 % 80 % 20 % 50 % 90 % ECSF P1 ECSF P7 ECSF P10 Other ECSF P1 ECSF P7 ECSF P10 Other Course A 6 ECTS Credits P1: 1.5 ECTS P7: 1.5 ECTS P10: 2.1 ECTS Course B 5 ECTS Credits P1: 1.5 ECTS P7: 1.2 ECTS P12: 1.1 ECTS Course n 4 ECTS Credits P1: 1 ECTS P7: 2.5 ECTS Profile 1: 40 ECTS Profile 2: 20 ECTS Other: 77 ECTS Profile 7: 31 ECTS Profile 12: 12 ECTS Figure 2: Course Scoring Principle •Cybersecurity Architect •Cybersecurity Auditor •Cybersecurity Educator •Cybersecurity Implementer •Cybersecurity Researcher •Cybersecurity Risk Manager •Digital Forensics Investigator •Penetration Tester For each Role Profile, required Skills and Knowledge are listed. This mapping between Role Profiles and required Skills and Knowledge is specified in the matrices in Appendix A, resp. Appendix B. The mapping is the basis for our analytical features implemented in the Curricula Designer. 3 CURRICULA DESIGNER The ECSF is a theoretical framework that must be transferred to practice through real applications. The Curricula Designer was one of the first pilot software applications where ECSF was deployed to help study program designers to create and analyze cybersecurity programs. The Curricula Designer web application [ 13 ] was first released in 2021 and since then it has been regularly updated and maintained. This paper describes the major updates of the tool applied in 2023. The current Curricula Designer user interface is depicted in Figure 1. The application is divided into three sections: Available Courses, Your Curricula and Statistics. In the Available Courses, the user may either select pre-configured sample courses or define new courses. Each course is represented by a box that can be dragged to the middle section representing semesters of a study program. The right section contains statistical data about the curriculum and is updated after any addition or removal of a course. The statistics are computed according to either NIST NICE or ECSF framework. In both versions, the content of a curriculum is analyzed and required knowledge and skills are being checked. NIST NICE Work Roles and ECSF Role Profiles that correspond to learning outcomes of the curriculum are displayed, together with information about concrete knowledge and skills that are supported. However, in the previous versions, no quantitative analysis was provided, in particular about the amount of workload necessary to gain required knowledge and skills. That imperfection allowed to incorrectly mark some study curricula as compliant with requirements of some Role Profiles even though the required Knowledge and/or Skills are covered only negligibly. This weakness is fixed by the course and curriculum content scoring introduced in the next sections. 4 METHODOLOGY FOR COURSE SCORING In the course scoring, we need to estimate the amount of workload necessary to gain particular knowledge and skills. We work on a course level, as courses are the main building blocks for study programs. In most European countries, the expected workload for a single course is usually evaluated by ECTS credits. One ECTS credit represents 25 - 30 hours of students’ work. Therefore, a course evaluated by 7 ECTS credits should require around 196 hours to complete, including in-person and remote work. The average higher-education courses require between 3 and 8 ECTS credits, but exceptions often exist. Using ECTS, we can compute the total working time necessary for each course. If we are able to estimate, what portion of course workload is relevant to specific ECSF Knowledge and Skills, we can compute the estimate of time necessary to gain specific ECSF
ARES 2023, August 29–September 01, 2023, Benevento, Italy Hajny et al. Figure 3: Course Add Menu Knowledge and Skills. Furthermore, since the ECSF framework maps Knowledge and Skills to Role Profiles, we can also compute the total time devoted to training towards specific Role Profiles. This information may be then used, e.g., to evaluate if the study program covers particular Role Profiles only superficially or significantly. We can also sort the targeted Role Profiles according to time devoted to them in the study program. The method for course scoring is depicted in Fig. 2. First, each course is evaluated by the ECTS credits according to the expected amount of student’s time necessary to pass it. Then, a percentage is assigned to each ECSF Skill and/or Knowledge the course covers. The rest content is marked as "Other". By this evaluation, we get the relative number of ECTS credits allocated to specific ECSF Knowledge and Skills (see Fig. 2, details provided for example Course A). We sum these credits across all courses in the curriculum. Now, as we have the relative ECTS for each Knowledge and Skill, we may compute the relative ECTS credits devoted to ECSF Role Profiles by just adding all ECTS credits of those Skills/Knowledge that are required by selected Role Profiles. By this procedure, we get the evaluation of a whole curriculum with respect to the ECSF. 5 IMPLEMENTATION INTO CURRICULA DESIGNER The practical implementation of the scoring mechanisms described above into the Curricula designer web application was rather straightforward. We had to modify the course definition components, program the functions for computing course scores and curriculum scores and update the statistical section. The new "Add course" menu is depicted in Fig 3. Here, a user may select any ECSF Knowledge and/or Skills and assign percentage of course workload that is assigned to it. Any workload not relevant to ECSF can be marked as "Non-ECSF Skill/Knowledge". We also implemented safeguarding functions to be sure that total workload sums to 100 %. Figure 4: Statistics Section In the background, the Curricula Designer maintains the information about ECTS assigned to Skills and Knowledge defined in the ECSF. The information about supported Role Profiles is provided in the "Statistics" section. Here, the supported ECSF Role Profiles are listed in the ECSF Framework submenu, see Fig. 4. For any ECSF Role Profile that is marked as supported, there must be all required Knowledge and Skills present in some of the courses in curricula. Each Role Profile is also evaluated by relative ECTS credits devoted to teaching Knowledge/Skills defined for that particular profile. The higher this number is, the more the required Knowledge/Skills are represented in the curriculum. The list of supported ECSF Role Profiles is sorted according to the ECTS scores. By hovering the mouse over a particular ECSF Role Profile, a list of required Knowledge and Skills is displayed. The application is now available at https://www.sparta.eu/curricula-designer/. 5.1 Future Work Currently, the Curricula Designer application shows only total relative effort (represented by ECTS credits) assigned to gaining all Skills and Knowledge required by particular Role Profiles. Although necessary Knowledge and Skills are identified for each Role Profile, the exact effort necessary for obtaining them is hidden in the application. In the next updates of the tool, we may provide also these deeper statistics on workload linked to individual Knowledge and Skills taught in the study program. The only obstacle is finding a clean and simple method to present this data. The Curricula Designer tool was updated with the latest definition of the ENISA ECSF, as specified in the Appendices. At the moment, ENISA does not provide a definition of Role Profiles, Knowledge and Skills mapping in a machine-readable open format. Therefore, any update of the ECSF must be manually introduced to the web application, which causes delays and man effort. Our future work aims at the ability to import machine-readable definitions of the ENISA ECSF in a common format, such as xml. This functionality would make the tool easier to update and maintain.
Curricula Designer with Enhanced ECSF Analysis ARES 2023, August 29–September 01, 2023, Benevento, Italy 6 CONCLUSION We described the updated version of the Curricula designer web application. As a major contribution, we introduced a scoring mechanism and quantitative evaluation of education/training content. Using simple scoring features, the user may learn not only what ECSF Role Profiles are supported by the designed curriculum, but also to what extend they are part of the curricula. This rigorous scoring mechanism gives users the ability to better analyze the focus of study programs based on numerical evaluation. Furthermore, we also implemented the latest ENISA ECSF version from Sept. 2022 and provided comprehensive matrices of Skills/Knowledge mapping to Role Profiles in Appendices. ACKNOWLEDGMENTS The work was supported by the Ministry of the Interior of the Czech Republic under grant VJ03030003 under program IMPAKT 1 and the European Union’s Horizon Europe project #101087529 CHESS. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. REFERENCES [1] 2022. CyberSeek. https://www.cyberseek.org. [2] ANSSI. 2021. SecNumedu, labeling of higher education courses in cybersecurity. https://www.ssi.gouv.fr/en/cybersecurity-in-france/formations/secnumedulabeling-of-higher-education-courses-in-cybersecurity/. [3] European Commission, Sport Directorate-General for Education, Youth, and Culture. 2017. ECTS users’ guide 2015. Publications Office. https://doi.org/doi/10. 2766/87192 [4] ECSO. 2018. Gaps in European Cyber Education and Professional Training. https://www.ecs-org.eu/documents/publications/5bf7e01bf3ed0.pdf. [5] ENISA. 2020. Cybersecurity Skills Development in the EU. https: //www.enisa.europa.eu/publications/the-status-of-cyber-security-educationin-the-european-union. [6] ENISA. 2022. CYBERHEAD - Cybersecurity Higher Education Database. https: //www.enisa.europa.eu/topics/cybersecurity-education/education-map. [7] ENISA. 2022. European Cybersecurity Skills Framework. https://www.enisa. europa.eu/topics/cybersecurity-education/european-cybersecurity-skillsframework. [8] ENISA. 2022. European Cybersecurity Skills Framework (ECSF) - User Manual. https://www.enisa.europa.eu/publications/european-cybersecurity-skillsframework-ecsf. [9] ENISA. 2022. European Cybersecurity Skills Framework Role Profiles. https://www.enisa.europa.eu/publications/european-cybersecurity-skillsframework-role-profiles. [10] NIST. 2020. NIST Special Publication 800-181 Revision 1: Workforce Framework for Cybersecurity (NICE Framework). https://nvlpubs.nist.gov/nistpubs/ SpecialPublications/NIST.SP.800-181r1.pdf. [11] Awais Rashid, Howard Chivers, George Danezis, Emil Lupu Imperial, and Andrew Martin. 2019. The Cyber Security Body Of Knowledge. https://www.cybok.org/ media/downloads/cybok_version_1.0.pdf. [12] REWIRE. 2022. European Cybersecurity Blueprint. https://rewireproject.eu/wpcontent/uploads/2022/11/REWIRE_R3.2.1_European-cybersecurityblueprint_Final_ForRelease.pdf. [13] SPARTA. 2022. Cybersecurity Curricula Designer. https://www.sparta.eu/ curricula-designer/. [14] Masaryk University. 2023. CyQUAL. https://www.cyqual.cz/?lang=en.
ARES 2023, August 29–September 01, 2023, Benevento, Italy Hajny et al. APPENDIX A Table 1: Mapping of Knowledge to Profiles (P) [9] Knowledge P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 Advanced and persistent cyber threats (APT) x Auditing standards, methodologies and frameworks x Auditing-related certification x Computer networks security x x x x x Computer programming x x x Computer Security Incident Response Teams (CSIRTs) operation x Computer systems vulnerabilities x x x x Conformity assessment standards, methodologies and frameworks x Criminal investigation procedures, standards, methodologies and frameworks x Cross-domain and border-domain knowledge related to cybersecurity x Cyber threat actors x Cyber Threat Intelligence (CTI) sharing standards, methodologies and frameworks x Cyber threats x x x x x Cybersecurity attack procedures x x x x Cybersecurity awareness, education and training programme development x Cybersecurity controls and solutions x x x x x x Cybersecurity education and training standards, methodologies and frameworks x Cybersecurity maturity models x Cybersecurity policies x x Cybersecurity procedures x Cybersecurity recommendations and best practices x x x x x Cybersecurity related laws, regulations and legislations x x x x x Cybersecurity risks x x Cybersecurity standards, methodologies and frameworks x x x x x x Cybersecurity trends x Cybersecurity-related certifications x x x x x x x x x Cybersecurity-related requirements analysis x Cybersecurity-related research, development and innovation (RDI) x Cybersecurity-related technologies x x x Digital forensics analysis procedures x Digital forensics recommendations and best practices x Digital forensics standards, methodologies and frameworks x Ethical cybersecurity organisation requirements x Incident handling communication procedures x Incident handling recommendations and best practices x Incident handling standards, methodologies and frameworks x Incident handling tools x Information technology (IT) and operational technology (OT) appliances x Legacy cybersecurity procedures x Legal, regulatory and legislative compliance requirements, recommendations and best practices x x x Legal, regulatory and legislative requirements on releasing or using cybersecurity related technologies x Malware analysis tools x Management practices x Monitoring, testing and evaluating cybersecurity controls’ effectiveness x x Multidiscipline aspect of cybersecurity x Offensive and defensive security practices x Offensive and defensive security procedures x Operating systems security x x x x x Pedagogical standards, methodologies and frameworks x Penetration testing procedures x Penetration testing standards, methodologies and frameworks x Penetration testing tools x Privacy impact assessment standards, methodologies and frameworks x Privacy-by-design standards, methodologies and frameworks x Privacy-Enhancing Technologies (PET) x Resource management x Responsible information disclosure procedures x x Risk management recommendations and best practices x Risk management standards, methodologies and frameworks x x Risk management tools x Secure coding recommendations and best practices x Secure development lifecycle x x Secure Operation Centres (SOCs) operation x Security architecture reference models x Testing procedures x x Testing standards, methodologies and frameworks x Threat actors Tactics, Techniques and Procedures (TTPs) x APPENDIX B
Curricula Designer with Enhanced ECSF Analysis ARES 2023, August 29–September 01, 2023, Benevento, Italy Table 2: Mapping of Skills to Profiles (P) [9] Skill 1 2 3 4 5 6 7 8 9 10 11 12 Analyse and comply with cybersecurity-related laws, regulations and legislations x Analyse and consolidate organisation’s quality and risk management practices x Analyse and implement cybersecurity policies, certifications, standards, methodologies and frameworks x Analyse business processes, assess and review software or hardware security, as well as technical and organisational controls x Anticipate cybersecurity threats, needs and upcoming challenges x Anticipate required changes to the organisation’s information security strategy and formulate new plans x Apply auditing tools and techniques x Assess and enhance an organisation’s cybersecurity posture x Assess the security and performance of solutions x Audit with integrity, being impartial and independent x Automate threat intelligence management procedures x Build a cybersecurity risk-aware environment x Build resilience against points of failure across the architecture x Carry out working-life practices of the data protection and privacy issues involved in the implementation of the organisational processes, finance and business strategy x Collaborate with other team members and colleagues x x x x Collect information while preserving its integrity x Collect, analyse and correlate cyber threat information originating from multiple sources x x Collect, evaluate, maintain and protect auditing information x Communicate, coordinate and cooperate with internal and external stakeholders x x Communicate, explain and adapt legal and regulatory requirements and business needs x Communicate, present and report to relevant stakeholders x x x x x x x x Comprehensive understanding of the business strategy, models and products and ability to factor into legal, regulatory and standards’ requirements x Conduct ethical hacking x Conduct technical analysis and reporting x x Conduct user and business security requirements analysis x Conduct, monitor and review privacy impact assessments using standards, frameworks, acknowledged methodologies and tools x Configure solutions according to the organisation’s security policy x Coordinate the integration of security solutions x Decompose and analyse systems to develop security and privacy requirements and identify effective solutions x x Decompose and analyse systems to identify weaknesses and ineffective controls x x x Define and apply maturity models for cybersecurity management x Design systems and architectures based on security and privacy by design and by defaults cybersecurity principles x Design, apply, monitor and review Information Security Management System (ISMS) either directly or by leading its outsourcing x Design, develop and deliver learning programmes to cover cybersecurity needs x Develop and communicate, detailed and reasoned investigation reports x Develop codes, scripts and programmes x Develop cybersecurity exercises including simulations using cyber range environments x Develop evaluation programs for the awareness, training and education activities x Develop, champion and lead the execution of a cybersecurity strategy x Draw cybersecurity architectural and functional specifications x Enable business assets owners, executives and other stakeholders to make risk-informed decisions to manage and mitigate risks x Establish a cybersecurity plan x Explain and communicate data protection and privacy topics to stakeholders and users x Explain and present digital evidence in a simple, straightforward and easy to understand way x Follow and practice auditing frameworks, standards and methodologies x Generate new ideas and transfer theory into practice x Guide and communicate with implementers and IT/OT personnel x Identify and exploit vulnerabilities x Identify and select appropriate pedagogical approaches for the intended audience x Identify and solve cybersecurity-related issues x x x x x Identify needs in cybersecurity awareness, training and education x Identify non-cyber events with implications on cyber-related activities x Identify threat actors TTPs and campaigns x Identify, analyse and correlate cybersecurity events x Implement cybersecurity recommendations and best practices x Implement cybersecurity risk management frameworks, methodologies and guidelines and ensure compliance with regulations and standards x Influence an organisation’s cybersecurity culture x Integrate cybersecurity solutions to the organisation’s infrastructure x Lead the development of appropriate cybersecurity and privacy policies and procedures that complement the business needs and legal requirements; further ensure its acceptance, comprehension and implementation and communicate it between the involved parties x Manage and analyse log files x Manage cybersecurity resources x Model threats, actors and TTPs x Monitor new advancements in cybersecurity-related technologies x Motivate and encourage people x x Organise and work in a systematic and deterministic way based on evidence x Perform social engineering x Practice all technical, functional and operational aspects of cybersecurity incident handling and response x Propose and manage risk-sharing options x Propose cybersecurity architectures based on stakeholder’s needs and budget x Provide training towards cybersecurity and data protection professional certifications x Review and enhance security documents, reports, SLAs and ensure the security objectives x Review codes assess their security x Select appropriate specifications, procedures and controls x Think creatively and outside the box x Understand legal framework modifications implications to the organisation’s cybersecurity and data protection strategy and policies x Understand, practice and adhere to ethical requirements and standards x Use and apply CTI platforms and tools x Use penetration testing tools effectively x Utilise existing cybersecurity-related training resources x Work ethically and independently; not influenced and biased by internal or external actors x Work on operating systems, servers, clouds and relevant infrastructures x Work under pressure x