Evaluation of NFDI4Earth education training events based on NFDI4Earth educational resources (NFDI4Earth Deliverable D1.3.9)

Deliverable D1.3.9 Evaluation of NFDI4Earth education training events based on NFDI4Earth educational resources Remon Sadikni ([email protected]), Yomna Eid , Kemeng Liu , Farzaneh Sadeghi 2024-12 DOI: 10.5281/zenodo.17860036 nfdi4earth.de Citation Remon Sadikni, Yomna Eid, Kemeng Liu, Farzaneh Sadeghi. 2025. Evaluation of NFDI4Earth education training events based on NFDI4Earth educational resources (NFDI4Earth Deliverable D1.3.9). NFDI4Earth Community on Zenodo. https://doi.org/10.5281/zenodo.17860036 License This work is licensed under a Creative Commons “Attribution 4.0 International” license. Acknowledgement This work has been funded by the German Research Foundation (DFG) through the project NFDI4Earth (DFG project no.460036893, https://www.nfdi4earth.de/) within the German National Research Data Infrastructure (NFDI, https://www.nfdi.de/). NFDI4Earth D1.3.9 Executive summary This report presents a common questionnaire for evaluating NFDI4Earth training courses, covering both online courses on the EduPortal platform and in-person EduHub events such as workshops and summer schools. The questionnaire evaluates teaching materials, lecturers, participant background, and general settings. Evaluation results from four training events are presented, demonstrating the practical application of this approach. The questionnaire will be used in EduTrain (online) courses and events in the future to ensure high-quality training in research data management for the Earth System Sciences community. Evaluation of NFDI4Earth education training events Abbreviations D1.3.15 – Concept for establishing and maintaining a network of NFDI4Earth education and training sites (EduHubs) EduHubs – A network of NFDI4Earth education and training sites EduPortal – An online platform for NFDI4Earth training courses, see https://edutrain.nfdi4eart h.de/ EduTrain – Education and Training Materials and Services, see https://www.nfdi4earth.de/2par ticipate/education-training ES – Earth System ESS – Earth System Sciences FAIR – Findability, Accessibility, Interoperability, and Reusability HSBO – Bochum University of Applied Sciences LMS – Learning Management System M1.3 – Measure 1.3: Education and Training Materials and Services NFDI – National Research Data Infrastructure (Nationale Forschungsdaten Infrastruktur), see http://www.nfdi.de NFDI4Earth – NFDI Consortium Earth System Sciences / NFDI Konsortium Erdsystemforschung, see https://www.nfdi4earth.de/ OER – Open Educational Resources RDM – Research Data Management TA – Task Area TUM – Technische Universität München UNIHH – Universität Hamburg UNIMS – Universität Münster Evaluation of NFDI4Earth education training events Contents 1. Introduction 1 2. Theoretical basis of teaching evaluation 1 3. Evaluation method 2 4. Evaluation form for online courses 2 5. Evaluation form for EduHub events 3 6. Evaluation results of training events 4 6.1. Evaluation of “Image Analysis for Mapping” exercises . . . . . . . . . . . . . . . 4 6.2. Evaluation of PhD course “Python for Geospatial Analysis” . . . . . . . . . . . . 6 6.3. Evaluation of workshop “How to create publishable netCDF data” . . . . . . . . 7 6.4. Evaluation of workshop “Spatial Data Science across Languages (SDSL)” . . . . . 8 7. Outlook 11 References 11 A. Appendix 12 Evaluation questionnaire for the netCDF workshop in 2023 . . . . . . . . . . . . . . 12 Evaluation questionnaire for “Spatial Data Science across Languages (SDSL)” workshop 13 Merged evaluation form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Evaluation of NFDI4Earth education training events 1. Introduction The goal of this report is to present a common questionnaire for evaluating NFDI4Earth training courses. These education activities provide online and in-person training in the area of research data management in the earth system sciences. Apart from developing the EduPortal platform, which offers online courses, the EduTrain team members host EduHub events such as workshops and summer schools (see D1.3.15). To ensure that the provided courses and teaching events (e.g., workshops, lectures, etc.) meet the needs of the recipients and are effectively contributing to the recipients’ learning experience, we conduct evaluations of the provided events. 2. Theoretical basis of teaching evaluation Evaluation of teaching activities is an empirical investigation and systematic analysis of the teaching concepts, requirements, processes, and effects. It provides valuable feedback regarding the quality and outcomes of the learning materials, and serves as a basis for further improvement. From a theoretical point of view, evaluation can be seen as a form of problem-solving strategy which aims at delivering reliable empirical data and decomposing complex problems to make rational decisions on teaching processes (Rindermann 2001, 2003). There are usually three types of evaluation: self-evaluation, external evaluation, and student evaluation. In self-evaluation, teaching personnel assess their own performance, while external evaluation is conducted by experts. However, in this report, we will focus on the third type of evaluation, student evaluation, where the assessment is provided by the participants of the courses or training events. Student evaluation can be conducted in different formats, the most prominent examples are open interviews and questionnaires. There is a wide variety of questionnaires, in the most basic form, the questions about successful learning (or teaching) can be classified in 4 categories: 1. the material (complete, up-to-date, understandable) 2. the teacher (clarity, motivation, competence), 3. the student (motivation, pre-existing knowledge, absence), 4. and the general setting (media facilities, size of audience). Of course, these categories can be subdivided to a much higher extent, but for our purpose this classification is sufficient. 1 Evaluation of NFDI4Earth education training events 3. Evaluation method We envisioned a simple common evaluation questionnaire that the course participants can fill out after the course. A challenge in evaluating teaching events in the context of NFDI4Earth is that the format of events might vary, meaning that the courses can include self-learning material but also lectures and workshops held by a teacher. In order to find a set of questions for these different events, we made use of the fact that every institution of the EduTrain team has experience in providing and evaluating ESS teaching events. Our approach was to compare and eventually merge the different questionnaires we used in our EduTrain activities. In the following, we start with the evaluation form of online courses and then we extend this one to get a complete form for EduHub events. 4. Evaluation form for online courses The online courses in our EduPortal have two functions: 1. They can be used as a self-learning resource. 2. They back up the EduHub events with training material. In this paragraph, we show the questionnaire for the self-learning resources, the second case will be described in detail in sec. 5. The seven questions for the online courses are shown in Tab. 1. Table 1: Evaluation form for online courses Question (rating: 1=not at all, 7=perfect) 1 2 3 4 5 6 7 1. Overall, how do you like the course? 2. The structure is logical / clear. 3. The content is illustrated with examples. 4. Difficulty of the content: 1=far too easy, 4=perfect, 7=far too difficult. 5. Speed of the teaching: 1=far too slow, 4=perfect, 7=far too fast. 6. My previous knowledge was: 1=too little to follow, 4=perfectly sufficient, 7=I knew everything before. 7. Theory and practice are linked. While the first question is about the overall perception of the course, all the other questions deal with the quality and classification of the training material. Only in question 6, we incorporate the previous knowledge of the course participants to get feedback about the difficulty level. The rating has 7 possible nuances from “not at all” to “perfect”; in case the rating deviates from the norm, the meaning of the numbers 1 to 7 is defined in the question (4, 5, 6). 2 Evaluation of NFDI4Earth education training events 5. Evaluation form for EduHub events For the evaluation of EduHub events, we need additional questions about the quality of the teacher and, if the course takes place in-person, of the equipment in the class room. The first 7 questions of the evaluation form of Tab. 1 are extended by the questions shown in Tab. 2. Table 2: Additional evaluation questions for EduHub events Question (rating: 1=not at all, 7=perfect) 1 2 3 4 5 6 7 8. There are enough tutors present. 9. The lecturers seem well prepared. 10. The lecturers manage their time well. 11. It is important for the lecturers that participants learn something. 12. Accompanying documents and materials are accessible, complete, useful and up-to-date. 13. The equipment is in good condition. 14. What did you like about the workshop? 15. What can be improved about the workshop? Questions 9, 10, 11 are about the quality of the lecturer(s), question 12 deals with the quality of the accompanying material. In case the event takes place in-person, questions 8 and 13 come into play asking for the satisfaction with the general setting in the course room. At the end of the questionnaire, there are 2 questions where participants can enter free text to share some thoughts about what was right and what was wrong about the event. The complete questionnaire including the first 7 questions can be found in Tab. 9. With these questions we cover all 4 aspects of training events that we discussed in section 2. 3 Evaluation of NFDI4Earth education training events 6. Evaluation results of training events In the already given training events, we used different evaluation methods and forms to gain experience on their respective advantages and disadvantages. The recently merged questionnaire shown in this report was only used once (see 6.3) up to now, but will be applied from now on to every course. In this section, we will show the evaluation results of the following events: 1. Exercises for “Image Analysis for Mapping” at the Technical University of Munich during the winter semester of 2022/2023 2. PhD course “Python for Geospatial Analysis” at Aalborg University Copenhagen, held from 24th to 26th April 2023 3. Workshops ”How to create publishable netCDF data” conducted by the Integrated Climate Data Center, University Hamburg in 2023 and 2024 4. Workshop “Spatial Data Science across Languages (SDSL)” conducted by University Münster on September 18th-19th. 6.1. Evaluation of “Image Analysis for Mapping” exercises In 2022, the project “Coding4Geo” was funded and developed as one of our educational pilots (Tab. 3). Post-development, the material has been successfully integrated into the ”Image Analysis for Mapping,” lecture offered during the winter semester as part of the Cartography M.Sc. program at the Technical University of Munich. Subsequently, a comprehensive evaluation process was carried out by the course organizer to assess the effectiveness and impact of the course materials. To do so, they used their evaluation template, which shared considerable similarities with the NFDI4Earth evaluation survey. The University of Munich's evaluation process has provided valuable insights into the effectiveness of the course material. Below are the outcomes of the evaluation related to the portion of material covered by the Coding4Geo project: • Structure of content: students unanimously believed that the contents of the course logically build on each other. • Clarity of learning outcomes: the intended learning outcomes were found to be clearly defined in the course description, as unanimously agreed upon by students. • Roll of exercises: students universally believed that the exercise modules contributed to a deeper understanding of the subject matter. • Comprehension: all participants indicated their ability to reproduce key concepts from this module. 4 Evaluation of NFDI4Earth education training events This feedback was highly valuable and encouraged the second edition in the series of the SDSL workshops to be organized internationally incorporating the suggested improvements on the 18th-19th of September 2024 at the Geographical Institute of the Charles University, Prague, Czechia. The evaluation questionnaire was also forwarded and adapted for use by the organizers at Charles University of Prague. 7. Outlook Up to now, we developed a common questionnaire for evaluating our EduTrain Courses by the participants. The next step is to use this evaluation form for every online course and every EduHub event so that we get comparable evaluation results for securing a high quality level. In the future, we will use feedback of our participants about the questionnaire to improve our concept. References Rindermann, H. (2001): Lehrevaluation – Einführung und Überblick zu Forschung und Praxis derLehrveranstaltungsevaluation an Hochschulen. Mit einem Beitrag zur Evaluation computerbasierten Unterrichts. Landau: Empirische Pädagogik. Rindermann, H. (2003): Lehrevaluation an Hochschulen: Schlussfolgerungen aus Forschung und Anwendung für Hochschulunterricht und seine Evaluation. Zeitschrift für Evaluation (2003), 2, S. 233-256. 11 Evaluation of NFDI4Earth education training events A. Appendix Evaluation questionnaire for the netCDF workshop in 2023 Table 7: Evaluation questionnaire used for the netCDF workshop in 2023 Question 1 2 3 4 5 6 7 The course structure is logical / clear. (1=not at all, 7=perfect) The importance / relevance of the contents is made clear. (1=not at all, 7=perfect) The content is illustrated with examples. (1=not at all, 7=perfect) Difficulty of the content: 1=far too easy, 4=perfect, 7=far too difficult. Speed of the teaching: 1=far too slow, 4=perfect, 7=far too fast. My previous knowledge was: 1=too little to follow, 4=perfectly sufficient, 7=I knew everything. The content corresponds to its title and description. (1=not at all, 7=perfect) Theory and practice are linked. (1=not at all, 7=perfect) The tasks / presentation topics are relevant. (1=not at all, 7=perfect) There are enough tutors present. (1=not at all, 7=perfect) The lecturers speak clearly (speed, volume, language skills). (1=not at all, 7=perfect) The lecturers are able to explain complicated issues. (1=not at all, 7=perfect) The lecturers seem well prepared. (1=not at all, 7=perfect) The lecturers manage their time well. (1=not at all, 7=perfect) The lecturers are dedicated and convey enthusiasm. (1=not at all, 7=perfect) The lecturers encourage students’ questions and active participation. (1=not at all, 7=perfect) The lecturers are friendly in teacher-student interaction. (1=not at all, 7=perfect) The lecturers moderate discussions well. (1=not at all, 7=perfect) It is important for the lecturers that students learn something. (1=not at all, 7=perfect) Accompanying documents and materials are accessible, complete, useful and up-to-date. (1=not at all, 7=perfect) I have learned a lot taking this course. (1=not at all, 7=perfect) In this course, I learned more: 1=from the lecturers, 7=by myself. The room is suitable (windows, accessibility, visibility). (1=not at all, 7=perfect) There is enough equipment for each participant. (1=not at all, 7=perfect) The equipment is in good condition. (1=not at all, 7=perfect) 12 Evaluation of NFDI4Earth education training events Evaluation questionnaire for “Spatial Data Science across Languages (SDSL)” workshop Location: Institute of Geoinformatics, University of Münster Workshop Homepage: https://r-spatial.org/sdsl/ Form prepared by: Yomna Eid Response rate: 90% Table 8: Evaluation questionnaire for SDSL workshop Question Response Type Name (optional) How did you participate? (a) Online, (b) In-person (multiple-choice) How likely is it that you would recommend the SDSL workshop to a friend or colleague? (1=Not at all likely, 10=Extremely likely) (scale 1 to 10) Overall, how would you rate the workshop? (a) Excellent, (b) Very good, (c) Good, (d) Fair, (e) Poor (multiple-choice) What did you like about the workshop? (free-text) What can be improved about the workshop? (free-text) How organized was the workshop? (a) Extremely organized, (b) Very organized, (c) Somewhat organized, (d) Not so organized, (e) Not organized at all (multiple-choice) Prior to the workshop, how much information that you needed did you get? (a) All, (b) Most, (c) Some, (d) Little, (e) None (multiple-choice) What did you think about the length of the workshop in proportion to the content? (a) Much too long, (b) Too long, (c) About right, (d) Too short, (e) Much too short (multiple-choice) Is there anything else you would like to share about the workshop? (free-text) 13 Evaluation of NFDI4Earth education training events Merged evaluation form Table 9: Merged evaluation form for EduHub events Question (rating: 1=not at all, 7=perfect) 1 2 3 4 5 6 7 1. Overall, how do you like the workshop? 2. The structure is logical / clear. 3. The content is illustrated with examples. 4. Difficulty of the content: 1=far too easy, 4=perfect, 7=far too difficult. 5. Speed of the teaching: 1=far too slow, 4=perfect, 7=far too fast. 6. My previous knowledge was: 1=too little to follow, 4=perfectly sufficient, 7=I knew everything. 7. Theory and practice are linked. 8. There are enough tutors present. 9. The lecturers seem well prepared. 10. The lecturers manage their time well. 11. It is important for the lecturers that participants learn something. 12. Accompanying documents and materials are accessible, complete, useful and up-to-date. 13. The equipment is in good condition. 14. What did you like about the workshop? 15. What can be improved about the workshop? 14