Sustainable Knowledge Transfer and Examination Formats in Thermodynamics

Practice Paper Recommended citation: Demant, L. M., & Schädlich, S. (2025). Sustainable Knowledge Transfer and Examination Formats in Thermodynamics. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631676. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. Sustainable Knowledge Transfer and Examination Formats in Thermodynamics L. Demant 1 Hochschule Ruhr West, University of Applied Sciences Bottrop, Germany 0009-0001-4319-7099 S. Schädlich Hochschule Ruhr West, University of Applied Sciences Bottrop, Germany Conference Key Areas: Engineering skills, professional skills, and transversal skills, Improving higher engineering education through researching engineering education Keywords: Thinking aloud, Examination Formats, Module concept ABSTRACT The thermodynamics module presents students with many challenges, which is why the teaching and examination concept is being revised. The aim is for students to acquire sufficient knowledge to pass the module examination. Currently, the content is often practised schematically, which contradicts the requirements of teachers who want to prepare students for professional life and promote their problem-solving skills. This different focus creates a gap in the learning process, particularly in the area of ‘reflecting and understanding’. This is to be closed in the new module concept, adapted to individual learning processes in order to ensure educational equity. An essential ‘treshold competence’ is recognising overall and subsystems and energy balance derivation. The new concept is based on problem-orientated work with flipped classroom elements and is supplemented by homework, digital exercises with feedback, interviews and tutorials. Individualised support is provided on the basis of knowledge gained from exercises and interview approaches. The ‘thinking aloud’ method in interviews enables students to reflect on their knowledge and teachers to recognise obstacles to learning. The tutorials based on this define and motivate performance groups in a targeted manner. Both the homework and the interviews and tutorials are part of the examination performance and supplement the existing forms of examination in the form of digital 1 L. Demant [email protected] examinations and technical discussions. The success of the project is evaluated not only through examination results, but also through feedback and the continuous participation of the students. ___________________________________________________________________ 1 INTRODUCTION The thermodynamics module presents students with various challenges: on the one hand, they have to learn many technical topics, some of which contradict the everyday understanding of the terms ‘energy’, ‘heat’ and ‘temperature’. On the other hand, they have to learn and be able to apply a new structured approach to problems, which requires an understanding of the specialised topics and the terms used. A ‘treshold competence’ to be learnt is the recognition of overall systems and subsystems and the application of the structured approach for energy balance derivation, which are equations that are important for calculating the system. The main aim of the teaching staff is to support the development of this understanding and its application to practice-orientated problems in order to prepare students as well as possible for the requirements of their further studies and later professional life. The structured approach to be used is shown in Figure 1, whereby the area of ‘reflecting and understanding’ is of particular importance for solving problems. Figure 1: Systematic approach to the processing of thermodynamic tasks The students' focus is on passing the module examination. The long-term significance of the module for their professional life is lost sight of. Conventionally, the thermodynamics module is completed with a written examination, whereby the conventional preparation of the students usually consists of practising a ‘standard calculation method’ using a large number of exercises for various specialist topics. This approach tends to focus less on ‘reflecting and understanding’ and the processing of problems other than those practised is often not particularly successful. With this in mind, teaching and examinations in the thermodynamics module have been adapted in recent years. In addition to the ‘flipped classroom’ concept and collaborative teaching methods, examinations are currently held during the semester via the Moodle learning platform, in which calculation skills and understanding are also tested using multiple-choice questions. Individual technical discussions (1:1) take place at the end of the semester to check the understanding of interrelationships and the application of structured procedures to practice-orientated problems in greater depth. The experience of the last two years has shown that although the motivation of students and the acquisition of understanding has improved as a result of a form of examination during the semester, this has not been to an extent that does justice to the objective of ‘sustainable’ vocational preparation. This led to further investigations into student learning behaviour in the thermodynamics module, which were also carried out as part of the ‘OK!Thermo’ research project (project of 4 universities; funded by the Ministry of Schools and Education NRW). Surveys regarding learning strategies and individual interviews with students at two universities revealed that more than half of the students surveyed are best able to deal with tasks that are short and abstract and that only contain the values required to solve the task. The students seem to concentrate on memorising calculation sequences and picking out seemingly ‘suitable’ formulas from the formulary. It became obvious that the area of ‘reflecting and understanding’ is omitted by many students and that the procedure for recognising systems and setting up corresponding energy balances could obviously not be consolidated in the long term. This approach of the students does not correspond to the challenges they will face in their later professional lives. There, they are more likely to be confronted with extensive tasks that contain more than just the necessary information and are not abstract, but relate to current energy technology topics. 2 CONTEXT AND PRACTICAL WORK The competences mentioned at the beginning and in particular the understanding of facts can only be recognised insufficiently on the basis of the results of digital examinations. For this reason, the previous module concept is currently being supplemented and improved by new teaching and examination formats in which students have to demonstrate that they have acquired a basic understanding, have mastered the systematic approach and can solve unknown problems themselves. The aim is to pursue educational equity through individualised support. To this end, the students' need for support must first be determined, which can be achieved through various measures. Therefore, the new module concept is intended to complement the previous concept with various measures that are mandatory for students but not graded. An evaluation matrix was developed and trialled for documentation purposes. Figure 2: planned lecture and examination schedule for the module Thermodynamics Figure 2 shows in which week of lectures the corresponding forms of examination are planned. While the homework has to be handed in weekly, the interviews, Moodle exams and the technical discussion (oral exam) only take place on certain dates. The tutorial was integrated into the lecture period on two dates after the interviews. This approach is intended to achieve the goals of both sides: the aim of the teachers is to consolidate the students' knowledge in the long term and thus prepare them in the best possible way for the demands of further studies and everyday working life, which does not require schematic calculations, but that calculation methods and thought processes can be explained and justified and results critically scrutinised. The measures are also intended to help students achieve their goal of passing their examinations. The measures were implemented for the first time in summer semester 2024 and the resulting data was evaluated in winter semester 2024/25, meaning that initial findings are already available. The lecture and examination schedule will also be implemented again in summer semester 2025. Last year, 55 students from two degree programmes took part in the module. All participating students are engineering students. The introductory thermodynamics class is mandatory in their program. 2.1 Homework In order to ensure the continuous participation of students, they have to prepare what they have learnt in the form of short homework assignments after each lecture. The homework is documented digitally in a portfolio and consists of maths problems, explanations of the topics in the students' own words and short summaries. They serve as a basis for exam preparation, among other things. Students receive individual feedback on each homework assignment from the teacher and, if necessary, information on errors and insufficient understanding of the assignment as well as offers of specific assistance. By documenting the feedback in an Excel overview for each student, individual developments become visible and more targeted support can be provided. Points from 0 (= not completed) to 5 (= exceptionally good) were awarded for the qualitative assessment of the homework. The specially created collection of formulae was awarded points from 0 (= not created) to 3 (= completely self-created). After each course, students were asked to summarise what they had learned. This submission was awarded points from 0 (= not worked on) to 5 (= very well reflected). 2.2 Digital Exercises (Moodle) Self-learning skills are strengthened by supplementing the digital exercises on the Moodle learning platform with structured subject-specific feedback, so that students initially receive hints on the correct approach and only finally are shown the correct solution. The students' results in the exercises are to be used to recognise where students need support and to provide them with individual support in the tutorials. The students were given a total of ten tests with exercises on various topics. For the evaluation of the digital exercises, the best attempt of each student from each completed exercise test was analysed. This was documented in an Excel evaluation matrix. In detail, it was analysed how many of the ten tests the students took part in, how often they completed each test on average and the best result they achieved on average in all tests completed. 2.3 Interviews In weeks 3 and 6, a personal interview will be conducted with the students in which a thermodynamic task is to be worked on using the ‘Think Aloud’ method based on the method of Ericsson and Simon (1984). Students are asked to say out loud what is going through their minds when solving the task. The aim is not for them to explain what they are doing, but simply to shed light on their thinking and approach (Konrad, 2020). According to Völzke (2012), verbalising thinking should influence the problemsolving process. Students should be guided through the task by the interviewer if they have problems with the solution during the interview so that the structured approach can be internalised. Even during the interviews, thinking aloud can lead to students reflecting on their own knowledge and gaining insights. The interviews should also help to identify the students' support needs and support them in the tutorials. The interviewer was either a research assistant or a student tutor. The documentation takes the form of audio recordings, from which transcripts are created that enable a systematic analysis according to Mayring (2015). The analysis was carried out using MaxQDA software. Codes were defined in advance, which were used to mark the sections in the interviews. This made it possible to determine whether the students had applied the structured approach. 2.4 Tutorial Based on the (internal) assessment of homework, exercises and interviews, students' individual learning strategies are determined and the need for support identified. This allows performance groups to be defined and more targeted motivation to be provided. The original plan was to divide the students into groups for the tutorial based on these results. However, in order to give students more freedom, they were left to choose their own tutorial group. According to Artelt, 1999, adapting the teaching and examination situation to the students' learning strategies can have a positive influence on learning success. Last year, the students were divided into two tutorial groups with different support needs. The subdivision is intended to appeal to all students in order to avoid negative connotations with support groups. High-performing students should also be supported with this concept, which is based on the individual needs of the students. The tutorial groups are organised within the planned scope of teaching so that the time burden on students is not too high. The students' results in the tutorials are not assessed qualitatively. Only the fact that they have taken part in the tutorials counts as an examination achievement. 3 RESULTS AND INSIGHTS So far, the following data has been analysed in the project: - Learning strategies according to Artelt (1999) - 1st interview of the students from the first degree programme (18 students) - Results of the digital exercises - Homework (incl. formulary and ‘what did I learn?’) - Results of the Moodle digital exam - Results of the oral Exam Figure 3: Excerpt from the evaluation matrix - best students in the Moodle digital exam Figure 3 shows a section of the evaluation matrix. The table is filtered according to the best results in the Moodle digital exam. It can be seen that the best students, with two exceptions, regularly took part in the 10 exercise tests. They did their homework better than average and reflected on what they had learnt in the course. The students who did well in the Moodle digital exam also did well in the technical discussion. The best students (with one exception) rated themselves as ‘Advanced’ for the tutorial. Figure 4: Excerpt from the evaluation matrix - worst students in the Moodle digital exam Figure 4 also shows a section of the evaluation matrix. The table is filtered according to the worst results in the Moodle digital exam. The students with the worst results took part in a maximum of 6 of the 10 exercise tests. The best score they achieved on average in the exercise tests is 51%. For the most part, the homework was completed below average. With one exception, the lower-scoring students opted for the tutorial at the ‘Basic’ level. Figure 5: Students' approach to solving the task in Interview 1. Each line represents the interview process of one student (total of 18 students) Figure 5 shows an overview of the analysis of the interviews. The sections of the texts were subdivided by colour using codes. A legend can be found on the left side of the figure. 14 of the 18 students chose an abstract rather than a realistic sketch to complete the task. 14 out of 18 students were able to complete the task and came to a conclusion. Three of the students were able to solve the task without help and without a formulary, one of whom stuck to the systematic solution strategy. Two of the students from the degree programme dropped out of the module; both had already experienced difficulties in the interview. They were unable to apply the solution strategy and did not solve the task. Overall the interview process shows that the students still have problems with solving tasks that include energy balance derivations after a few weeks in thermodynamics class, even if it is a topic that is trained regularly. Steps that are important for the process of “reflect & understand”, for example “draw system boundary” or “name thermodynamic system”, take a lot of time for the students or are not used at all. This makes it difficult for the students to solve the given tasks not only in the interview, but also later in the Moodle digital exam and the oral exam at the end of the semester. The students who did poorly in the Moodle digital exam (see Figure 4) did not apply the solution strategy sufficiently or at all in the interview and were dependent on help to solve the task. Students who did well in the Moodle digital exam (see Figure 3) also needed help in the interview. Surprisingly no correlation can be seen between the results of the digital exam and the average best result in the digital Moodle-exercises. Figure 6: Connection between the assessment in the Moodle digital exam and the assessment of the homework Figure 6 shows the results of the Moodle digital exam and the (internal) assessment of the homework. A cloud of points can be seen in the top right-hand section of the diagram. This shows that the higher the quality and therefore also the assessment of the homework, the better the result in the Moodle digital exam. The students who did not submit any homework (rating = 0) were repeating the module and 80% of them achieved below-average results in the Moodle digital exam and therefore did not pass the exam. These are indications that the students have better results if they reflect their learning process through the whole semester and work on topic related exercises on a regular base. 4 CONCLUSIONS AND IMPLICATIONS An initial correlation can be recognised between the homework and the Moodle digital exam, as a high quality of homework also leads to a good result in the Moodle digital exam. Homework is a useful way of following up on what has been learnt, but only if it is accompanied by individual and in time feedback. Through additional feedback submitted by the students, it was recognised that the interviews help to gain knowledge independently. The students stated that speaking their thoughts out loud helped them to find the solution. However, further analysis of the data and the collection of comparative data this year is necessary to draw definitive conclusions. Prior studies, for example by Hassan and Mat (2005), show that students in thermodynamics tend to learn effectively in an active learning environment, while according to Fowler et al. (2009) 79% of engineering students show a tendency to visual learning. So this seems to be factors that should be focused on when implying a new teaching concept in thermodynamics. Successful approaches have been blended learning and online thermodynamics courseware, as Bullen and Russell (2007) show. These approaches are already part of the concept in this project. From the teachers point of view this new approach has a medium impact on them. Parts of this teaching concept, as of the homework, the digital and the oral exam, were already implemented in the thermodynamics module. But still the interviews forced the teacher to invest more time into the teaching concept. Each interview took about 20 Minutes, so for 18 students there were six hours needed to complete the interviews. This time and work was divided between a research assistant and two student tutors, so the work for the teacher was reduced and they just needed to work on the task that was given to the students in the interview. The analysis of the interviews also takes a lot of time, because a closer look at each interview is necessary. Compared to the “traditional” way of teaching, this teaching method takes significantly more time and effort, which is only possible because the work is divided between multiple people. The students perceptions are also divided – some would rather listen to a frontal lecture, but the predominantly feedback is that the new teaching method is more work for the students than before, but they can better understand what they are doing in class and why they are doing it. 5 ACKNOWLEDGEMENTS AI (DeepL Translator) was used for language refinements. REFERENCES Artelt, C. (1999). Lernstrategien und Lernerfolg – Eine handlungsnahe Studie. Hogrefe Verlag Göttingen Bullen, P., Russell, M. (2007). A blended learning approach to teaching first year engineering degree students. In International Conference on Engineering EducationICEE. Ericsson, K. A., Simon, H. A. (1985). Protocol analysis: Verbal reports as data. MIT Press, Cambridge, MA. L. Fowler, J. Armarego, M. Allen (2009). CASE Tools: Constructivism and its application to learning and usability of software engineering tool. School of Engineering, Murdoch University. Hassan, O., Mat, R. (2005). A comparative study of two different approaches in teaching thermodynamics. In Proceedings of the Regional Conference on Engineering Education December (Vol. 1213). Konrad, K. (2020). Lautes Denken. In: Mey, G., Mruck, K. (eds) Handbuch Qualitative Forschung in der Psychologie. Springer Reference Psychologie. Springer, Wiesbaden.