Teach Engineering Students Expert Thinking Through Thinking Aloud During Problem-Solving

Practice Paper Recommended citation: Qi, H., Li, B., & Chen, C. (2025). Teach Engineering Students Expert Thinking Through Thinking Aloud During Problem-Solving. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631216. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. 1 ENHANCE STUDENTS' PROBLEM SOLVING SKILLS THROUGH THINK-ALOUD EXERCISE H Qia, B Li a , C Chena a University of California San Diego, La Jolla, United States Conference Key Areas: Engineering skills, professional skills, and transversal skills Keywords: Think aloud, Problem-solving, metacognition ABSTRACT Think-aloud is a valuable strategy for engineering students to develop problem-solving skills. By articulating their thought process, students become more aware of their reasoning, monitor their understanding, and identify problem-solving gaps and conceptual errors. This paper presents a series of in-class interventions implemented to foster think-aloud habits and enhance metacognitive problem-solving skills among engineering students, including oral in-class strategy discussions and structured written homework. To evaluate the impact of the structured homework and students' perception, a mixed-methods study was conducted. Quantitative analysis revealed positive correlations between the quality of students’ written problem-solving processes and their exam performance. In addition, survey responses indicated mixed students’ perceptions: while some students expressed concerns about the additional workload, many valued the structured written homework approach for its clarity and reflective benefits. 1 INTRODUCTION Problem-solving is a critical skill in engineering, requiring not only technical knowledge but also strategic thinking and adaptive reasoning (Bransford et al., 2004; Voss 1983). Research in the learning sciences highlights a key distinction between experts and novices: how they organize knowledge and approach problem solving. Experts tend to identify relevant principles, develop a plan, and anticipate outcomes—often verbalizing their thinking along the way (Chi et al., 1981). Novices, by contrast, often dive into calculations without a clear strategy, bypassing the planning phase and attempting to understand principles only after reaching a solution (Larkin, 1983). This difference reflects not just disparities in content knowledge but in metacognitive ability—the capacity to plan, monitor, and evaluate one’s thinking—which is essential for effective and transferable problem-solving. Without explicit support in developing these skills, students often resort to rote memorization: recalling solutions to familiar problems but struggling to apply the same concepts in new contexts. Think-aloud is a cognitive strategy in which individuals verbalize their thoughts in real time while engaging in a task, such as reading, analyzing, or problem solving (Cowan, 2019). It makes internal reasoning visible, allowing both learners and instructors to observe and evaluate cognitive and metacognitive processes, helping students move beyond rote memorization to a deeper engagement with core principles. By articulating why a concept or method is used, students develop a 2 clearer understanding of problem structure and reasoning, which in turn promotes conceptual learning and the transfer of knowledge across contexts (Schmidmaier et al., 2013 & Bransford et al., 2004). Previous studies have documented some of the impacts. McKeown and Gentilucci (2007) found that think-aloud strategies support English language learners by enhancing their language comprehension and academic engagement. Ebner (2013) demonstrated that think-aloud techniques can improve vocabulary acquisition among university students. Ortlieb, Norris, and Ehri (2012) showed that using think-aloud methods significantly enhances science comprehension in young children. Mathew (2014) employed think-aloud protocols to assess undergraduate students’ understanding of engineering concepts, while Dounas-Frazer (2016) used them to help students model and troubleshoot electrical circuits in a physics course. Gardin (2010) and Van Someren et al. (1994) investigated the role of think-aloud activities in fostering expert-like thinking, emphasizing how verbalizing thought processes can mirror the reasoning patterns of experienced problem solvers. Thus, designing learning activities that promote think-aloud is essential for developing students’ metacognitive and problem-solving skills. There are many widely used pedagogical approaches that can prompt students to think aloud, such as Think-Aloud Pair Problem Solving (TAPPS) (Noh et al., 2004; Johnson, 1999; Rofiqah et al., 2020), Socratic Questioning (Makhene, 2019; Sahamid, 2016), Problem-Based Learning (PBL) (Wijnia et al ,2024; Trullàs et al. 2022), structured reflection (Matheson et al., 2017; Dounas-Frazer & Reinholz, 2015), and oral exams (Qi et al, 2022; Iannone et al, 2020; Gardner & Giordano, 2023). For example, Henjes (2007) used structured reflection to help students solve complex word problems, demonstrating that guided verbalization enhances problem-solving effectiveness. Qi (2022) implemented both oral exams and video-based assignments in a Statics and Dynamics course to engage students in think-aloud practices and improve their problem-solving skills. Despite its documented benefits, literature on the impact of the use of think-aloud in undergraduate engineering courses remains relatively limited, partly because of the dynamic and non-linear nature of metacognitive processes, which can be challenging to capture and assess. For example, during Think-Aloud Pair Problem Solving in mid-to-large class sizes, it can be difficult for instructors to evaluate the effectiveness of students’ metacognitive processes and provide timely feedback. On the other hand, while video assignments more naturally promote the think-aloud process through verbal articulation, they remain challenging to assess and grade efficiently. Although oral exams (Qi, 2022) have been shown to be effective in promoting deep thinking and revealing students’ reasoning, they can be difficult to implement at scale without adequate teaching assistant (TA) support. Thus, it is important to design and implement learning activities that actively engage students in think-aloud processes, while also capturing their reasoning in a format that allows for efficient grading and actionable feedback. This paper presents a series of in-class interventions implemented to foster think-aloud habits and enhance metacognitive problem-solving skills among engineering students, including oral in-class strategy discussions and structured written homework. The oral in-class strategy discussion helps students form the habits of thinking aloud and getting peer and instructor validation during class time, 3 but it is challenging to capture each individual student’s status. Students were asked to document in writing their problem-solving strategies and key takeaways while solving homework problems. The study aims to understand the impact of documenting the think-aloud process on students’ learning performance, and to understand students’ perceptions of such learning activities. The two primary research questions are:1) How does the quality of written think-aloud assignments correlate with students’ assessment performance? 2) What are students’ perceptions of enhancing think-aloud through written assignments? 2 METHODOLOGY 2.1 Description of the study - Study setup and data collection method The study was conducted during the Winter 2025 term at a public research university in the United States, in a 2nd year engineering course: Statics and Introduction to Dynamics, with a total enrollment of 208 students. The class met twice weekly for 80-minute lectures, which were taught by the professor and introduced new concepts, along with a 50-minute discussion session led by the TA that focused on problem-solving. The think-aloud learning activities in this course consisted of three main components: Raise students’ awareness on difference between rote learning & problem-solving As part of a broader instructional intervention aimed at improving students' learning skills, the instructor delivered an 80-minute lecture on evidence-based learning strategies. This session introduced the concept of elaboration—explaining ideas to oneself or others—and its role in enhancing understanding and retention. Students were also introduced to Bloom’s Taxonomy (Bloom, 1964) and emphasized that the goal of learning extends beyond rote memorization to conceptual understanding, and problem-solving. The lecture highlighted how strategies like think-aloud support deeper learning. In-lecture “Think-Pair-Share” to prompt students to exercise think-aloud During in-class problem-solving, the instructor regularly incorporated “think-pair-share” activities (Mundelsee & Jurkowski, 2021). Students were prompted to discuss problem-solving strategies with a partner, focusing on key decisions and the rationale behind each step. These discussions typically lasted 1–2 minutes and occurred in small groups of 2–3 students seated nearby. Prompts were provided to guide students' thinking; an example is shown in Box 1. Students were encouraged to actively participate in these group discussions. However, due to time constraints, the level of engagement and outcomes of these discussions were not formally recorded. Structured Written Documentation of Think-Aloud Processes in Homework Students were given opportunities to engage in think-aloud exercises through their weekly homework. Each assignment consisted of 3 to 6 questions and included two parts. Part 1 was completed online and automatically graded through a digital learning platform. Part 2 required students to document their problem-solving process for selected questions. Specifically, they were asked to include: a) a pre-attempt outline of their problem-solving strategy, b) a detailed record of their actual work, including diagrams, equations, calculations, 4 or code, and c) a post-problem-solving reflection summarizing key learning takeaways. Students were provided with sample entries for the written documentation of Think-Aloud processes and were encouraged to adopt a format that suited their thinking style, as long as they clearly demonstrated their reasoning, preparation, and learning—not just the final answers (see Box 2 for examples). Box 1. Sample in-class think-aloud exercise prompts Box 2. Examples of written documentation of Think-Aloud processes for homework 2.2 Grading of the Written Documentation of Think-Aloud Processes in Homework Assignments The online portion of the homework was auto-graded through the course platform, allowing students to receive immediate feedback on answer correctness. The written documentation was submitted via Gradescope and graded out of 50 points: 30 points for the problem-solving work, and 20 points for the pre-attempt strategy outline and post-attempt reflection. Each homework assignment contained 3 to 6 questions. Due to limited TA resources, 1–2 responses per student were randomly selected for detailed grading based on performance, while the remaining were evaluated for completion. The written documentation of Think-Aloud processes were each graded on a 3-point scale: 0 = missing, 1 = insufficiently developed strategy, and 2 = well-thought out strategy or reflection. 5 2.3 Analysis Method This study employed a mixed-methods approach. First, we examined the correlation between students’ problem-solving processes and their performance outcomes. The problem-solving process was assessed through the pre-attempt strategy outlines and post-attempt reflections documented in homework. Performance outcomes were measured using several metrics, including online homework scores, midterm exam grades, and final exam grades (see Table 1). Due to limited TA resources, three final exam questions and their corresponding homework questions were randomly selected for analysis. We conducted multiple correlation analyses between the following pairs: P1 total & O1; P1 total & O2; P1-1 & O3-1; P1-3 & O3-3; P1-5 & O3-5. Table 1. Data used to measure problem-think aloud process and problem-solving outcome (i=homework #, j=final exam question number) Problem-solving think-aloud process Problem-solving outcome P1-j:(subset of P1) homework question 1 think-aloud grade for corresponding final exam content P1-1: homework 6 Question 1 P1-3: homework 5 Question 3 and Question 4 P1-5: homework 6 Question 6 O1: Quiz 2 in class grade O2: Final exam total grade O3-j: Final Exam questionby-question grade Next, we investigated students’ perceptions of the homework format, particularly the utility and preference for the written documentation, including the pre-attempt outline and post-attempt summary. This data was collected through an end-of-quarter survey administered after the final exam. A total of 196 out of 208 students completed the survey and consented to the research, which was distributed alongside a final learning reflection assignment that contributed modestly to their course grade. Students were asked to rate their agreement with the statement: “The homework written documentation of the think-aloud process helps me improve my problem-solving skills,” using a Likert scale (strongly disagree to strongly agree). They were also asked to select their preferred homework format. Descriptive analysis was conducted on the quantitative responses. Additionally, students were invited to explain their homework format preference through open-ended comments. A thematic analysis was performed to better understand the reasoning behind their choices. 3 RESULTS 3.1 How does the quality of written think-aloud assignments correlate with students’ assessment performance? A Pearson correlation analysis was conducted using students’ homework performance index. The correlation results and corresponding R values are presented in Table 2. A low positive correlation was observed in most comparisons. The strongest correlation was found between the overall homework performance 6 index and the final exam score. While correlations between think-aloud performance and specific exam questions were weaker, the data suggests that students retain critical information more effectively when they explicitly articulate their problem-solving approach in weekly assignments. This delayed effect implies that regular practice of written documentation of Think-Aloud processes assignment may help students develop more effective engineering problem-solving strategies over time. Table 2. Correlation study of think-aloud process and problem-solving outcome and their Pearson correlation R values Correlation Study R-value P1 total & O1 R=0.238 P1 total & O2 R=0.358 P1-1 & O3-1 R=0.154 P1-3 & O3-3 R=0.184 P1-5 & O3-5 R=0.052 3.2 What are students’ perceptions of enhancing think-aloud through written assignments? Students’ perceptions of the think-aloud activities were analyzed using survey data. Approximately 38.3% of students agreed or strongly agreed that these activities enhanced their problem-solving skill, 23.5% were neutral, and 38.2% felt the activities did not contribute to improvement. Students were also asked to indicate their preferred homework format from five options:1) online auto-graded + written documentation with preand post-attempt summaries (current format); 2) online auto-graded + written documentation without written documentation of Think-Aloud processes 3) online auto-graded only; 4) written documentation only with written documentation of Think-Aloud processes;5) written documentation only without written documentation of Think-Aloud processes. A majority of students (56.6%) preferred the second option—online auto-graded homework combined with written documentation but without the written think-aloud documentation. This suggests that while students value receiving immediate feedback and recording their calculations, many are less inclined to engage in reflective components of the assignment. Students valued the online portion of the homework for its immediate feedback and ability to validate their answers. The written component was appreciated for two main reasons: (1) it provided practice in engineering documentation and served as a reference for future review or discussion, and (2) it helped mitigate the risk of losing points when final answers in the online system were incorrect. While 38.3% of students recognized the value of the preand post-attempt summaries, only 17.9% preferred the current full-format homework (online + written documentation with written think-aloud). In contrast, 14.8% preferred online-only homework, and 10.8% preferred written-only assignments. Overall, 74.5% of students favored a hybrid 7 approach that combined auto-graded online problems with written documentation, while 25.5% preferred a single-mode format (online-only or written-only). A thematic analysis of open-ended survey responses provided further insight into students’ preferences regarding the homework format, particularly the inclusion of preand post-attempt summaries. On the positive side, many students felt that this format helped improve their problem-solving skills and facilitated deeper learning. They shared that articulating their strategies before and after solving problems reinforced key concepts and supported long-term retention. They viewed the summaries as useful tools for consolidating problem-solving strategies and appreciated the immediate feedback provided by the online portion of the homework, which allowed them to validate their approach in real time. Others noted that the reflective process acted as a checkpoint that helped clarify their understanding and guided their learning. This positive theme is evident in sample student comments: “Helped me fully understand the problems.” “The summaries are very useful for retention of problem-solving strategies. I find that the online portion of the homework is useful for getting instant feedback on your approach by being able to check the correctness of an answer.” However, not all students found value in the written think-aloud. A number of responses expressed frustration with what was perceived as unnecessary complexity. These students felt that combining auto-graded online work with written documentation, particularly the written think-aloud, made the process cumbersome and overly time-consuming. Some perceived the reflective writing as redundant or unproductive, noting that it did not meaningfully enhance their understanding. Some sample comments: “I’m an engineer writing essays shouldn’t have to be a prerequisite to turn in my engineering homework” “I often felt that pre and post attempts were repetitive and not productive for me personally.” These mixed responses highlight the importance of balancing cognitive benefits with workload and perceived relevance in designing effective learning activities. To better understand which groups of students found the think-aloud exercises helpful, we compared students’ perceived helpfulness ratings with their scores on a beginning-of-quarter pre-quiz, which assessed foundational physics knowledge required for the course. Pre-quiz scores were grouped into five categories based on students' responses to the perceived helpfulness of the think-aloud activities. The results of this analysis are presented in Fig. 1. This pattern is aligned with cognitive psychology literature, suggesting that the expert reversal effect is at play. Students with more prior knowledge will find instructional interventions that scaffold learning to be more tedious and less helpful. Some high-performing students may even be negatively affected by these interventions. This pattern is further supported by both the median and average pre-quiz scores across the groups. Students who perceived the think-aloud prompts as unhelpful scored significantly higher on the pre-quiz than those who found them helpful. This suggests that students who entered the course with stronger foundational knowledge may have felt less need for structured reflection, while those with weaker backgrounds were more likely to benefit from and appreciate the think-aloud activities. 8 Fig. 1. Students’ response to “The homework written documentation written think-aloud summary helps me to improve problem-solving skills” vs their pre-quiz score. Table 3. Correlation study of think-aloud process and problem-solving outcome and their Pearson correlation R values Perceived Usefulness of Homework Think-aloud Prompt Median Pre-quiz Score Average Pre-quiz Score Strongly agree 45.00 50.00 Agree 45.00 50.00 Neutral 60.00 57.77 Disagree 61.00 58.59 Strongly disagree 69.00 65.78 Person R -0.952 -0.955 4 DISCUSSION AND CONCLUSIONS In this study, we examined how think-aloud strategies can be taught and their impact on student learning. The findings open new possibilities for future pedagogical practices and research. However, several challenges remain. First, capturing students’ actual metacognitive activity is inherently difficult. The written pre-attempt outlines and post-attempt reflections may not fully represent the cognitive and metacognitive processes students engage in during problem-solving. Second, writing down the dynamic and non-linear think-aloud process can be tedious, less engaging, and even disruptive to students’ natural flow of strategy development. More organic approaches—such as video-recorded think-alouds—could offer richer insight into student thinking. However, these come with practical challenges, particularly in terms of grading and scalability. Third, these limitations highlight the broader challenge of developing valid, efficient, and scalable metrics for assessing think-aloud processes while also providing students with constructive feedback to help them refine their metacognitive strategies, problem-solving skills, and expert thinking. 5 ACKNOWLEDGEMENTS This material is based upon work supported by the United States National Science Foundation under Grant No.2315777.