Reviewing Reviews: Using Multistage Peer Reviews to Provide Feedback and Improve Student Learning

Practice Paper Recommended citation: Fisher, R., Nair, D. J., Mario, J. V., & Aryampa, S. (2025). Reviewing Reviews: Using Multistage Peer Reviews to Provide Feedback and Improve Student Learning. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631505. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. Reviewing Reviews: Using multistage peer reviews to provide feedback and improve student learning R.M. Fisher a, D. J. Nair a,1, J. Videla Marió a, S. Aryampa a a School of Civil and Environmental Engineering, UNSW Sydney Conference Key Areas: Engineering skills, professional skills, and transversal skills Keywords: Feedback, Reflection, AI analysis, Meta-reviews, Peer Review ABSTRACT A fundamental skill in engineering practice is providing and receiving feedback. However, this is not always explicitly incorporated in curriculums. In this study, ‘metareview’ tasks were incorporated into four courses within civil and environmental engineering. While the core stages of the meta-review process were consistent (peer review–review of feedback – reflection and correction), variations included different cohorts, discipline areas, assignment types, feedback types and marking structures. The process was evaluated using the feedback itself, student perceptions via surveys and teacher reflections. Thematic analysis of the feedback using generative AI identified key overall themes. Feedback most appreciated by peers was that which was specific, actionable, constructive and had a professional or supportive tone. The influence of expertise of those providing feedback showed higher technical engagement and critique from those more familiar with the topics, and more structural and presentation/audience based suggestions from those less familiar. However, both provided useful constructive feedback, and differences in assigned grades were not significant. Post activity evaluation showed significant improvements in assignment grades, with positive student perceptions. While facilitation of the second layer of the review process was challenging, our findings show that ‘meta-review’ type tasks can help support students in learning what effective feedback is. 1 Corresponding Author D.J Nair [email protected] 1 INTRODUCTION Feedback supports students to deepen their understanding, reflect on their performance, and improve. In the education context, feedback can be defined as the information provided by an agent (e.g. teacher, peer, self) regarding aspects of one's performance (Hattie, 2007).Over the last two decades, there has been increasing interest in understanding how effective feedback operates, specifically in higher education contexts. Previous research has identified key characteristics of effective feedback in higher education. Li and De Luca (2012) found that students prioritise timely feedback as a critical component. Dawson et al. (2019) provided empirical evidence from two Australian universities showing that students value high-quality feedback comments (84%) that are usable, detailed, specific, and personalised, enabling them to take actionable steps for improvement. However, their study revealed a gap on what makes feedback effective: while both students and staff defined feedback's purpose as improvement, staff mostly focused on design elements like timing and modalities (53%) whereas students prioritised the quality of the comments. To address these challenges systematically, Carless and Boud (2018) introduced Feedback Literacy, defining it as a lifelong capability connected to professional skills. Recent empirical validation by Woitt et al. (2025) showed that feedback literacy comprises two key dimensions: feedback attitudes (dispositions toward feedback) and feedback practices (cognitive and behavioural engagement with feedback), supporting the need for structured activities that develop both components. A recurrent challenge in higher education, particularly in large engineering courses, is providing detailed, timely, individual feedback. Peer feedback offers a scalable solution: research shows it can be as effective as instructor feedback for improving student understanding (Price et al., 2013). A meta-analysis by Huisman et al. (2019) examining 24 studies found peer feedback significantly improves academic writing compared to no feedback (g = 0.91) and performs similarly to teacher feedback. However, effective peer feedback requires training, as students often lack clear understanding of evaluation criteria, leading to superficial or unconstructive feedback (Philippakos, 2017). The process of undertaking peer assessment also promotes the re-evaluation of the students' own work (internal feedback) as well as undertaking critical thinking, and problem solving (Low et al., 2022). Still, these concepts are not always included in curriculum or remain part of implicit learning outcomes. This gap requires a more targeted process where critical thinking, reviewing, and providing feedback are directly practiced through structured reflection on feedback quality and effectiveness. Meta-reviews (that is, the process where students reflect on and review feedback provided by peers – Figure 1) provide structured practice by having students assess the quality and usefulness of peer feedback, developing the analytical and reflective capabilities essential for Engineering professional practice. In this research paper, staged reviews, ‘meta-reviews’ were incorporated into four different courses within Civil and Environmental Engineering at UNSW Sydney, Australia. Civil and Environmental Engineering was selected due to the research team’s disciplinary expertise and requirement for graduates to develop professional competencies. While the core stages of the process were consistent (peer review– review of feedback – reflection and correction), variations included different cohorts, discipline areas, assignment types, feedback types and marking structures. Figure 1. Schematic of the meta-review approach. Green arrow indicates peer-review stage, blue arrow indicates meta-review stage, and pink arrows show potential learning benefits gained by students. 2 METHODOLOGY 2.1 Task Set-up and Administration The staged peer review process comprised three sequential steps (Figure 2). This process was facilitated using the Moodle Workshop tool (Moodle, 2024), a widely used platform for online peer review activities. The process comprised: 1. Submission: Students submit their assessment to Moodle Workshop (Module 1). 2. Peer-review stage: Students would then be paired with one to two students to provide feedback. Rubrics were provided to students showing clear guidelines about how to evaluate each section of the submission. 3. Meta-review: After students would receive their feedback, they would provide feedback on their feedback (i.e. meta feedback) in Module 2A and B. Another rubric was provided to the students to clarify the expected criteria of the meta feedback. This approach was implemented across four courses (Table 1) with consistent rubric structures (shown in APPENDIX A). Figure 2. Administration of the staged peer review process using the Moodle Workshop activity. Table 1. Summary of courses incorporating staged peer feedback activities Code Type Size Submission type First level Second level Third level / Reflection UGRD-1 Fourth year core capstone course. 34 3-page site visit report detailing current infrastructure systems, each student focused on one of 4 systems 2 peers (one same topic, one different topic) individually provide written comments and summative grade (5%) according to 3x3 rubric Feedback quality of peers evaluated with written comment and summative grades (5%) according to a 3x3 rubric Reflection encouraged in final group report and interview. Optional survey UGRD-2 Fourth year core capstone course. Repeat 54 PGRD Elective course for PG and Fourth year elective course for UG 40 Preliminary Project Design Brief related to a transport planning and management topic 2 peers individually provide written comments and summative grade (5%) according to 3x3 rubric Feedback quality of peers evaluated with written comment and summative grades (5%) according to a 3x3 rubric Reflection encouraged in final group report, Resubmission of the preliminary report and interview. Optional survey UGRD+ PGRD Fourth year elective course, also taken by postgrad students 127 Draft presentation on the use of sustainability assessment methods, each student focused on one of 9 themes 2 peers (one same topic, one different topic) individually provide written comments and indicative grades (formative) according to 3x3 rubric Feedback quality of peers evaluated with written comment and summative grades (5%) according to a 4x3 rubric No formalized reflection activity. Optional survey 2.2 Data collection methods Data collection included student perceptions via surveys, course content from the peer review activities, and teacher reflections of the process. Permission to survey students as well as to use anonymised course data (written comments and grades) was acquired (Human Ethics iRECS6901). Post course surveys were used to gather student responses regarding the effectiveness of the activities in improving reviewing skills and improving student understanding of assessment requirements. Surveys were distributed to students via email following course and/or activity completion, however response rates were low (11 out of 250 students). 2.3 Mixed-method analysis A mixed-methods approach was adopted to encompass both the quantitative and qualitative information gathered. Quantitative analysis examined student-assigned marks for first (feedback) and second-level (meta-feedback) reviews. The following hypotheses were tested: a) In courses where students provided feedback on same and different topics (UGRD-1, UGRD-2, UGRD+PGRD), do the marks given differ? b) Do students who received higher marks subsequently give higher meta feedback ratings? For qualitative analysis, the written feedback was analysed using initial qualitative data categorization and exploration of topics using a large language model (LLM)-based approach as outlined by Hayes (2025). This LLM framework provided a systematic and interactive foundation for generating labels and identifying illustrative quotes (prompt examples provided in APPENDIX B). Using the processed information, research questions examined included: how important is student expertise in providing useful feedback, how do students respond to feedback tone, and effectiveness based on post task reflections. 3 RESULTS 3.1 Qualitative analysis of feedback comments Recurrent feedback themes in the written comments from all courses were identified using the LLM based approach, the following key themes were reported. Clarity and Specificity: Across all courses, students emphasised the necessity for feedback that was clear, specific, and unambiguous. In UGRD-1 and UGRD-2, reports explicitly noted feedback that lacked specificity or clarity led to confusion and limited its utility. Similarly, in PGRD, students reported that vague connections between research questions and methodologies weakened feedback applicability. Practicality and Actionability: Students repeatedly valued feedback offering practical steps toward improvement. According to the UGRD+PGRD and UGRD actionable suggestions significantly enhanced student comprehension of assessment requirements and helped in refining their submissions. Students specifically praised instances where peer feedback outlined clear pathways or actions. Balance and Constructiveness: Many students appreciated feedback that incorporated recommendations and constructive criticism. In UGRD-2 there were instances where overly negative or positive feedback was not aligned with the constructive recommendations, diminishing its usefulness. Technical Details: Technical specificity, including quantitative data and precise methodological details, was frequently highlighted as critical to the perceived credibility and usefulness of peer feedback. For instance, PGRD noted student interest in detailed technical explanations related to queuing theory models and data collection processes. While for UGRD-1 and UGRD-2, students were often suggested to incorporate figures, maps, and diagrams to enhance their reports. Focusing on the meta-review feedback students generally reported improved clarity regarding assessment criteria and greater confidence in understanding assignment expectations. Course specific details showed: - UGRD-1 and UGRD-2: Students particularly valued the depth of feedback regarding comprehensive site analyses and infrastructure assessments, contextualising the onsite observations and linking them to assignment requirements. - UGRD+PGRD: Students emphasised the importance of clearly justified methodological choices and explicit articulation of data boundaries and inputs, highlighting the value of transparency in sustainability assessment methodologies. - PGRD: The applicability and relevance of theoretical concepts such as queuing theory emerged as a critical factor. Students expressed appreciation for feedback clearly linked to practical implications and realistic scenarios, underscoring the importance of integrating theoretical knowledge with real-world applications. 3.2 Trends in Grades Analysis of peer review and meta feedback marks tested two hypotheses regarding the topic matching effects and correlation between received marks and subsequent given marks on the meta feedback process. Descriptive statistics for peer review feedback and meta feedback marks for the four courses are presented in Table 2. Independent sample t-tests examined whether students marked differently when reviewing same versus different topics across three courses (UGRD-1, UGRD-2, UGRD+PGRD). No significant differences were found in any course for either peer review marks (p-values: 0.910, 0.400, 0.431) or meta feedback marks (p-values: 0.732, 0.720, 0.810). This suggests that topic matching did not influence student marking behaviour. Table 2: Descriptive statistics of Feedback Marks and Meta Feedback Marks by course. Note: courses in which the tasks were formative are underlined. Course Peer Review Marks Meta Feedback Marks UGRD-1 Mean: 80.00, Std: 14.15 Mean: 85.69, Std: 18.65 UGRD-2 Mean: 79.62, Std: 11.60 Mean: 79.02, Std: 20.12 PGRD Mean: 74.39, Std: 23.41 Mean: 79.74, Std: 21.40 UGRD+PGRD Mean: 74.67, Std: 17.54 Mean: 83.95, Std: 23.79 Correlation analysis tested whether students who received higher submission marks subsequently gave higher marks on the meta feedback process. Results varied across courses: UGRD+PGRD showed a small but significant positive correlation (r = 0.174, p = 0.021), while UGRD-1 (r = 0.219, p = 0.073), UGRD-2 (r = 0.027, p = 0.787), and PGRD (r = 0.046, p = 0.689) showed no significant relationships. The third hypothesis received limited support, being supported in only one course. Notably, the significant correlation was found only in the formative assessment course, suggesting that grading stakes may influence student feedback behaviour. 4 DISCUSSION 4.1 How important is topic familiarity in the quality of feedback provided? For subjects UGRD-1, UGRD-2 and UGRD+PGRD students were assigned into different topics, and their work was reviewed by peers who completed the assignment on the same topic, or a different topic. In UGRD-1 and UGRD-2, students wrote site reports on one of four infrastructure systems: water, waste, energy or transport. In UGRD+PGRD, students used 9 different sustainability assessment methods to track consumption or flows of resources in their households (e.g. carbon footprint, flows of single use plastic). There was no statistically significant difference in marks assigned at either review stage between those in Same vs Different groups. The qualitative analysis of peer feedback comments revealed differences between same and different topic reviewers, primarily in the depth versus breadth of feedback. Same-topic reviewers provide more specialised and technically detailed feedback and were more critical at pointing out misconceptions or missing technical concepts. In contrast, different-topic reviewers offered more general, structure-focused, and audience-oriented feedback with suggestions focusing on the general structure, readability, broader context, and communication effectiveness. Analysis of the meta feedback comments showed that same-topic reviewers had a higher tendency to dispute/debate their received feedback compared to differenttopic reviewers. On the other hand, feedback received from different-topic reviewers was praised for its organisation and clarity. Some students also expressed appreciation for insights from outside their topic area. Overall, students from both groups valued clear, specific feedback with actionable steps. These findings suggest that topic familiarity can influence the nature of peer feedback, even if it didn't significantly affect the marks given. Domain expertise level can determine the sophistication of peer feedback, with high-knowledge students providing metacognitive feedback and low-knowledge students providing surfacelevel task feedback (Alqassab et al., 2018). Our findings align with the work on domain expertise and demonstrates the educational value of peer review type activities in providing corrective feedback, even when lacking certain expertise. From an educational practice perspective, there is value in feedback from varied cohorts further supporting the utility of peer review type activities. A final student perception survey (Third level/Reflection on Table 1) was conducted, but only completed by a handful of participants in the PGRD and UGRD+PGRD. It showed 67% of respondents slightly disagreeing that it’s reasonable to have feedback from peers instead of an instructor. This scepticism towards peer feedback, also identified in other studies (Kaufman et al., 2010; Kaufman & Schunn, 2010), suggests that despite benefits, students may still doubt the legitimacy of peer feedback compared to instructors. 4.2 Constructive or Critical? Tonal nuances in feedback Tone of the feedback provided emerged as an important element during the study. In some courses (UGRD 2), a number of students perceived the feedback as overly harsh as exemplified by "Your criticism tone needs a little bit of detuning…”. In other courses, students reported appreciating a professional and respectful tone when identifying problems, with a focus on the work itself rather than personal comments. Tone was introduced as one of the criteria against which feedback was compared in UGRD+PGRD. When students used motivational language in their feedback, this encouragement was noted and appreciated in the meta review feedback. Similarly, the contrast was also true, as feedback that only identified problems without any positive comments or encouragement was viewed negatively. Similar findings have been found in areas such as peer-review process in academic journals (Chong & Lin, 2024). Feedback tone that is professional, has a respectful language and focuses on the work was consistently valued, while hostile, demeaning, or overly critical tone hindered the recipient's ability to engage constructively with the feedback. Another issue encountered was when there was a disconnect between the written comments and the grades assigned on the rubric, with students noting confusion when positive-sounding feedback was accompanied by lower grades. 4.3 Effectiveness of feedback in improving assessment outcomes The final student perception survey included nine questions about the effectiveness of the peer review activity using Likert scales alongside a final open response field. Comments highlighted variability in peer feedback quality. Some students expressed concerns about receiving feedback that was occasionally superficial or perfunctory, indicating inconsistencies in peer engagement. One student explicitly remarked, "Task is only effective when students complete it properly," emphasizing the need for consistent participation and thoughtful peer engagement. Despite these concerns, the quantitative results indicated robust support for the peer feedback approach. Approximately 94% of students agreed or strongly agreed that peer comments led directly to improvements in their manuscripts. Additionally, around 67% perceived the feedback they provided to peers as useful, reflecting a generally positive outlook toward the feedback exchange. In PGRD, the preliminary design report was graded by the instructor to examine how the activity affected assessment grades. Statistical analysis of grades before and after peer feedback confirmed its effectiveness, showing a significant improvement in student performance (mean grades increased from 7.99 to 8.54, p < 0.001). The end of term Course Evaluation Report further reinforced these findings, highlighting that 100% of respondents broadly agreed that the feedback received during the course facilitated their learning as did feedback provided by the instructor. These findings underscore the value of structured, quality feedback within the teaching environment. 4.4 Instructor reflections on feedback and recommendations Instructors from each course provided their reflections on the meta feedback process and identified the following comments and recommendations. Activity Administration in Moodle Workshop: The staged peer review activity was administered using a series of layered Moodle workshop activities. This workflow required students to copy the feedback they provided in Module 1 and submit it in Module 2. This double handling created some challenges: some students didn’t complete the activity, couldn’t find where the feedback was or mixed up which of the Module 2 to submit to (luckily the last was rare). This created some extra work for instructors. Alternatives to this method could be running the session in class rather than asynchronously over a week, where troubleshooting could run in real-time. Streamlining the process could also be enabled by specialised software which appears to offer an automatic workflow but requires investment and integration. Explicit Instruction in Feedback Provision: While the task was explained in each class, and the importance prefaced in terms of professional engineering skills, student engagement remained an issue in some classes. Further targeted training in providing and receiving feedback, along with clear professional expectations, could increase student engagement in the activity. Rubric Enhancement: During the study, we identified gaps in the existing rubric, particularly regarding feedback tone, and subsequently we included tone criteria in the UGRD+PGRD rubric in response. As rubrics are important in guiding student expectations and structuring feedback, they may require ongoing refinement to enhance consistency and clarity in future peer evaluations. Emphasis on Reflective Practice: To close the feedback loop we suggest integrating structured reflective activities within all courses, to solidify student learning from the review processes. These may take the form of students writing up and documenting changes made to their assessments (much like the academic publishing peer review process), journal entries identifying areas for improvement or actively discussing in class corrective actions. Instructors were keen to continue with the activity in their own courses and explore potential applications in younger cohorts in order to scaffold and build feedback skills throughout programs. 5 CONCLUSIONS The findings of the meta-review process analysis identified the utility of the activity not only in increasing student confidence and familiarity with the submission expectations, but also in providing and receiving constructive and useful feedback. The activity was used in four different courses ranging in disciplines and cohorts, and in different assessments from formative, summative, reports and practice presentations. Across all the courses, it was found that a balance between positive reinforcement and constructive criticism was positively received. The most positively received feedback maintained a respectful, encouraging, and supportive tone while still providing honest and constructive criticism. While different types of feedback were provided by those completing the same or different topic (depth vs breadth), both still provided constructive feedback, and differences in assigned grades were not significant. Survey outcomes suggest this peer feedback process supports student learning and manuscript improvement. The staged review approach offers a scalable process to helps students internalize assessment criteria, develop nuanced understanding of effective feedback, and promotes reflection – all of which are important for professional engineering practice. Targeted refinements, including training in feedback guidelines and expectations, comprehensive rubrics, and streamlined activity administration, could further improve the engagement, consistency and scalability of the process.