Reviewing Manuscripts in Engineering Education Journals - Reflections and Advice

Workshop Recommended citation: Edström, K., de Andrade, M., Bernhard, J., Chance, S., Björklund, T., Male, S., Thompson, G., Broadbent, R., May, D., Mitchell, J., Saunders, F., & Ventura-Medina, E. (2025). Reviewing Manuscripts in Engineering Education Journals - Reflections and Advice. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17632005. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. REVIEWING MANUSCRIPTS IN ENGINEERING EDUCATION JOURNALS – JOINT REFLECTIONS AND ADVICE K Edström a,1, M de Andrade b, J Bernhard c, S Chance d, T Björklund e, S Male f, G Thomson g, R Broadbent h, D May i, J Mitchell j, F Saunders k, E Ventura-Medina l a KTH Royal Institute of Technology, Stockholm, Sweden, 0000-0001-8664-6854 b University College London, London, UK, 0000-0002-0710-5580 c Linköping University, Linköping, Sweden, 0000-0002-7708-069X d TU Dublin, Dublin, Ireland, 0000-0001-5598-7488 e Aalto University, Helsinki, Finland, 0000-0002-3471-746X f University of Melbourne, Melbourne, Australia, 0000-0001-9852-3077 g Aston University, Birmingham, UK, 0000-0002-7104-4348 h Aston University, Birmingham, UK, 0000-0002-8160-5030 i University of Wuppertal, Wuppertal, Germany, 0000-0001-9860-1864 j University College London, London, UK, 0000-0001-5682-0460 k Manchester Metropolitan University, Manchester, UK, 0000-0002-1644-2511 l Eindhoven University of Technology, Eindhoven, The Netherlands, 0000-00021041-945X Conference Key Areas: Improving higher engineering education through researching engineering education; Building the capacity and strengthening the educational competences of engineering educators Keywords: Engineering Education Research, Journal, Peer review, Editorial process; Generative AI ABSTRACT This workshop focuses on peer review of journal manuscripts in the field of engineering education research. Together we will consider how to review manuscripts fairly, constructively and effectively. We will also discuss the ethics and implications of generative AI on submissions and on the peer review process. Both experienced and new reviewers are invited to this workshop, with a particularly warm welcome extended to doctoral students in engineering education research. The aim of the workshop is to contribute to the capacity for peer review among the engineering education research community, and to seek consensus or at least improved understanding of any tensions. 1 Corresponding Author K Edström [email protected] 1 BACKGROUND AND RATIONALE 1.1 Peer review – how a research field works The function of the peer review process is first to support journal editors in making fair decisions by helping them identify the manuscripts that deserve to be published. The task is further to give constructive support to the authors in improving their manuscript before publication. The peer review process often goes through some iteration to help authors improve their research ideas and approaches, as well as how they communicate these ideas, methodologies and results to the readers. It is through this process of both selection and enhancement that the quality of publications is safeguarded. By extension, this is how the whole research field establishes and maintains respect. It is fair to say that peer review is what makes a research field legitimate, and the role of reviewers is therefore vital. 1.2 Engineering education is a diverse field The engineering education research community is characterised by a great diversity, with scholars from various scientific backgrounds. This can be reflected in the review process as approaches differ from one scientific community to another. There may be differences in how we see the purpose of research, what constitutes quality work, what methodologies are appropriate, how manuscripts should be written, etc. Negotiating such tensions can be a productive way to develop the field and our understandings of it (Beddoes, 2012; Gardner & Willey, 2018). It is, hence, important for the engineering education research community to maintain an ongoing discussion to develop and communicate some common understandings of how to review manuscripts. 1.3 The work of reviewing Given that the quality of the research field depends on it, reviewing manuscripts is a contribution that all scholars need to make to sustain and further the field. Being a peer reviewer is an inherent part of an academic identity and a way to express scholarly collegiality. In addition to such altruistic reasons, it is also a rewarding task to review manuscripts, since much can be learned from engaging with the work of others and from experiencing the editorial process from the inside. Such insights can be helpful when preparing one’s own manuscripts and taking them from submission to successful publication. 1.4 Learning to review Given the importance of reviewing, it is a wise investment to improve one’s skills to do it well. As reviewing can also be time-consuming, it is an advantage to do it reasonably effectively. Many publishers provide resources on reviewing in general terms (Elsevier, 2025; Taylor & Francis, 2025; Wiley, 2025). Otherwise, reviewing has traditionally been learnt through practice, but even then there are ways to strengthen reflections and learning. One is through the feedback sent to reviewers by most journals after a decision is made on a manuscript. Each reviewer can seek some level of feedback and validation by reflecting on the comments by editors and other reviewers. Some engineering education journals have organised mentorship programs for reviewers (Jensen et al., 2020). Any research group can organise an activity in which young researchers can learn to review journal manuscripts under guidance as a way to support everyone’s development (Auby et al., 2024). This SEFI conference workshop, together with its predecessors at earlier conferences, is also intended to contribute to the collective review competence of the community. 1.5 Implications of AI The rapid development of generative AI creates profound consequences in higher education. In fact, there are critical implications for every single process that is based on evaluating written text. For example, assignments and essays are assessed as indicators of student learning, and written applications for academic positions contain research and teaching portfolios with visionary and reflective statements. Of course these challenges apply to the highest degree also to peer review of conference and journal submissions. Whether we want to or not, editors and reviewers are standing on the frontline to protect the integrity of the field. During the workshop we will briefly discuss how this affects the work of editors and reviewers in terms of what constitutes ethical AI use (see for instance Taylor & Francis, 2025), how we can recognise signs of misuse, and how to handle such cases. 2 WORKSHOP PARTICIPANTS AND OBJECTIVES 2.1 Target audience To this workshop both experienced and new reviewers are invited, with a particularly warm welcome extended to doctoral students in engineering education research. During the workshop there will be an opportunity for participants to express interest in becoming a reviewer. 2.2 Facilitators The workshop is facilitated by a team of editors of leading engineering education journals: • European Journal of Engineering Education (published by SEFI) • Journal of Engineering Education (published by ASEE) • Australasian Journal of Engineering Education (published by AAEE) • IEEE Transactions on Education (published by IEEE) • SEFI Journal of Engineering Education Advancement (published by SEFI). 2.3 Expected learning outcomes The aim of the workshop is to provide an arena for open and transparent discussion about reviewing and the peer review process, considering the roles and perspectives of authors, editors, and reviewers. In a group activity, participants are asked to jointly share wisdoms and develop advice on: • How to support editors in making fair decisions, i.e. provide recommendations • How to support authors in improving their manuscripts, i.e. provide constructive suggestions • What aspects of a manuscript to focus on, and how to apply the review criteria • Ethical aspects of peer reviewing – “do’s and don’ts” • The ethics and implications of generative AI – how to recognise misuse • How to manage time when making reviews 3 WORKSHOP DESIGN 3.1 Time plan The workshop is in total 60 minutes, and the emphasis is on interaction in groups and in a plenary. Table 1. Time plan Run time Activity Notes 10 min Introductions Introduction to key journals in the field of engineering education, their different aims and scope, and review criteria. Introduction to the group activity. Participants are divided into groups of about four, each group facilitated by one or more journal editors. 25 min Group activity The group task is to make a virtual poster: “Advice for reviewers”. The poster consists of a slide in an online collaborate writing environment. All participants can write concurrently during the discussion, without the need for reaching full consensus. 22 min Plenary discussion Joint discussion about the results – in search for synthesis, collected wisdom and conclusions. 3 min Closing time Finally, participants are invited to sign up to receive documentation from the session, including the posters that were produced during the workshop. They are also encouraged to sign up to volunteer as reviewers for the journals. 4 WORKSHOP RESULTS The advice for reviewers produced during the workshop is here presented as one long list: • Provide constructive and actionable feedback • Be brave - it can be difficult to provide feedback on a reject decision but if you have justification then this is a valid recommendation. You can always contact the Associate Editor (AE) if you have questions, or want to discuss your thoughts. • Use the confidential comments to the AE if there are things you want to flag to the Associate Editor (to help them make their decision) but you do not want to share these specific thoughts with the author. • Think carefully about the phrasing of your feedback - be positive where you can, but don’t shy away from giving specific and clear feedback where improvements can be made. • Make your feedback and comments useful - put yourself in the shoes of the author(s). • You are providing a recommendation not a decision - there is an AE who will provide the final recommendation and an Editor-in-Chief (EIC) who will make the final decision. • Transparency and proportionality is important for the author (and understanding this by both author and reviewer is very important) - i.e. how big of an issue is the issue you have identified in the manuscript? What recommendation or “score” does this really require? • Reviewing is a great way to learn about writing papers! • You are trusted, editors and authors rely on you. Keep it in mind. • Don’t be generic. • Actionable comments. => Do not tell me x is bad, tell what precisely is bad and ideally, what would be better? • If you do not get the paper, it's not your fault, it's the authors. • Ask questions to the author if you dont understand the message. It's helping the author to understand how the audience might react. • Does the papers audience fit to the journals audience? As a reviewer you are a representative of the journal's audience, does the audience benefit from the presented answer, solution, ideas etc. (the specific audience of the journal you are reviewing!). • Find the good in the paper but be critic (no sugarcoating) • Dont sit on the fence - as a reviewer do make a decision • Answer: Does the problem, research question, methodology and results match each other? • Consider different contexts of work, communities - might be different to the reality of the reviewer. • A paper doesn't have to be fancy in structure. • Creativity is also good, don't throw it out if it's still easy to see what the contribution of the author is. • For quantitative data, look beyond the textbook, genuinely interact with the data and ask which questions you can answer based on this. • Is this a flaw that can be fixed or is it fatal? • Read the paper in detail or double check that what you asked for is included • Remember that there is an editorial pipeline • Be aware of your expertise and what you can contribute • Frame the review and be specific about what’s your school of thought/expertise • Read the criteria thoroughly • Be critical of your own review • Seek support and learning opportunities on how to review • Write constructively and not harshly • Mind your juniors, and people starting in the field • Learn from what the other reviews say(or not) • Make sure feedback is actionable - here is how it can be improved, highlight the goods, be generous in what the paper/research could be, think of it as a conversation, being interested in the topic, try to reassure the author that you have read the work • Good references (rather than fancy or own references) • Check for scope -area and style of the paper (more research vs practice papers) • Assess scholar level • See that paper has a logical flow • Degree of novelty? What do we look for? Grounded in recent literature and theory. • Strong /defendable methodology • Language - correct grammar and clear/simplified? • Be honest about the limitations of the work • Feedback should highlight the main 3 points to improve, and things perhaps that are good • Are there research questions • Arguments based in the literature and overview in the review of the existence pool of knowledge • Conceptual frameworks applied in the entire article, e.g. models, theories • Alignment among research question, methodology, findings • The methodology, arguments for the choice of methods o Descriptions of the participants o Data collection process • Results/findings in a systematic way • Discussion and conclusion answering the research question • Are there explicit limitations stated? • Conclusion with implication for the practice/or research community • References - sometimes it is necessary to search on your own. E.g. if there are only references 15-30 years old. • Be positive to new areas and please do not review the paper when you are too tired. Instead ask for another week - and keep the editor informed. • Be polite and try to be constructive - maybe you could? • Try to list the major concerns with example of what you feel is “wrong”. • If you are going to reject a paper, be clear about the significant problems that are causing you to reject it • Less aggressive wording, but more constructive • Don’t feel fully responsible for the decision of accepting/rejecting - we are all doing the best we can with the publication process, and it is the editor’s final call • Check random sources throughout the paper to ensure they are: (1) real and not AI-generated, and (2) being cited for what the source is actually discussing • Review for coherency across the entire paper - introduction sets the context, framework connects literature and theory, methods are drawn from the framework, results and discussion clearly align with the methods described, literature referenced in the discussion was already mentioned in the literature review, etc. In addition, one group wrote about the matter of AI and reviewing: • How to find AI produced content o How much of results and discussion comes from the material presented in the article o Very generic introductions/content might be a sign if using AI • Be mindful of using AI in review o Be aware of consequences of putting unpublished non-reviewed material into AI not only for ethical but also for false training of AI engines o AI is a trend and can’t be avoided, learn how to use it Two groups also wrote advice for editors: • For Editors: Appraise reviews • Would be nice to have a very brief training or guide for reviewers, other than just the journal criteria or guideline for authors • Provide support for the reviewer on how to be more constructive • Request to editors - can you provide: (1) an example of a published paper in your journal that you are proud of, and (2) an example of a paper that would be rejected with clear reviewer feedback as to why it would be rejected o What does it mean for a paper to be useful? Show us an example :) • These trainings or supports are not to teach a distinguished senior researcher how to review a paper, but to help reviewers become familiar with any particular criteria and desires for publications in that specific journal. o Rather than a reviewer having to read many papers from that journal to learn what they consider a worthy publication, there is guidance directly from the editor o This saves time on both sides! ACKNOWLEDGEMENTS The editors who facilitated the workshop wish to thank all reviewers and recognise their vital contribution to the field. REFERENCES Auby, H., Grundy, L. S., Huffman, S., Cantilina, K., Gavitte, S. B., Kaczynski, S. E., ... & Koretsky, M. D. (2024). Reflections on a mentored group peer review process. Journal of Engineering Education, 113(4), 1110-1114. Beddoes, K. (2012). Using peer reviews to examine micropolitics and disciplinary development of engineering education: a case study. Discourse: Studies in the Cultural Politics of Education, 35(2), 266–277. https://doi.org/10.1080/01596306.2012.745735 Elsevier (2025, 24 March). How to conduct a review. https://www.elsevier.com/reviewer/how-to-review Gardner, A., & Willey, K. (2018). The role of peer review in identity development for engineering education researchers. European Journal of Engineering Education, 44(3), 347–359. https://doi.org/10.1080/03043797.2018.1500526 Jensen, K., Direito, I., Polmear, M., Hattingh, T., & Klassen, M. (2021). Peer review as developmental: Exploring the ripple effects of the JEE Mentored Reviewer Program. Journal of Engineering Education, 110(1). https://doi.org/10.1002/jee.20376 Taylor & Francis (2025, 17 March). Guidelines for Peer Reviewers. https://editorresources.taylorandfrancis.com/reviewer-guidelines/ Taylor & Francis (2025, 17 March). AI Policy. https://taylorandfrancis.com/ourpolicies/ai-policy/ Wiley (2025, 24 March). Step by step guide to reviewing a manuscript https://authorservices.wiley.com/Reviewers/journal-reviewers/how-to-perform-apeer-review/step-by-step-guide-to-reviewing-a-manuscript.html.