Attitudes and perceptions towards the use of artificial intelligence chatbots in medical journal peer review: A protocol for a large-scale, international cross-sectional survey

This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0). European Science Editing / ese Citation Ng JY, Bhavsar D, Dhanvanthry N, etal. Attitudes and perceptions towards the use of artificial intelligence chatbots in medical journal peer review: A protocol for a large-scale, international cross-sectional survey. Eur Sci Ed. 2025;51:e159921. https://doi.org/10.3897/ese.2025.e159921 Received 26 May 2025 Accepted: 8 Aug 2025 Published: 10 Oct 2025 Original Article Attitudes and perceptions towards Attitudes and perceptions towards the use of artificial intelligence the use of artificial intelligence chatbots in medical journal peer chatbots in medical journal peer review: A protocol for a large-scale, review: A protocol for a large-scale, international cross-sectional surveyinternational cross-sectional survey JeremyY. Ng1,2,3,4, Daivat Bhavsar1,2, Neha Dhanvanthry1,2, Lex Bouter5,6, Teresa Chan7, Annette Flanagin8, Alfonso Iorio3,9, Cynthia Lokker3, Hervé Maisonneuve10,11, Ana Marušić12, David Moher13,14,15, Holger Cramer1,2 1Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany [email protected]orjeremy[email protected] orcid.org/0000-0003-0031-5873 orcid.org/0000-0003-3682-918X orcid.org/0009-0002-9002-151X orcid.org/0000-0002-3640-8046 2Robert Bosch Center for Integrative Medicine and Health, Bosch Health Campus, Stuttgart, Germany 3Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Canada orcid.org/0000-0002-3331-8766 orcid.org/0000-0003-2436-4290 4School of Public Health, Faculty of Health, University of Technology Sydney, Sydney, Australia 5Department of Epidemiology and Data Science, Amsterdam Universities Medical Center, Amsterdam, The Netherlands orcid.org/0000-0002-2659-5482 Ng etal. Attitudes and perceptions towards the use of artificial intelligence X XX doi.org/10.3897/ese.2025.e159921 Data Availability Statement The data that support the findings of this study are available on request from the corresponding author. Author contributions Concept – J.Y.N.; Design – J.Y.N., L.B., T.C., A.F., A.I., C.L., H.M., A.M., D.M., H.C.; Supervision– J.Y.N., H.C.; Resource H.C.; Materials – H.C.; Literature Search – J.Y.N., D.B., N.D.; Writing – J.Y.N., D.B., N.D.; Critical Reviews – J.Y.N., D.B., N.D., L.B., T.C., A.F., A.I., C.L., H.M., A.M., D.M., H.C. Declaration of Interests Hervé Maisonneuve, Ana Marušić, and Jeremy Y. Ng are members of the international advisory board of European Science Editing but had no role in the editorial decision-making process for this manuscript. All other authors declare that they have no competing interests. Funding The authors declared that this study received no financial support. Ng etal. / doi.org/10.3897/ese.2025.e159921 Page 2 / 12 European Science Editing / ese 6Department of Philosophy, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands 7School of Medicine, Toronto Metropolitan University, Toronto, Ontario, Canada orcid.org/0000-0001-6104-462X 8JAMA and the JAMA Network, Chicago, Illinois, USA orcid.org/0000-0001-9114-6221 9Department of Medicine, McMaster University, Hamilton, Ontario, Canada 10Consultant, Lyon, France orcid.org/0000-0001-8365-7558 11Scientific Committee, Institute of Research and Action on Fraud and Plagiarism in Academia, Geneva, Switzerland 12Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, University of Split School of Medicine, Split, Croatia orcid.org/0000-0001-6272-0917 13Centre for Journalology, Methodological and Implementation Research Program, Ottawa Hospital Research Institute. Ottawa, Canada orcid.org/0000-0003-2434-4206 14School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada 15Institute of Health Policy, Management & Evaluation, Dalla Lana School of Public Health, University of Toronto Ng etal. / doi.org/10.3897/ese.2025.e159921 Page 3 / 12 European Science Editing / ese Keywords: artificial intelligence, attitudes, chatbots, generative artificial intelligence, peer review, reviewers, survey Abstract Background: Artificial intelligence (AI) chatbots are advanced conversational programmes capable of performing tasks such as identifying methodological flaws, verifying references, and improving language clarity in manuscripts. Their use in peer review has the potential to enhance efficiency, reduce reviewer workload, and address inconsistencies in review quality. However, concerns remain regarding their reliability, ethical implications, and transparency in decision-making, and little is known about how peer reviewers perceive these tools. Objectives: To assess peer reviewers’ attitudes and perceptions towards the use of AI chatbots in the peer review process, including their familiarity with AI, perceived benefits and challenges, ethical considerations, and expectations for future roles. Methods: An international cross-sectional survey will be conducted among academic peer reviewers. The survey will collect data on participants’ prior experience with AI, perceptions of the utility of chatbots in supporting peer review, concerns related to ethics and transparency, and anticipated future applications. Results: This study will report descriptive and comparative analyses of reviewers’ responses, highlighting patterns in attitudes and perceptions by demographic and professional characteristics. Conclusions: The findings may offer evidence to inform the development of future policies and best practices for the ethical and effective integration of AI chatbots in peer review, with the goal of improving review quality while addressing potential risks. Attitudes and perceptions towards the use of artificial intelligence Ng etal. / doi.org/10.3897/ese.2025.e159921 Page 4 / 12 Background Artificial intelligence (AI) broadly refers to the capability of computer systems or computercontrolled robots to perform tasks typically associated with human intelligence, such as reasoning, problem-solving, generalizing, and learning from experience.1,2 Although current AI programmes lack the versatility of human intelligence, specialized applications have permeated numerous fields, including self-driving cars, speech transcription, healthcare, and education.2 In many domains, AI has demonstrated benefits such as increased productivity, fewer errors, and cost savings. For example, in medicine, AI systems can improve diagnostic accuracy, optimize treatment plans, and reduce healthcare costs of many healthcare systems.3,4 These hypothetical benefits have sparked interest in applying AI to scholarly publishing, including the peer review process, where efficiency and quality improvements are very much needed.5 AI chatbots, a subset of AI programmes, are generative AI tools designed to simulate human conversation through text or speech.6 AI chatbots have demonstrated versatility in applications ranging from customer service to education.6,7 Within scholarly publishing, AI chatbots hold promise for tasks such as improving language clarity, identifying methodological flaws, verifying references, and standardising review quality.8 The integration of chatbots into peer review workflows could lower reviewer workloads, streamline processes, and address long-standing issues such as bias in and inconsistency between review reports.8 Moreover, AI chatbots may address reviewer fatigue and help peer reviewers cope with the increasing volume of requests, ensuring more timely and consistent evaluations.9,10 Despite these potential advantages, the use of AI chatbots in peer review faces challenges. Major concerns include the reliability of AI chatbots in evaluating complex scientific content, the risk of overlooking critical nuances, the potential for manipulation of the peer review process (such as by injecting prompts), and ethical considerations related to the use of AI.11 For example, well-documented limitations of ChatGPT include generating plausible but incorrect or misleading content, inaccurate citations, and references to non-existent sources.11-13 Additionally, as AI chatbots rely on training data sets that may lack currency or inclusivity, there is a risk of perpetuating outdated information and biases.14 Ethical concerns also extend to issues of plagiarism, confidentiality, intellectual property rights, transparency, research integrity, and accountability.8 While the use of AI chatbots in peer review may be acceptable under open peer review models, where the author’s consent to broader sharing of manuscript content is typically implied, this practice raises ethical and procedural concerns for the peer review of manuscripts received by journals that follow closed (or confidential) peer review models. In such cases, sharing unpublished manuscripts with AI chatbots, particularly those hosted on proprietary platforms that store user input, could compromise reviewer confidentiality and violate journal policies.15 Furthermore, for certain medical articles, reviewers may have access to sensitive patient information that is not intended for publication. Sharing this information with AI systems may constitute a breach of patient privacy and, in some jurisdictions such as the United States, could potentially violate the Health Insurance Portability and Accountability Act.16,17 Furthermore, questions remain about the levels of responsibility peer reviewers should be allowed to delegate to AI chatbots and whether such tools could inadvertently Attitudes and perceptions towards the use of artificial intelligence Ng etal. / doi.org/10.3897/ese.2025.e159921 Page 5 / 12 influence decision-making or perpetuate systemic biases.12 Additionally, of the peer review work delegated to AI chatbots, it is unclear to what extent reviewers can rely on AI chatbots to be accurate, as AI chatbots may misinterpret information and provide incorrect conclusions.8,12 The scholarly publishing community has started to address these issues through policies for editors and peer reviewers. Academic publishers, such as Springer Nature, JAMA Network, and PLOS, have incorporated guidelines on the use of AI chatbots in their editorial processes, requesting that peer reviewers refrain from uploading any manuscript information to AI chatbots and that any use of AI for peer review (that is, evaluation of the claims made in the manuscript) must be disclosed in review forms.18-20 Sage has also mandated that reviewers should not use AI chatbots to create review reports because doing so may result in breaches of confidentiality and copyright concerns.21 Similarly, Science has stated that AI chatbots cannot be used for review purposes, because reviewers are required to write their review reports independently and should not seek external inputs without the permission of the editor.22 Other organizations, including the World Association of Medical Editors and International Committee of Medical Journal Editors, have recommended that the use of AI chatbots should be prohibited in cases where confidentiality cannot be guaranteed, and if reviewers do use AI chatbots, they must have permission from the journal with clear disclosure of how the chatbots were used.23,24 However, codifying and regulating the responsible use or the lack thereof by peer reviewers may become increasingly difficult with the rapidly evolving roles and popularity of AI chatbots. Furthermore, despite growing interest in the role of AI chatbots in peer review, there is little research on the attitudes and perceptions of peer reviewers towards the use of such tools.25 For example, a recent 6-question poll of 5229 manuscript authors conducted by Nature in March 2025 inquired about their general attitudes towards the use of AI in the publication process, including peer review: the majority (60% of respondents) stated that it was ‘not appropriate’ to use AI for initial peer review reports.25 Peer review is essential for maintaining the integrity of published research, and as interest in using AI within scholarly publishing continues to rise, it is crucial to understand how peer reviewers perceive and engage with AI chatbots, along with the foreseeable impacts on the peer review process and publishing overall. This understanding will help leverage the potential of AI chatbots while mitigating their limitations, ultimately enhancing the quality, transparency, and fairness of the peer review process. To address this gap, this article proposes an international cross-sectional survey to assess peer reviewers’ experience with using AI chatbots, perceived benefits and challenges, ethical concerns, and anticipated roles for AI chatbots in the peer review process. By providing insights into peer reviewers’ perspectives, such a study can inform the development of ethical guidelines, practical recommendations, and evaluations needed for the effective integration of AI chatbots in peer review. Methods Open science statement A complete study protocol, with the data analysis plan, has been registered on the Open Science Framework (OSF) (https:// doi.org/ 10.17605 /OSF.IO/ FHC2M) before Attitudes and perceptions towards the use of artificial intelligence Ng etal. / doi.org/10.3897/ese.2025.e159921 Page 6 / 12 recruiting participants for the proposed survey. The study materials and data will be made available via OSF as they become available, and the final manuscript will be posted as a preprint prior to submission to a peer reviewed journal. Research ethics approval Ethics approval was obtained from the University Hospital Tübingen Research Ethics Board (REB Number: 080/2025BO2) to conduct this study. Study design We will conduct an anonymous, crosssectional, and closed survey of peer reviewers for medical journals to investigate their attitudes and perceptions towards the use of AI chatbots in the peer review process. The survey will be administered online using SurveyMonkey,26 a secure web-based survey tool. The survey will include both closedended questions (for example, multiple choice, yes/no) and open-ended questions (for example, free-text responses), addressing the following topics. • Demographic information: Age, sex, country of employment, level of education, primary area of expertise, publication record, and years of experience as a peer reviewer. • Experience with AI chatbots: Familiarity with AI chatbots, prior use of AI chatbots in their professional or academic work, and likelihood of permitted AI chatbot use in the peer review process in the future. • Role of AI chatbots in peer review: Perceptions regarding the potential roles of AI chatbots in peer review, such as aiding in identifying methodological flaws, detecting plagiarism, verifying references, translating research materials, or assessing the quality of writing. • Perceived benefits of AI chatbots in peer review: Views on potential benefits, such as reducing workload, improving efficiency, standardizing review quality, addressing biases, and ensuring greater consistency in decision-making. • Perceived challenges of AI chatbots in peer review: Concerns about the reliability and accuracy of AI chatbots, issues with authorship of peer review reports, risk of amplifying biases, lack of transparency, and potential over-reliance on AI. • Ethical considerations: Perceptions of the ethical implications of integrating AI chatbots in peer review, including concerns about accountability, confidentiality, data privacy, intellectual property rights, and their potential impact on the integrity of the peer review process. • Additional comments and feedback: Through an open-ended question, participants will be given the opportunity to provide additional comments and feedback on the use of AI chatbots in peer review, share opinions on future integration, and suggest potential areas for improvement or guidelines for ethical use. The survey will be piloted with a small group of peer reviewers, who will be excluded from the final survey, to ensure clarity, relevance, and comprehensiveness of the questions. Suggestions for potential peer reviewers will be solicited from the co-authors of this study. Feedback from the pilot will be incorporated into the final survey design. Sampling framework A comprehensive list of all journals indexed in MEDLINE (approximately 5300 as of November 2024) will be compiled along with their unique identifiers assigned by the US National Library of Medicine (NLM IDs).27 A search strategy using these NLM IDs will be formulated in Ovid MEDLINE (a platform to search biomedical literature), limiting the search to records indexed within Attitudes and perceptions towards the use of artificial intelligence Ng etal. / doi.org/10.3897/ese.2025.e159921 Page 7 / 12 two months preceding the search. This time frame was selected because reviewers who have submitted a peer review report during this period are likely to be still actively engaged in research and available to respond to the invitation to participate in the survey. The ‘corresponding authors’ of all types of research articles will be considered for inclusion. Any duplicate records will be removed before the recruitment process. All PMID numbers – Ovid MEDLINE uses PMID, short for PubMed ID, numbers to uniquely identify citations within its database – corresponding to the identified articles will be exported from Ovid as a .csv file, and this file will be processed using an R script (based on the easyPubMed package) to extract author names, affiliated institutions, and email addresses.28 Additionally, the ‘Find Full Text’ function in EndNote will be used to retrieve the articles in PDF. These files in PDF will then be processed with another R script for text recognition to extract email addresses. All resulting data will be compiled into a master list, which will be thoroughly checked for errors or duplicates before distributing the survey. The search strategy can be found here: https:// osf. io/n cajv. Inclusion criteria To be eligible for participation, participants must have previously served as peer reviewers of research articles submitted to medical journals (of any kind, whereby the research they have reviewed contributes to the field of medicine). Eligible participants must have completed and submitted at least one peer review report to at least one professionallevel medical journal (MEDLINE-indexed) within the past 24 months. Those who have peer reviewed exclusively for student journals (for example, high school, undergraduate, or graduate journals) will not be eligible to participate. Recruitment of participants Prospective participants will be recruited from various academic disciplines within the medical field. We will use convenience sampling to recruit participants, targeting medical researchers identified through our sampling framework. An email will be sent to potential participants containing a recruitment message, approved by the University Hospital Tübingen research ethics board, outlining the purpose of the study and a link to the survey. Upon clicking the link, participants will be directed to a web page with an informed consent form. Participants must indicate consent on the form before proceeding with the survey. This will be a closed survey, meaning only invited participants will be able to participate. If participants do not respond to the initial invitation email, reminder emails will be sent after every 2 weeks. The third reminder will be followed by a 4-week waiting period to accommodate any remaining interested participants before the survey is closed permanently. Participants will have a total of 8 weeks to complete the survey. The survey will be administered online via SurveyMonkey. There will be no monetary compensation offered to participants, and participation will be completely voluntary. Participants can skip questions they do not wish to answer and can withdraw from the survey at any time by simply closing the browser window. Participation will be anonymous and confidential throughout the study; however, settings will allow tracking the respondents for purposes of follow-up reminders and limiting the response to only one from each participant. Attitudes and perceptions towards the use of artificial intelligence Ng etal. / doi.org/10.3897/ese.2025.e159921 Page 8 / 12 Sample size Given that our initial list of names and email addresses is likely to contain duplicates, nonfunctioning emails, and other inconsistencies, we estimate that approximately 40,000 corresponding authors will be contacted after removing duplicates. For articles with two or more corresponding authors, each will be invited. This estimate was based on the following assumptions: approximately 5300 journals will be selected, and we will retrieve the PMID numbers for articles published within these journals over the 2 months preceding the search. At least one PMID number per journal per month will be retrieved, based on earlier studies using this sampling framework, leading to a total of 120,000 PMIDs from MEDLINE. After processing 120,000 PMIDs, we will have more than 70,000 unique names and email addresses. Based on past surveys conducted,29,30 we anticipate a response rate of 3%–5%. Survey instrument A complete copy of the survey to be used for piloting can be found here: https:// osf. io/c 748y. The survey will be created, distributed, and collected using SurveyMonkey, a secure online survey tool (https:// www.surv eymonkey .com/). The survey instrument was developed through a review of the literature and input from experts in AI and scientific research. All authors of the study protocol reviewed the survey draft and will review the edited version prior to it being circulated. The survey begins with a screening question to confirm that participants identify themselves as medical researchers who have submitted a peer review report within the past 24 months to a MEDLINE-indexed journal. Following this, there are eight demographic questions, asking participants about their current position, research area, sex, age, and country of employment. Participants will then answer another eight questions about their familiarity with and experience of using AI chatbots, followed by eight more questions about the views of the participants on the role of AI chatbots in the peer review process. Subsequently, participants will answer four more questions about the perceived benefits and challenges of using AI chatbots in peer review. The survey concludes with an open-ended question, allowing participants to provide additional feedback or thoughts about the use of AI chatbots in peer review. In total, the survey includes 30 questions and is expected to take about 15 minutes to complete. Data management and analysis All responses will be collected via SurveyMonkey, and the data will be exported for analysis using Microsoft Excel. Descriptive statistics, such as frequencies and percentages, will be calculated to summarise the survey responses. Demographic information, such as participant role and years of experience, will also be analyzed to provide insights into any trends or patterns using inferential statistics. To explore potential differences in attitudes and perceptions between different subgroups (for example, senior versus early-career peer reviewers), cross tabs may be created for key demographic variables provided the sample size is adequate. Qualitative data collected through open-ended questions will undergo inductive coding and thematic analysis by two authors31,32 To ensure consistency in coding, a pilot coding process will be conducted, with two authors independently coding the first three survey responses. Afterwards, the authors will collaborate to develop a unified coding framework. Once a consensus on the codes is reached, all responses will be grouped into thematic categories, which will be clearly Attitudes and perceptions towards the use of artificial intelligence Ng etal. / doi.org/10.3897/ese.2025.e159921 Page 9 / 12 defined and described. The results will be reported in the form of tables, with distinct themes and illustrative quotes where relevant. Ethical considerations Ethics approval was obtained from the University Hospital Tübingen Research Ethics Board (REB Number: 080/2025BO2) to conduct this study. Participation in the survey will be voluntary, and participants will have the right to withdraw from the study at any point before submitting their completed survey. All data collected will be kept confidential and anonymous, and no identifying information will be collected. Therefore, once the survey is submitted, participants will not be able to withdraw from the study as their responses will be collected without any personal identifiers, including IP addresses. Discussion The main purpose of this study is to gather the attitudes and perceptions of peer reviewers of medical journals regarding the use of AI chatbots in the peer review process, and the potential impact of such use on how peer review is conducted at a time when these AI chatbots are being increasingly deployed in scholarly publishing. AI chatbots can automate time-consuming tasks in the peer review process, provide writing and editing assistance, and potentially improve the efficiency and consistency of reviews. They may also help lower reviewer workload, enhance the overall quality of feedback provided to authors, and allow human reviewers more time for evaluating the novelty, importance, and quality of research and to provide insights that AI chatbots cannot provide. However, the use of AI chatbots in peer review raises significant concerns, including ethical issues, risks of bias, risk of manipulation of the peer review process, and challenges related to the accuracy and transparency of automated reviews. These are potential benefits and risks of AI chatbots that require validation with further data and practice in the research process. To guide the responsible integration of AI chatbots in peer review, a clear ethical framework is necessary. Understanding peer reviewers’ perceptions and concerns about AI chatbots is crucial to shaping policies and practices that address these challenges. The results of this survey could inform the development of guidelines by academic journals or publishers on the responsible use of AI chatbots in the peer review process, balancing their potential benefits with the need for rigorous, unbiased, and transparent review practices. Strengths and limitations This proposed study uses a cross-sectional survey design, which has several strengths as well as limitations. The key strengths are that the approach is cost-effective and relatively quick to administer, and allows us to gather data from a large sample of peer reviewers for medical journals, making the results generalizable to the broader community of medical researchers. As the researchers in our sample are likely to represent diverse medical disciplines, we expect a broad range of opinions regarding the use of AI chatbots in the peer review process, providing valuable insights. Additionally, by collecting names and email addresses only from the past two months, we minimize the likelihood of encountering inactive or bounced emails, which helps ensure the accuracy of our contact list. As to the limitations of the design, there is the potential for recall bias, common to