Questionable Practices in the use of AI

AI-assisted AI-related AI-generated AI-based honest errors (unintended) misconduct & crime (intentional) Questionable practices in the use of AI 1 14.10.2025, Dr. Renat Shigapov, FDZ UB Mannheim Simon Kolstoe. Defining the Spectrum of Questionable Research Practices (QRPs), UKRIO, 2023, https://doi.org/10.37672/UKRIO.2023.02.QRPs Agenda 1. Introduction 2. Questionable practices by research stage 3. Summary 2 Introduction 3 Who defines Questionable Research Practices (QRPs)? 4 •No single global authority. •QRPs are defined collectively across international, national, and institutional levels. •There are discipline and journal specific requirements on good research practices. International frameworks 5 •ALLEA European Code of Conduct (2023) defines good research practices and violations of research integrity including research misconduct and other unacceptable practices (https://allea.org/code-of-conduct). •Singapore Statement on Research Integrity (2010) defines responsibility to report and respond to irresponsible research practices including misconduct, falsification, plagiarism, etc. (https://www.wcrif.org/guidance/singapore-statement) •The UK Research Integrity Office [UKRIO] (2023) defines QRP as minor infractions or research practices, including avoidable errors, which fall short of the definition of intentional research misconduct (https://ukrio.org/ukrioresources/publications/code-of-practice-for-research). National, institutional, disciplinary, and editorial 6 •Institutions have integrity offices, ombudspersons, and codes of good research practices which also define violation of good practices. •Funders require compliance with integrity standards. E.g., DFG has the Code of Conduct “Guidelines for Safeguarding Good Research Practice” (https://wissenschaftliche-integritaet.de/en/code-of-conduct) and „Rules of Procedure for Dealing with Scientific Misconduct“ (https://www.dfg.de/resource/blob/339200/dfg-80-01-v0524-en.pdf). •Professional societies define good and bad practices for their fields. •Journals apply COPE guidelines (Committee on Publication Ethics, https://publicationethics.org) or their own guidelines to define QRPs. Definition: Spectrum of Questionable Research Practices (QRPs) 7 Simon Kolstoe. Defining the Spectrum of Questionable Research Practices (QRPs), UKRIO, 2023 https://doi.org/10.37672/UKRIO.2023.02.QRPs •QRP is “a spectrum of behaviours, ranging from honest errors and mistakes at one end, through to more serious behaviours at the other”. •“Everyone involved in research may at times engage in QRPs, and thus it is up to everyone involved in research to recognise and address the problem in their own, as well as others’ research”. We say AI and mean AI system 8 AI system Hardware Software AI model Data AI models are trained on large amounts of data Spectrum of QRPs with AI (systems) 9 Simon Kolstoe. Defining the Spectrum of Questionable Research Practices (QRPs), UKRIO, 2023, https://doi.org/10.37672/UKRIO.2023.02.QRPs AI-assisted AI-related AI-generated AI-based honest errors (unintended) misconduct & crime (intentional) QRPs with AI don’t just repeat traditional integrity risks — they amplify them. Example: The lead author used ChatGPT to „correct“ the bibliography without notifying co-authors 16 https://retractionwatch.com/2024/05/20/journal-taking-corrective-actions-after-learning-author-used-chatgpt-to-update-references „Journal taking ‘corrective actions’ after learning author used ChatGPT to update references“ Co-author: Example: Corrigendum to „Breeding distrust: …“ 17 https://journals.sagepub.com/doi/10.1177/25148486241258320 Example: Fake citations using fake papers 18 Ibrahim, H., Liu, F., Zaki, Y. et al. Citation manipulation through citation mills and pre-print servers. Sci Rep 15, 5480 (2025). https://doi.org/10.1038/s41598-025-88709-7 •Citation pollution •Research pollution •As well as problems for AI-assisted literature reviews Research lifecycle Plan & Design stage: •Study design •Writing a data management plan •Writing a project proposal 19 Overview of QRPs in Plan & Design stage QRPs: •AI-written study design without disclosure and/or manual checks. •AI-written data management plan without disclosure and/or manual checks. •AI-written project proposal without disclosure and/or manual checks. Problems: •AI plagiarism. => Misconduct. •Over-reliance on AI. How to improve? •If you use AI-written texts, verify and edit them. You are the author, not AI. •Disclose substantial AI assistance. •Check the funder requirements. 20 AI misuse in grant applications: Worldwide 21 https://doi.org/10.1126/science.zq9eef8 https://euraxess.ec.europa.eu/worldwide/asean/news/european-research-council-issues-warning-ais-use-grant-applications AI misuse in grant applications: DFG Guidelines 22 https://www.dfg.de/resource/blob/167398/10-20-en.pdf Research lifecycle Collect & Create stage: •GenAI-generated figures and visuals •Data fabrication with genAI •Synthetic data without disclosure 23 Overview of QRPs in Collect & Create stage QRPs: •GenAI-generated images and visuals are not permitted by some journals. •AI-fabricated observations and AI-manipulated raw data. •Synthetic data mixed with real data without disclosure. Problems: •AI fabrication. => Misconduct. •Usage of copyright-protected materials. How to improve? •Check the rules. Follow the rules. •Disclose substantial AI assistance. •Be transparent. 24 Springer Nature policy on genAI images 25 https://www.nature.com/nature-portfolio/editorial-policies/ai Fundamental tricks in machine learning: Misreporting 32 “Misreporting is any error or misleading presentation of the model’s specification or evaluation results.” Leech, G., Vazquez, J. J., Kupper, N., Yagudin, M., & Aitchison, L. "Questionable practices in machine learning." arXiv preprint, 2024, https://doi.org/10.48550/arXiv.2407.12220 Research lifecycle Evaluate & Archive stage: •AI-written peer reviews •AI-assisted manipulation of peer review process •Failure to archive replication packages 33 Overview of QRPs in Evaluate & Archive stage QRPs: •AI-written peer reviews. •AI-assisted manipulation of peer review process. •Failure to archive replication packages (with codes, data, models, and prompts). Problems: •Biased evaluation. Confidentiality violation. Compromised research integrity. •Fake reviewer identities. Fabricated reviews. Replication crisis. How to improve? •Avoid sharing confidential or unpublished manuscripts with third-party tools. •Disclose substantial AI assistance. Use AI responsibly. •Check the rules. Follow the rules. Ordnung muss sein. 34 Prompt injection in peer review process: Author 35 Gibney, Elizabeth. "Scientists hide messages in papers to game AI peer review." Nature 643.8073 (2025): 887-888. https://doi.org/10.1038/d41586-025-02172-y 👎 AI misuse in peer review process by a reviewer 36 Scenario: •Reviewer pastes confidential manuscript or grant text into a cloud-based AI system to “summarize” or “evaluate.” •The AI tool processes sensitive, unpublished content. Risks: •Confidentiality breach (manuscript not meant to be shared externally). •Intellectual property leakage (ideas, data, and methods are exposed). •Bias or inaccuracy if reviewer relies uncritically on AI’s output. •Violation of review ethics/guidelines and trust in the peer review process. DFG: General Guidelines for Reviews https://www.dfg.de/resource/blob/167398/10-20-en.pdf Failure to archive replication package Replication package: •should contain codes, data, AI models, prompts, documentation, license, and data (and code) management plan, •be archived in institutional, general-purpose or disciplinespecific data repository. 37 Research lifecycle Share & Disseminate stage: •AI-written papers •AI plagiarism •Undisclosed writing assistance 38 Overview of QRPs in Share & Disseminate stage QRPs: •AI-written papers. With or without listing AI as co-author (AI is not an author!). •AI-giarism, AI-paraphrase-plagiarism, etc. •Undisclosed writing assistance. Problems: •Undermines research integrity, originality, and accountability. •AI is not an author. Authorship and accountability rest solely with humans. How to improve? •Treat AI-generated text as potentially derivative. •Disclose substantial AI assistance. Specify how the tool was used. •Use AI responsibly. 39 What do „Science“ and „Nature“ say? 40 https://doi.org/10.1038/d41586-025-03046-z https://doi.org/10.1126/science.zxxd90o https://doi.org/10.1126/science.z87syeh AI-giarism is plagiarism using AI 41 A human enters a prompt into an AI system, copies the generated response into a paper, and submits it as their own for peer review. The human acknowledges the use of AI tools. A human enters a prompt into an AI system, copies the generated response into a paper, and submits it as their own for peer review. The human does not acknowledge the use of AI tools. Undisclosed AI UseDisclosed AI Use References and the next talk 1. S. Kolstoe. Defining the Spectrum of Questionable Research Practices (QRPs), UKRIO, 2023, https://doi.org/10.37672/UKRIO.2023.02.QRPs. 2. Leech, G., Vazquez, J. J., Kupper, N., Yagudin, M., & Aitchison, L. "Questionable practices in machine learning." arXiv preprint, 2024, https://doi.org/10.48550/arXiv.2407.12220. 3. J. Woodhams with contributions from K. Dally, S. Neave, J. Parry, and J. Scott, 2025, „Embracing AI with integrity“, report of UKRIO, https://doi.org/10.37672/ukrio.2025.06.embracingaiwithintegrity. 4. „The Concordat to Support Research Integrity“, https://ukcori.org/research-integrity-concordat. Register to our talk on 22.10.2025 „Research Data Management Seminars: Responsible use of AI“ https://www.bwl.uni-mannheim.de/en/details/research-data-management-seminars-responsible-use-of-ai-inresearch 48