Building better research: Changing research systems by introducing graduate incentives for openness, reusability and reproducibility

Building betterresearch: Changingresearchsystems by introducing graduateincentives for openness, reusabilityand reproducibility Tracey L. Weissgerber, PhD EXCELScIOR ERA Chair team leader Invited Research Coordinator University of Coimbra [email protected] Tracey Weissgerber ReproducibiliTeach, Meta-research Methods If we wantto improve researchcultureand practice on a largescale, we must focuson changing systems ratherthanindividuals. We needcoordinatedactionamongstakeholders to testand implementevidence-based interventions: •Researchers •Researchinstitution& departments •Funders •Publishers& editors An example: How can researchers and organizations create a culture of methods sharing? Promoting Reusable and Open Methods and Protocols Objective: Improve the clarity of methods and reusable step-by-step protocols in life sciences preprints and papers Approach: Recommends actions for 4 stakeholder groups •Researchers •Research institutions & departments •Funders •Publishers & editors Full report Brief overview for researchers Eachstakeholder canuse differentleversto facilitatewidespreadchangein practice(e.g. policy change, adjustingCV formats, changingevaluationcriteria) Leverscanbe appliedto promoteuptakeof many practicesor outputs (e.g. pre-registration, open protocols, open data) Strategy 2: Require reproducible research and open science practices in theses https://elifesciences.org/articles/89736 TREASURE Transforming Research Assessment through Responsible Research training of Early-Stage Researchers: An institutional pilot project treasur[email protected] www.uc.pt/iii/treasure Inês Almeida Junior researcher Thesis Research article WHY WE NEED TO CHANGE INCENTIVES & REWARDS Thesis Research article STUDY DESIGN STEP-BY-STEP PROTOCOL SOFTWARE & CODE WHY WE NEED TO CHANGE INCENTIVES & REWARDS DATA MATERIALS REWARDED PRACTICES AND OUTPUTS PRACTICES •Author contribution statements •Unique Identifiers for research materials/resources •Stakeholder engagement, citizen science or participatory research •Science communication, dissemination, policy recommendations [activities, materials, policy briefs] •Using evidence synthesis techniques to assess the quality of evidence and identify research gaps •Reporting of null results •Reporting guidelines •Preregistered reproducibility or replication studies [email protected] https://www.uc.pt/iii/treasure/ PROJECT TEAM So, what are some of the research practices we are training, supporting, and rewarding through TREASURE? Pre-registration https://help.osf.io/article/158-create-a-preregistration#Go-to-the-OSF-Prereg-Challenge-landing-page What: Deposit a time-stamped study protocol on a public repository before you start collecting data (can be public or embargoed). Why: Protects against outcome switching, p-hacking and other questionable research practices. If you change your design, just explain the modifications in the paper. Registered report https://www.cos.io/initiatives/registered-reports What: The journal reviews your introduction and methods before you start the study, requests revisions, and may decide to provisionally accept the paper as long as you follow the specified methods to complete the study. Why: Get reviewer feedback on the study design to fix problems before collecting any data. The decision to publish the study is based on the quality of the design, not whether the results are surprising or interesting. NOTE: some sports science journals offer the Registered Report format: Journal of Experimental Physiology; Human Movement Science; Science and Medicine in Football; Psychology of Sport and Exercise; Reports in Sport and Exercise and Journal of Sports Sciences What Preregistered replication or reproduciblity studies https://doi.org/10.5281/zenodo.15785597 Why Replication study: •Researchers attempt to confirm the results of previously published studies by repeating a previously published study Reproducibility study •Using data and, if available, code from a previous study to determine whether they can reproduce the results. •Findings from many previous studies have been shown not to reproduce or replicate. Knowing which findings do, and don’t, replicate, and under what conditions, is essential for researchers and other society stakeholders •Conducting a reproducibility or replication study helps early career researchers to learn how to avoid problems by improving the conduct and reporting of their own studies to improve reproducibility and replicability. •Can provide important information about why studies aren’t reproducible or don’t replicate, which can lead to new insights and research directions. Data sharing (make it FAIR, open as possible) What: Share data (and metadata) on a public repository in a Findable, Accessible, Interoperable and Reusable (FAIR) way. This may be open access or, for sensitive data, restricted access. Why: Allows others to reproduce findings, reuse data for evidence synthesis or to answer new research questions; Access to high-quality (meta)data allows others to find relevant studies and datasets, even when the data is not fully open, fostering collaborations; Students learn robust data management and curation skills, including metadata standards and data sharing protocols, which are increasingly valuable in different environments NOTE: data sharing rate of less than 5% in sports science (Borg et al., 2020) https://www.data.cam.ac.uk/data-management-guide/sharing-your-data; https://guides.hshsl.umaryland.edu/data; Figure: NAU MOOC O Essencial da Gestão de Dados de Investigação; codata.org WHAT IS DATA? Unique identifiers for research materials and resources What: Unique persistent identifiers are ID numbers or codes assigned to a research material to help researchers identify exactly what was used (version, laboratory, etc). Examples include Research Resource Identifiers (RRIDs) for cell lines, antibodies, model organisms, plasmids, software and tools and core labs, or CAS numbers for chemicals. Why: It improves transparency of research methods and increases reproducibility of methods and results. Some unique persistent identifier systems offer warnings about problematic materials. Allow researchers to create new persistent identifiers for resources that they create, adding them to their CV / ORCiD. https://www.rrids.org/; https://www.cas.org/cas-data/cas-registry; https://www.uniprot.org/ Author contribution statements What: Include a statement that specifies the tasks that each author completed during the research. This statement may follow the CRediT taxonomy. Why: Provides documentation of experience; can be especially useful for early career researchers Contributor Roles Taxonomy https://credit.niso.org Preprints What: Research papers are posted on a public server prior to peer review. Authors can post updated versions as they improve and refine the paper. Why: Authors can get feedback before the work is published, ability to list work in funding applications (particularly valuable for early stage researchers), scooping protection https://github.com/OpenScienceMOOC/Module-6-Open-Access-to-Research-Papers/issues/25; https://www.nlm.nih.gov/oet/ed/pmc/preprints/01-100.html Assessingimpact •Does thisprogramchangeuptakeof practices andoutputs in graduatestudentresearch? •Cangraduatestudentsbe a trojan horseto encouragewidespreadadoptionof practices withinresearchgroups? Challenge Whichpracticesand outputs canyouimplementin yourthesisresearch? Howcouldyouencourageothersto implement some of thesepracticesand outputs? Show outputs beyondpublicationson yourCV Questions? Comments? Emotional outbursts…? Tracey Weissgerber ReproducibiliTeach, Meta-research Methods [email protected]