Research Data Management Seminars: Replication

Replication: "I packed my replication package and in it I put …" Research Data Management Seminars: Replication Dr. Phil Kolbe 1 Agenda •Reproducibility and replicability •Reasons and incentives for reproducibility and replicability I •Requirements for reproducibility and replicability •Reasons and incentives for reproducibility and replicability II •Replication package Research Data Management Seminars: Replication 2 Reproducibility and replicability 3 Matrix defining reproducible research (from The Turing Way) Reasons and incentives for reproducibility and replicability I •Replication crisis •Transparency and trust in science and research •Error Detection and Debugging •“Standing on the shoulders of Giants” •Professional Reputation •Fosters cooperation/collaboration •And more later… Research Data Management Seminars: Replication 4 Reasons and incentives I: Replication crisis •Many published scientific findings cannot be replicated or reproduced •Wide variety of fields affected •Causes: P-Hacking, Publication Bias, Low Statistical Power, Lack of Transparency •Consequences: Erosion of trust, wasted resources and barriers to scientific progress •Solutions: open data, pre-registration of studies, replication studies and higher standards for publication Research Data Management Seminars: Replication 5 Reasons and incentives I: “Standing on the shoulders of Giants” •Highlights how scientific progress builds on prior discoveries •Reliable, replicable research allows scientists to trust, check and build upon the findings of others •With validated results, researchers can focus on extending knowledge rather than redoing past work •Reduces duplication of effort and redirects resources to new, innovative research rather than revisiting unreliable studies •Otherwise, you're just standing on a pile of wastepaper Research Data Management Seminars: Replication 6 Requirements for reproducibility and replicability What do we need for reproducibility and replicability? Key requirements: •Data, Data Access and Documentation •Folder and File Structure •Analysis Documentation •Publication of code and data Research Data Management Seminars: Replication 7 Requirements for reproducibility and replicability: data, data access and documentation Data access description: •Where is the data located? → DOI, website, RDC •Who created the data? → research institution, public authority, company •Who can access the data? → public use file, scientific use file, confidential data •How can the data be accessed? → open access, registration, application for use, costs Research Data Management Seminars: Replication 8 Requirements for reproducibility and replicability: data, data access and documentation 9 ID age Q1 Q2 … 1 18 2 - … 2 20 - 999 1,4 3 19 0 2,4 4 19 xxx 3,6 5 199 5 2.3 6 20 1 7 19 3 2,1 Do you understand this data set, and can you use it? What problems could there be? Requirements for reproducibility and replicability: Publication of code and data •only published manuscripts, program codes and data are accessible to third parties and ensure transparent and verifiable research •typical non-subject-specific options: –https://zenodo.org/: Per record limitation (50GB / 100 files), with 200 GB exception for one record –https://osf.io/: capacity of private projects and components utilizing OSF Storage to 5 GB and public projects and components to 50 GB –https://dataverse.harvard.edu/: deposit files of up to 2.5GB, and store up to 1TB of data •check for subject-specific options or use the institutional repository MADATA Research Data Management Seminars: Replication 16 Requirements for reproducibility and replicability: Publication of code and data •File formats for sharing → Non-proprietary, open standard, uncompressed, unencrypted 17 Plain text txt doc Tabular/structured data csv, xml xls, dta, sav Images tif, tiff gif Video mp4 wmv Formatted text PDF/A, docx, xml doc, ppt, PDF Audio wav, mp3 Overview: •https://kostceco.ch/cms/kad_main_de. html •https://www.lzv.nrw/dateif ormate/ Requirements for reproducibility and replicability: FAIR Data Research Data Management Seminars: Replication 18 GitBook Bot, Lambert Heller, datawomanHUB, mcancellieri, Bianca Kramer, Tony Ross-Hellauer, ilabastida, helenebr, Pedro Fernandes, & Jon Tennant. (2018). Open Science Training Handbook (1.1). Zenodo. https://doi.org/10.5281/zenodo.1212538 Replication package 19 Matrix defining reproducible research (from The Turing Way) Replication package Reasons and incentives for reproducibility and replicability II: more reasons and incentives! Policies of research institutions: Example: Code of Good Research Practice, University of Mannheim (2023) •“Studies must be replicable. Therefore, publications must include a complete and detailed description of the methods of data collection, the data and software used, the statistic analysis and the results in order to allow for verification of the results through replication.” Research Data Management Seminars: Replication 20 Source: https://www.uni-mannheim.de/en/research/our-research/good-research-practice/ Reasons and incentives for reproducibility and replicability II: more reasons and incentives! Policies of journals •More and more journals and publishers are requesting that raw data and code be made available when the article is published Research Data Management Seminars: Replication 21 Source: Vlaeminck (2021, p. 11) Reasons and incentives for reproducibility and replicability II: more reasons and incentives! Policies of journals Research Data Management Seminars: Replication 22 Source: Vlaeminck (2021, p. 13) Replication package: Components of a (generic) replication package I packed my replication package and in it I put … •Data •Code/Analysis Scripts •Metadata •Readme file •Licensing Research Data Management Seminars: Replication 23 Replication package: Components of a (generic) replication package •Data –Raw data –(Mostly) not processed data •Code/Analysis Scripts –Scripts or code files (e.g., Python, R, Stata, SPSS, MATLAB) that were used to create and process the dataset as well as analyze the data •Metadata –Detailed descriptions of the data files, variables, and the structure of datasets (codebooks) –Information on how the data was collected, any preprocessing, and how variables were constructed –Information on provenance, such as project information, consent forms and ethics documentation, if applicable 24 Replication package: Components of a (generic) replication package •Readme file –Instructions for running the analysis, explaining what each file does, how to execute the scripts, and how to interpret the results. –A brief summary of how to reproduce the analysis step-by-step. •Licensing –A clear statement about the use rights for the replication package, specifying under which terms others can reuse the data and scripts. –CC0: Works under a CC0 license can be freely used, modified, and shared without attribution –CC BY: Allows copying, sharing, and adapting a work, including commercially, with credit to the original author –CC BY-NC: Allows copying, sharing, and adapting a work for noncommercial use with credit to the original author (however NC is not ideal and should be used with caution) –CC is not designed for software! Use GNU General Public License or other suitable licenses 25 Replication package: Summary Research Data Management Seminars: Replication 32 Checklist: Data Code/Analysis Scripts Metadata Read me file Licensing Thank you very much for your attention! Research Data Center at the University Library: forschungsda[email protected] Dr. Phil Kolbe: phil.k[email protected] 33 Research Data Management Seminars: Replication Sources Cold Spring Harbor Laboratory Library (n.d.): Research Data Management: Folder Structure. LibGuides. Retrieved November 04, 2025, from https://cshl.libguides.com/c.php?g=1420116&p=10531416 GitBook Bot, Lambert Heller, datawomanHUB, mcancellieri, Bianca Kramer, Tony Ross-Hellauer, ilabastida, helenebr, Pedro Fernandes, & Jon Tennant. (2018). Open Science Training Handbook (1.1). Zenodo. https://doi.org/10.5281/zenodo.1212538 Schenk, Patrick, Vanessa A. Müller, Luca Keiser (2024): Social Status and the Moral Acceptance of Artificial Intelligence. Sociological Science 11:989-1016. DOI: 10.15195/v11.a36 Schenk, P., Müller, V., Keiser, L., & Abend, G. (2024): Repository Social Status and the Moral Acceptance of Artificial Intelligence [Data set]. University of Lucerne. https://doi.org/10.5281/zenodo.13850549 Senate of the University of Mannheim (2023): Code of Good Research Practice at the University of Mannheim. Retrieved November 04,2025, from https://www.uni-mannheim.de/en/research/our-research/good-research-practice/ The Turing Way Community (2024): Overview of Reproducible Research. Definitions. Retrieved November 04, 2025, from https://book.the-turing-way.org/reproducible-research/overview/overview-definitions/ Vlaeminck, Sven (2021) : Dawning of a New Age? Economics Journals’ Data Policies on the Test Bench, LIBER Quarterly: The Journal of the Association of European Research Libraries, ISSN 2213-056X, LIBER (Association of European Research Libraries), Den Haag, Vol. 31, Iss. 1, pp. 1–29,https://doi.org/10.53377/lq.10940 34 Research Data Management Seminars: Replication