LSE ORWG Guide: Code Review

Open Research Working Group Guide Code Review Guide Many quantitative researchers conduct data collection, data cleaning, data analyses, and statistical analyses, by writing code using programming languages, statistical packages, and software such as C, Java, Python, R, Stata, MATLAB, and others. A key principle of Open Research for quantitative researchers is that their statistical analyses are reproducible by other researchers using the same code. It is therefore vitally important that their code is reproducible, clear, and understandable by future researchers. One method of improving the reproducibility of analysis code is implementing code review. Code review generally refers to someone not involved in writing the original code checking the data cleaning and analysis codes. This guide gives some tips on implementing code review. More background, details, and tips can be found in this article about code review as a simple trick to enhance reproducibility. Other types of code review Use a code style guide that is common across your research team This may include overall format such as use of a header and annotations, and variable cleaning conventions such as using descriptive, unique variable names and naming binary variables so that 0=no and 1=yes. Ensure the reviewer has a copy of the style guide. Highlight any dependencies on other software. Document the computing environment. Record the exact version of the language (e.g., R 4.4.0; Python 3.12.2) and every package/library used (e.g., tidyverse 2.0.0, data.table 1.16.0, lme4 1.1-35.5; pandas 2.2.2). Share data to enable reviewer access. Whenever permitted by ethics approvals and contractual obligations, deposit raw (or suitably anonymized) datasets, codebooks, and environment files in a public repository such as the Open Science Framework (OSF). If full data sharing is not possible, provide a redacted version along with clear instructions for how qualified individuals can request access to the full dataset. Choose a reviewer who is familiar with the programming language, statistical package, or software, methodological approach, and possibly datasets used. Consider involving the code reviewer as co-author of the article, if interested. Consider using an explicit framework to acknowledge the different contributions of the authors, such as the CRediT framework, for example. The code reviewer should review the code line-by-line to check that it is working as intended by the methods section. The reviewer can build in checks that the code is working and label the code. The reviewer should provide constructive, respectful feedback to the research team, emphasising strengths as well as areas for improvement. Some journals implement code review as part of their peer review process, but it is best practice to review your code within your research team prior to submission. Include the code review method in the methods section. Code walkthrough - Analyst leads the researcher through code line by line, explaining what is happening at each step. Pair programming - Two analysts work together to write code Internal replication - Two analysts write the code independently, fixing errors until the results are the same. Matteo M. Galizzi -- Writing - r eview & editing; Supervision. Thomas A. Graves -- Writing - original draft; Writing - review & editing Dario Krpan -- Writing - review & editing. External replication - The code and the data are shared with someone outside the research group and see if they can reproduce the results. You can find more guides to open research in the social sciences at our Zenodo page.