scieee AI-readable full text Open interactive document viewer

Testing Research Software

Eisty, Nasir; Carver, Jeffrey; Kanewala, Upulee

Publisher
Zenodo
DOI
10.5281/zenodo.17258788
Repository
zenodo.org
Language
en

Abstract

This poster describes the work presented in a published paper [1], which reports the results of a survey of research software developers. The focus of the study was to understand how research software developers design test cases, handle output challenges, use metrics, execute tests, and select tools. Specifically, the study explores the following research questions; RQ1: What are the characteristics of the research software testing process? RQ2: What challenges do developers face throughout the testing process of research software? RQ3: How do research software developers design test inputs? RQ4: What specific challenges do research software developers face when determining the expected output of test cases? RQ5: What metrics do research software developers use to measure software quality and test quality of tests? RQ6: How do research software developers execute their tests? RQ7: What testing tools do research software developers use and what are their limitations? RQ8: What features should a testing tool specifically developed for research software contain? RQ9: Are demographic characteristics of research software projects related to the testing process? The poster will contain graphs reporting the detailed results related to these questions. Our overall findings show that research software testing varies widely. The primary challenges faced by research developers include test case design, evaluating test quality, and evaluating the correctness of test outputs. Our findings highlight the need to allocate more resources to software testing and provide more education and training to research software developers on software testing. References Eisty, N.U., Kanewala, U. & Carver, J.C. Testing research software: an in-depth survey of practices, methods, and tools. Empir Software Eng 30, 81 (2025). https://doi.org/10.1007/s10664-025-10620-6

Full text

Testing Research Software Nasir Eisty, University of Tennessee, Knoxville, TN Jeffrey Carver, University of Alabama, Tuscaloosa, AL Upulee Kanewala, University of North Florida, Jacksonville, FL Context Reference Conclusion Recommendations Objectives Key Results ❖Challenges: ➢Test case design ➢Evaluating test quality ➢Correctness of outputs ❖Approaches: ➢62% manual input design ➢Outputs often lack oracle ➢Non-determinism complicates validation ❖Metrics: ➢Statement coverage most used ➢Many don’t track quality metrics ❖Execution: ➢Mostly on push or end of cycle ➢Few use prioritization ❖Tools: ➢GitHub Actions, Pytest, Jenkins, Codecov ➢No standard set ❖Explore testing practices for research software ❖Understand challenges: test design, outputs, metrics, execution, tools ❖Investigate demographic influences (domain, team size, roles) ❖Research software is critical for science and innovation ❖Ensuring correctness is essential but challenging ❖Industrial testing tools are often inadequate ❖Research software testing is less systematic and automated than industry. ❖Oracle problem and test case design are biggest hurdles. ❖More resources, specialized tools, and education needed. ❖Funders/managers: allocate more testing resources ❖Developers: adopt documentation + systematic testing ❖Tool builders: create domain-aware and easy to use tools ❖Educators: integrate testing into research curricula Method ❖Survey of 131 research software developers ❖Topics: test input design, expected outputs, metrics, tools, limitations ❖Recruitment via US-RSE, UK-RSE, IDEAS-ECP, BSSW mailing lists Eisty, N.U., Kanewala, U. & Carver, J.C. Testing research software: an in-depth survey of practices, methods, and tools. Empir Software Eng 30, 81 (2025). https://doi.org/10.1007/s10664-025-10620-6 Demographics Characteristics of the Testing Process RSE? Degree type? Domain Source of testing knowledge #FTEs %time on testing Ranking of Testing Challenges Influence of Demographics (Systematic vs FTEs) Scan to read the paper