On the evaluation of research software: the CDUR procedure
Abstract
The goal of the proposed talk is to present the publication Gomez-Diaz T. and Recio T., On the evaluation of research software: the CDUR procedure, [version 2 ; peer review: 2 approved] 26 Nov 2019. F1000Research 2019, 8:1353, https://f1000research.com/articles/8-1353 Background: Evaluation of the quality of research software is a challenging and relevant issue, still not sufficiently addressed by the scientific community. Methods: Our contribution begins by defining, precisely but widely enough, the notions of research software and of its authors followed by a study of the evaluation issues, as the basis for the proposition of a sound assessment protocol: the CDUR procedure. Results: CDUR comprises four steps introduced as follows: Citation, to deal with correct RS identification, Dissemination, to measure good dissemination practices, Use, devoted to the evaluation of usability aspects, and Research, to assess the impact of the scientific work. Conclusions: Some conclusions and recommendations are finally included. The evaluation of research is the keystone to boost the evolution of the Open Science policies and practices. It is as well our belief that research software evaluation is a fundamental step to induce better research software practices and, thus, a step towards more efficient science.
Full text
Research Software evaluation: CDUR - new evaluation protocols Teresa Gomez-Diaz [*], Tomas Recio [**] [*] CNRS - Laboratoire d’informatique Gaspard-Monge (Est of Paris) [**] Universidad Antonio de Nebrija (Madrid), Universidad de Cantabria (Santander) This work is licensed under the Creative Commons Attribution 4.0 International License http://creativecommons.org/licenses/by/4.0/ SORSE - A Series of Online Research Software Events March 16th 2021, https://sorse.github.io/ https://sorse.github.io/programme/talks/event-039/ TGD (LIGM), T. Recio (Nebrija) CDUR SORSE, March 16th 2021 1 / 15
Context Local contex in my lab LIGM: LIGM’s HCERES evaluation (2013-18), february 2019 Important research software production: 66,50 free/open source How evaluate this research production ? Global context: Free/OSS, Open Science, evolutions in evaluation Free Software (R. Stallman, FSF, 1985) and OSS (OSI, 1988) Budapest Open Access Initiative (BOAI, 2002), open access definition 2 Reports EC Expert Groups - evolutions for Open Science (adoption & practice) I[9] (2017) Cabello Valdes C, Rentier B, et al.: Evaluation of research careers fully acknowledging Open Science practices. ... I[11] (2019) Gu´edon JC, Jubb M, et al.: Future of Scholarly Publishing and Scholarly Communication. Report of the Expert Group to the EC. [9,11] Software (use & prod.) is now included in Open Science best practices. [11] The conclusion is actually simple: the evaluation of research is the keystone. TGD (LIGM), T. Recio (Nebrija) CDUR SORSE, March 16th 2021 2 / 15
Goal of this talk (1/2) Software production is not correctly assessed in research evaluation. It is necessary to change evaluation practices, taking into consideration Open science evolutions. The goal of this talk is to present this work and motivate the lecture of: Gomez-Diaz T. and Recio T., On the evaluation of research software: the CDUR procedure [version 2 ; peer review: 2 approved] 26 Nov 2019 (V1, 5 Aug 2019). F1000Research 2019, 8:1353, https://doi.org/10.12688/f1000research.19994.2 +1 year of writing, double expertise, 80 references, 85 footnotes Choice of F1000Research: Section Science Policy Research and open peer review procedure. Note1: RS means research software. Note2: [nb] refers to the reference number nb in V2. TGD (LIGM), T. Recio (Nebrija) CDUR SORSE, March 16th 2021 3 / 15
Goal of this talk (2/2) The article proposes the CDUR protocol to take into account RS production in research evaluation. The protocol CDUR is flexible in order to be adapted to all different research evaluation contexts, all along the research life. There are four steps: (C) Citation clear RS identification as a research production, title, authors, version, dates... (D) Dissemination dissemination best practices any dissemination has its own goals and target public (U) Use evaluation of software aspects of RS correct results, facilitated reuse... (R) Research evaluation of research aspects of RS scientific work: algorithms, related publications, impact... TGD (LIGM), T. Recio (Nebrija) CDUR SORSE, March 16th 2021 4 / 15
Plan 1Context 2Goals 3Concepts Research Software (RS) RS author RS publications Reference and citation 4On research evaluation Research evaluation contexts Two evaluation methods 5Protocol(s) CDUR for RS evaluation 6Conclusions TGD (LIGM), T. Recio (Nebrija) CDUR SORSE, March 16th 2021 5 / 15
Concept: research software (RS) (1/3) [2] (1994) Partha D, David PA: Toward a new economics of science there may be important positive spillovers across projects in the form of “learning effects” [...] including the development of computer software for performing data processing, storage, retrieval and network transmission. [16] (2011) Kelly D: An Analysis of Process Characteristics for Dev. Scientific Soft. Scientific software is defined by (1) it is developed to answer a scientific question ; (2) it relies on the close involvement of an sientific expert ; and (3) it provides data to be examined by the person who will answer that question ... [12] (2011) TGD: Article vs. Logiciel: questions juridiques et de politique scientifique... (2009) TGD: Guide laboratoire pour recenser ses d´eveloppements logiciels (PLUME) logiciel du laboratoire tout programme utile pour faire avancer la recherche, qui a ´et´e produit avec la participation d’un membre du laboratoire. Il arrive souvent que des publications de recherche soient associ´ees. [18] (2012) Sletholt MT, Hannay JE, et al.: What Do We Know about Scientific Software Development’s Agile Practices ? software developed by scientists for scientists [19] (2016) Hettrick S: Research Software Sustainability Research software is developed within academia and used for the purposes of research: to generate, process and analyse results. This includes a broad range of software, programs written by researchers for their own use. [10] (2018) NASA Committee: Open Source Software Policy Options for NASA Earth and Space Sciences Research software – that is, the software that researchers develop to aid their science... TGD (LIGM), T. Recio (Nebrija) CDUR SORSE, March 16th 2021 6 / 15
Concept: research software (RS) (2/3) [16] (2011) Kelly D: An Analysis of Process Characteristics for Dev. Scientific Soft. Iexclusion of what can be included in other definitions: [...] control software whose main functioning involves the interaction with other software and hardware ; user interface software [...] ; and any generalized tool that scientists may use in support of developing and executing their software, but does not of itself answer a scientific question. Ithe importance of the correctness: If the software gives the wrong answer, all other qualities become irrelevant [12] (2011) TGD: Article vs. Logiciel: questions juridiques et de politique scientifique... definitions don’t care about software status: “project”, “ended”, disseminated, quality, scope, size, documented, maintained, team’s internal use for an article, currently used in several labs... The proposed definition of Research Software (section 2.1): Research software (RS) is a well identified set of code that has been written by a well identified research team. It is software that has been built and used to produce a result published or disseminated in some article or scientific contribution. Each RS encloses a set of files containing the source code and the compiled code. It can also include other elements as the documentation, specifications, use cases... TGD (LIGM), T. Recio (Nebrija) CDUR SORSE, March 16th 2021 7 / 15
Concept: research software (RS) (3/3) Conclusion of RS definition: what is done: code, software as a well identified set of files, who does it: author(s), but also contributors or scientific expert(s), to make what: research, science, that is, associated articles, important: quality and correctness of the produced scientific results. Moreover: software/computer program is a legal concept: Directive 2009/24/EC of the European Parliament & Council 23/04/2009 https://eur-lex.europa.eu/eli/dir/2009/24/oj [12] (2011) TGD: Article vs. Logiciel: questions juridiques et de politique scientifique... studies software as a legal object and as research production TGD and Recio T., Open comments on the Task Force SIRS report: Scholarly, Infrastructures for RS, RIO 2021, https://doi.org/10.3897/rio.7.e63872 Free/Open source Research Software production at the Gaspard-Monge Computer Science laboratory, FOSDEM’21, https://fosdem.org/2021/schedule/event/open_research_gaspard_monge/ Other works on research software, FAIR, and related concepts, see for example: IFAIR 4 Research Software, https://sorse.github.io/programme/workshops/event-016/ IResearch software definition - Subgroup 3, https://docs.google.com/document/d/1PvYiYJxd7-vrmTusTvS8fYp47Wu6v-c_XMu-LjIBKio/ IA Fresh Look at FAIR for Research Software, https://arxiv.org/abs/2101.10883 TGD (LIGM), T. Recio (Nebrija) CDUR SORSE, March 16th 2021 8 / 15
Concept: RS author What means RS author ? [12] (2011) TGD: Article vs. Logiciel: questions juridiques et de politique scientifique... legal concept: the author writes the code scientific concept: expert contributions, maybe no writing code without the scientific expert, the RS will not exist maybe other contributions: documentation, bug fixing, test, maintenance, translations... Definition of a RS author: in the article we select three rˆoles (limits can be fuzzy): I(i) RS leader, I(ii) main or important contributor (code writing), I(iii) minor contributor (code writing or other contribution). Persons with no code writing can be assigned with some participation percentage of code writing by the team. TGD (LIGM), T. Recio (Nebrija) CDUR SORSE, March 16th 2021 9 / 15