Replik zum Impulspapier "Die Weiterentwicklung von Bewertungsverfahren für Forschungsleistungen im Kontext von Open Science"

Reply to the impulse paper “The future development of assessment processes for research activity in the context of open science” Philipp Zumstein, Open Science Office, University of Mannheim https://orcid.org/0000-0002-6485-9434 David Philip Morgan, Open Science Office, University of Mannheim https://orcid.org/0000-0001-8213-451X Introduction The impulse paper “The future development of assessment processes for research activity in the context of open science” 1 (Alexander von Humboldt Foundation et al., 2025) identifies two aspects for Open Science about evaluation procedures. We will focus on the second here, but for the sake of completeness, we will also briefly discuss the first. The first aspect deals with evaluation procedures themselves and how they can become more open and transparent, in particular using open data bases (with reference to the Barcelona Declaration). There is a clear call for this in the paper: “Information systems that support the documentation and evaluation of research activities should therefore, in future and in the form of a common good, be open, freely usable, science-led, created while giving consideration to legal and ethical restrictions, and operated in a sustainable manner.” (p. 6) We welcome this clear position and see this impulse as both challenging and forward-looking. The second aspect is the inclusion of Open Science as an evaluation criterion itself. Here, the paper finds it much more difficult to make clear statements. In the end, one even comes to the statement “requesting open practices in the form of standardised criteria [..] is not […] to be recommended” (p. 8). Even if the wording still leaves some room for manoeuvre here, we consider this core statement to be a wrong impulse in the end. This does not initiate a process of change in scientific evaluation procedures and for the individual researchers it is a signal that everything will remain as before. There is a lack of incentives for change or reflection on the existing approach to research. Instead, the paper argues that Open Science should become part of good scientific practice in the long term. But how is such a transition realistically supposed to happen if you do not start creating incentives for precisely such aspects? What arguments does the paper offer for this conclusion? We have identified the following three points from the context, although some of them could only be read between the lines: 1. Open Science does not guarantee research quality 2. Opening up data, processes or results is not always possible (even with excellent research) 3. Open Science as an evaluation criterion could lead to undesirable developments in the field or represent a disadvantage for some subjects We would like to comment on all three points below. 1 We refer here to the English version of the impulse paper. However, we noted that there are some differences between the two versions of the impulse paper, e.g. “Weiterentwicklung” in the title was translated as “future development” rather than “further development”, so some nuances may be more accurately reflected in the German version. Point 1: Open Science and Research Quality The paper itself states that there is empirical evidence that “openness has positive effects on systems” (p. 9), but then emphasizes that Open Science is no guarantee of the quality of research. The same argument also applies to other criteria such as the number of citations or the reputation of a journal. Of course, no one would demand that Open Science serve as the sole criterion for research quality or even for the overall scientific evaluation. Rather, it should be a matter of taking Open Science aspects (among other criteria) into account in evaluation procedures for research activities. The current problems of research should also be considered here, as they also have an impact on quality: These range from the replication crisis and reproducibility problems, publication bias, HARKing, p-hacking to manipulated data. Open science as an open, transparent, replicable and reproducible approach to research counteracts many of these problems (without solving everything at once). Up to now the evaluation criteria and the resulting incentives for researchers are likely to be one reason for these generally undesirable developments. Therefore, we are convinced that Open Science must be an essential part of evaluating and assessing the quality of research. The impulse paper only states that “Open science practices are to be considered part of good scientific practice” (p. 7). While this is welcome, we believe it is not enough. In practice, policy documents, although they are important for the positioning of institutions, are rarely read and the wording in them, which is usually kept open, unfortunately does not necessarily lead to an adaptation of one's own research action. Only by creating real incentives based on further developed evaluation procedures with Open Science components can a change process to improve research quality be driven forward (see also the “Pyramid of Cultural Change” (Nosek, 2019). Without real incentives for Open Science, the status quo persists, there is more publication bias, more replication and reproducibility crises, and there is a failure to directly address the problems that undermine research quality, even though Open Science solutions are available. Point 2: Limits of Open Science Practices The individual circumstances of the specific research project can play a role in shaping Open Science practices. There may be good reasons not to open up certain data or processes, but that doesn't have to be a contradiction to getting involved in Open Science. Especially in the case of research data, the principle is “as open as possible, as closed as necessary”. Evaluation procedures that are based on this principle (e.g. EU funding) do not represent a disadvantage for different cases. If, in one case, the research data cannot be shared openly, then this data can still be made FAIR. In addition, if open data is not possible, one could always look at other aspects of open science, for example open access for publication, pre-registration and open materials/software. In general, Open Science is not a yes/no criterion, but perhaps more of an evaluation dimension. For example, one can ask to what extent Open Science should be applied in an existing research project. The potential (in the sense of fully fulfilled) would then always be project-dependent and no one would demand impossible Open Science practices. However, if someone in a research project puts a lot of effort into documenting materials for reuse, for example, then this should also be rewarded positively. It seems essential to us that evaluation procedures recognize that Open Science is multilayered and that one should not demand all-encompassing implementation from applicants. Point 3: Open Science in Different Subject Contexts The paper defines the core areas of Open Science as the “open access to research results such as to text publications, research data and software” (p. 4) These three points all focus on opening up research results and the last two (data, software) are not equally relevant across the multitude of subjects. But as the paper goes on to explain, “other aspects” such as transparent, reproducible and inclusive research are then added to Open Science. Taking such a holistic view makes it easier to transfer Open Science to different academic cultures, but the impulse paper does not convey this impression. In fact, many disciplines have already begun to use differentiated approaches and solutions for Open Science, so the inclusion of Open Science in evaluation processes is unlikely to be a disproportionate disadvantage for any discipline. For example, there is the registration of clinical trials in medicine, pre-registrations in psychology, preprints in mathematics, reproduction packages in economics, transparent approaches to qualitative research 2 , mosquito atlas 3 in the life sciences, diamond OA journals in linguistics, transcription editathons in history, OA blogs in law. We believe it is important that evaluation procedures take into account the diversity of open science approaches in order to avoid disciplinary disadvantages. In summary, we would like to reply to this point that Open Science is not limited to a single path. Rather, there are different forms of Open Science, which can be both project-dependent and subject-dependent. UNESCO also names this flexibility as one of the core principles in the design of Open Science: “there is no one-size-fits-all way to practice open science” (UNESCO, 2021, p. 19). Closing remarks Open Science can contribute a lot to improving and further developing the quality of research (see section ‘Open Science and Research Quality’). Therefore, we would like to see assessment procedures in Germany expanded to include Open Science in order to create incentives for researchers and tackle the cultural change. In the case of implementation, care must be taken to ensure that on the one hand no impossible or all-encompassing implementation is required (see section ‘Limits of Open Science Practices’) and, on the other hand, that projectand subject-dependent contexts are also taken into account (see section ‘Open Science in Different Subject Contexts’). We are also happy to be available for exchange formats and discussions and can contribute our experience with Open Science to a university with a focus on economics and social sciences. Literature Alexander von Humboldt-Stiftung, Deutsche Forschungsgemeinschaft, FraunhoferGesellschaft, Hochschulrektorenkonferenz, Leibniz-Gemeinschaft, Nationale Akademie der Wissenschaften Leopoldina, Deutscher Akademischer Austauschdienst, HelmholtzGemeinschaft, Wissenschaftsrat (2025). Die Weiterentwicklung von Bewertungsverfahren für Forschungsleistungen im Kontext von Open Science. Zenodo. https://doi.org/10.5281/zenodo.15818267 Alexander von Humboldt Foundation, Deutsche Forschungsgemeinschaft, Fraunhofer Society, German Rectors' Conference, Leibniz Association, German National Academy of Sciences Leopoldina, German Academic Exchange Service, Helmholtz Association of German Research 2 Cf. for example https://qdr.syr.edu/ati 3 https://mueckenatlas.com/ Centres, & German Science and Humanities Council (2025). The future development of assessment processes for research activity in the context of open science. Zenodo. https://doi.org/10.5281/zenodo.15836010 Nosek B. (2019). Strategy for culture change. Center for Open Science. https://www.cos.io/blog/strategy-for-culture-change UNESCO (2021). UNESCO recommendation on open science. UNESCO. https://doi.org/10.54677/MNMH8546 Suggested citation: Zumstein, P., Morgan D. P. (2025). Replik zum Impulspapier „Die Weiterentwicklung von Bewertungsverfahren für Forschungsleistungen im Kontext von Open Science“ = Reply to the impulse paper “The future development of assessment processes for research activity in the context of open science”. Zenodo. https://doi.org/10.5281/zenodo.17602816 This work is licensed under the Creative Commons Attribution 4.0 International license.