Responsible Use of Metrics for Assessment of (open) Research

#LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 Alicia Fátima Gómez IE University Library (Madrid, Spain) Responsible Use of Metrics for Assessment of (open) Research Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 What do you consider traditional vs. responsible metrics? 2 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 Limitations of the tradicional use of metrics 3 https://www.natureindex.com/newsblog/whats-wrong-with-the-h-indexaccording-to-its-inventor https://www.frontiersin.org/articles/ 10.3389/fnhum.2016.00556/full https://www.natureindex.com/newsblog/whats-wrong-with-the-jif-in-fivegraphs https://www.scholarlyassessmentreports. org/articles/10.29024/sar.10/ #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 4 Next-generation metrics : responsible metrics and evaluation for open science, Publications Office, 2017, https://data.europa.eu/doi/10.27 77/337729 Monitoring the open access policy of Horizon 2020 –Final report, Publications Office,2021, https://data.europa.eu/doi/10.2777/268348 Mutual learning exercise – Open science – Altmetrics and rewards – Horizon 2020 policy support facility, Publications Office,2018. https://data.europa.eu/doi/10.2777/ 468970 Evaluation of research careers fully acknowledging Open Science practices –Rewards, incentives and/or recognition for researchers practicing Open Science, Publications Office, 2017, https://data.europa.eu/doi/10.277 7/75255 European Commission: Directorate-General for Research and Innovation #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 Open science and the change of paradigme * JOURNAL ARTICLES * JOURNAL BASED METRICS (JIF, quartiles and deciles) * CITATIONS and related (h index) * More than journal articles: PREPRINTS, REPORTS, DATASETS * OBJECT LEVEL METRICS AND ALTERNATIVE METRICS OPEN SCIENCE Translated from: Alonso-Arévalo, J. (2019) El conocimiento es de todos y para todos ¿Qué es y qué implica la Ciencia Abierta? https://universoabierto.org/2019/09/30/el-conocimiento-es-de-todos-y-para-todos-que-es-y-que-implica-la-ciencia-abierta/ CLOSED SCIENCE OPEN SCIENCE Based on publishing articles Based on publishing any research outputs Individualistic science Collaborative science Access to research outputs for a few researchers Access to research outputs for the entire society Vertical, specialized science Horizontal, interdisciplinary science Science without citizen participation Citizen science, with and for Society Credit system based on the impact of publications Credit system based on the impact of the researcher Journal metrics, Impact Factor and Citescore or similar New metrics and next generation metrics, new indicators 5 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 Movement of the responsible use of metrics The Metric Tide: Role of Metrics in Research Assessment and Management. Wilsdon, J., et al. (2015). DOI: 10.13140/RG.2.1.492 9.1363 San Francisco Declaration on Research Assessment http://am.ascb.org/ dora/ 12/2012 07/2015 Bibliometrics: The Leiden Manifesto for research metrics. Hicks, D.,et al. (2015) Nature. 520, 429-31. DOI: 10.1038/520429a 07/2022 Agreement on Reforming Research Assessment. Coalition for Advancing Research Assessment (COARA). https://coara.eu/agr eement/theagreement-full-text/ 04/2015 The Hong Kong Principles for assessing researchers: Fostering research integrity (2019). DOI: 10.1371/journal.pbi o.30007 06/2019 Barcelona Declaration on Open Research Information Organizations performing, funding or evaluating research https://doi.org/10.5281/ zenodo.10958522 04/2024 6 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 Principles of the Leiden Manifesto 1. Quantitative evaluation should support qualitative, expert assessment. 2. Indicators used to evaluate performance should relate clearly to the program goals. 3. Protect excellence in locally relevant research. 4. Keep data collection and analytical processes open, transparent and simple. 5. Allow those evaluated to verify data and analysis. 6. Consider field differences in publication and citation practices 7. Base assessment of individual researchers on a qualitative judgement of their portfolio. 8. Avoid misplaced concreteness and false precision. 9. Recognize the systemic effect of the assessment and indicators. 10.Scrutinize indicators regularly and update them. 8 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 “China now moves from a strong focus on Web of Science-based indicators towards a more balanced combination of qualitative and quantitative research evaluation” (June 11th, 2020) https://blogs.lse.ac.uk/impactofsocialsciences/2020/06/11/for-chinasambitious-research-reforms-to-be-successful-they-will-need-to-besupported-by-new-research-assessment-infrastructures/ Farewell to “SCI worship”. Indicators based on WoS will not be applied directly in evaluation and funding at any level. An alternative citation index with Chinese characteristics and international influence will be established. These sources could be integrated in a national Current Research Information System (CRIS) From metrics to peer review. A new focus on novelty, scientific value, research integrity, innovation potential and societal outcomes will replace the “paper only” orientation in panel evaluations. Number of publications and journal impact factors will not count any more. Local relevance. Publications in high-quality Chinese journals will be encouraged, and the development of such journals will be supported. 9 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 What are the challenges? Which indicators would you consider? 10 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 TU Delft: Recognition & Rewards Perspective 2021-2024 https://www.tudelft.nl/en/about-tu-delft/strategy/recognition-rewards-perspective The modernisation of the system of recognition and rewards is very much connected to the university’s Open Science programme. 17 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 Utrecht University: Recognition and Rewards https://www.uu.nl/en/research/open-science/good-practices/recognition-and-rewards 18 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 TARA (Tools to Advance Research Assessment) Support DORA in identifying and understanding the criteria and standards used by universities to make decisions on hiring, promotion, and tenure. Develop practical resources and guidance on research assessment reform for institutions. https://sfdora.org/project-tara/ 19 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 https://sfdora.org/resource-library/ https://sfdora.org/resourcelibrary/?_resource_type=case-studies DORA Resource Library and Case Studies 20 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 Which sources do you know or do you use for research evaluation or monitoring? What are the bias? 21 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 Sources? https://www.lens.org/ https://www.scienceopen.com/ https://plumanalytics.com/ https://www.altmetric.com/ https://openalex.org https://search.opencitations.net https://www.semanticscholar.org/ https://graph.openaire.eu/ 22 #LIBER2025 Unlocking Libraries' potential: initiatives, methods and tools to monitor Open Science Lausanne, July 2, 2025 Alicia Fátima Gómez IE University Library THANK YOU! [email protected] http://es.linkedin.com/in/aliciafgomez https://orcid.org/0000-0003-4898-1680 23