Rethinking altmetrics as process-based indicators: a conceptual framework for construct clarity

1 Rethinking altmetrics as process-based indicators: a conceptual framework for construct clarity Adrián A. Díaz-Faes1, Zohreh Zahedi2,3 1INGENIO (CSIC-UPV), Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, Spain 2Department of Information Science, Faculty of Humanities, Persian Gulf University, Bushehr, Iran 3Centre for Science and Technology Studies (CWTS), Leiden University, the Netherlands Corresponding author: [email protected] ORCID: AADF: 0000-0003-1928-4608, ZZ: 0000-0001-5801-1886 Abstract: The Altmetrics Manifesto promised to broaden the notion of impact. However, reliance on citation theory has limited this promise, confining altmetrics to bibliometric principles by treating interactions with scholarly outputs that occur outside the norms and values that rule science through the same lens as citations. This study argues that this constitutes a form of ontological misalignment (fallacy of reification), whereby altmetrics are assigned entitative properties when they in fact capture processual aspects of engagement and use that may precede or contribute to societal impact. This misalignment is manifested as use of counts of social media and online interactions with scholarly outputs as direct indicators of societal impact. Aligning ontology and epistemology requires the redefinition of altmetrics as process-based indicators. We describe recent empirical developments applying interaction and networked approaches to translate this perspective into practice. We argue also that calls for production of socially relevant knowledge and increased interest in science communication represent a timely opportunity for use of altmetrics as process-oriented monitoring tools. To study societal impact, we suggest their use in a mixed-methods research design to identify engagement patterns that can be further explored using qualitative methods. Keywords: Science-society interactions, ontological shift, societal impact, engagement, processbased indicators. 2 1. INTRODUCTION Altmetrics emerged 15 years ago as promising web-based metrics for engagement, use, and impact. They were seen as expanding the scope of what can be measured systematically and what is worth measuring. Expectations about their ability to measure impact in a broader sense beyond journal-based scholarly publishing grew rapidly (Sugimoto et al., 2017). The aim of altmetrics 1 was to “value all research products” and “give a fuller picture of how research products have influenced conversation, thought and behavior” (Piwowar, 2013, p. 159). Although this ambition resonates with subsequent calls for reforms to research assessment 2 , which would acknowledge the diversity of the contributions in science and their impact (Rushforth & Hammarfelt, 2023), the initial promise and expectations surrounding altmetrics have yet to materialize (Wouters et al., 2019). We argue that these unfulfilled expectations are not due merely to data issues such as heterogeneity, coverage, and persistence or to the absence of standards but rather stem from the lack of a conceptual grounding. As altmetrics accumulate at a rapid pace and broaden narrow indicators of scientific impact, researchers rapidly leap into data analyses translating citation theory to social media, resulting in inconsistent interpretations of its meaning (Haustein, 2016; Wouters et al., 2019). A growing number of recent studies is eschewing bibliometric-based models and their application to social media and advocating for interactive and network approaches to analyze altmetric data. Examples include development of methods to account for the circulation of research beyond academic boundaries (Alperin et al., 2024; Costas et al., 2021), differentiation between types and degrees of engagement (Fang et al., 2022; Haustein et al., 2016), mapping of scientists’ engagement patterns (Robinson-García et al., 2018; Walter et al., 2019) and communication practices (Hare et al., 2024; Mongeon, 2018), clustering users based 1 Following Wouters et al. (2019), we use “metrics” to denote both data and indicators. 2 CoARA: https://coara.eu/app/uploads/2022/09/2022_07_19_rra_agreement_final.pdf 3 on their social media behavior (Araujo, 2020; Díaz-Faes et al., 2019; Pearce et al., 2014; Yu et al., 2019), and tracing socially relevant topics and misinformation (Shao et al., 2018; Van Schalkwyk et al., 2020). However, despite the merits of such approaches, the conceptual discussion required to underpin any shift in focus is mostly lacking. The present study aims to fill this gap by advocating for a shift in the conceptualization and application of altmetrics. We argue that a key obstacle to altmetrics realizing their initial promise is the absence of construct clarity, specifically the misalignment between what altmetrics capture (ontology) and how their meaning is interpreted and applied in research practice (epistemology). This arises because altmetrics remain grounded in bibliometric conventions and principles, and apply citation lens to interactions and exchanges that occur outside the norms and values that rule science. To improve construct clarity and position altmetrics as valuable tools to investigate science-society interactions 3 , we suggest they should be conceived as process-based indicators. We build on Thompson's (2011) framework to align ontology and epistemology to achieve construct clarity, and on Rescher’s work on process philosophy (2006). Our work is informed also by Haustein et al. (2016), Robinson-García et al. (2018), and Díaz-Faes et al. (2019). The former work addresses the need for specific frameworks, models, and theories to support interpretation of altmetrics. The latter two call for an interactive approach to altmetrics. Our paper is structured as follows: section 2 discusses the problems involved in applying a bibliometric model to societal impact, and shows that citation theory does not translate well to altmetrics. In this context, section 3 notes the importance of aligning ontology and epistemology and argues that altmetrics research suffers from a fallacy of reification (attributing entitative properties to processes). We argue that the shift towards a process view would make altmetrics more effective tools for studying science-society interactions. We also describe the empirical developments in the field in line with this view. Section 4 describes how the policy and evaluation 3 Costas et al. (2021) suggest that given the problems related to operationalizing ‘society’ as a well-bounded category, the distinctions science and non-science could be used as alternatives. 4 contexts create an opportunity space for use of altmetrics as process-oriented metrics. Section 5 presents the main implications of this study. 2. THE NEED TO DEPART FROM CITATION THEORY Citations have been discussed in the context of different competing and coexisting theories, mainly normative and constructivist (Moed, 2005). The normative view posits that citations are an institutionalized practice of science that requires scientists to acknowledge the sources and prior work on which they have built upon (Merton, 1973). Citations maintain “intellectual traditions and provide the peer recognition required for the effective working of science as a social activity” (Merton, 1988, p. 621). However, citation behavior is complex and cannot be described unidimensionally (Bornmann & Daniel, 2008; Brooks, 1986). The constructivist view challenges the normative theory, arguing that citations function as rhetorical devices used to persuade — for example, to convince readers of the novelty or value of one’s findings (Gilbert, 1977) — rather than to reflect intellectual influence. However, as Zuckerman (1987) notes, Gilbert’s (1977, p. 116) notion that some papers are “important and correct”, and therefore used as authoritative references to support authors’ own arguments, is itself grounded in peer recognition and typically manifested in high citation rates, which in turn make those papers authoritative. Empirical studies on the validity of both approaches to citation show that a normative account fits the data better than the persuasion model (Moed & Garfield, 2004; White, 2004). Besides theories, citing behavior can also be influenced by other factors, such as language biases, target audience, journal status, and the scope, format, or length of the paper (Cronin, 1981; MacRoberts & MacRoberts, 2018), although such biases tend to cancel out at aggregated levels (van Raan, 1998). 5 Despite the diverse motivations for citing and the presence of distorting factors 4 , it is generally agreed that citations reflect scholarly antecedents of work and serve as indicators of scientific impact (Aksnes et al., 2019; Martin & Irvine, 1983). Thus, one can assume that highly cited papers are generally scientifically relevant, and that citations can therefore be measured linearly and assumed to be positive (Bornmann & Daniel, 2008). However, such a convention cannot be applied to what altmetrics measure. The altmetrics niche emerged based on the inadequacy of bibliometric indicators and peer review — the traditional filters. The Altmetrics Manifesto describes these indicators as slow, narrow, and closed compared to altmetrics’ timeliness, diversity, and openness (Leckert, 2021). Citations have been criticized and described as: “counting measures [that] are useful, but not sufficient (…) influential work may remain uncited. These metrics are narrow; they neglect impact outside the academy and also ignore the context and reasons for citation” (Priem et al., 2010, p. 2). The Altmetrics Manifesto envisioned a change towards a more comprehensive understanding and assessment of science while in practice, the underlying logic does not represent any departure from the traditional filters. Altmetrics research rapidly became focused on counting 5 mentions, shares, and downloads of scholarly outputs in social media as if bibliometric conventions and the scientific ethos (the values and norms governing scientific activity) could be applied beyond the boundaries to academia. This can be seen clearly in the main topics addressed in altmetrics research. Much of the research on altmetrics has focused on two lines of inquiry. The first examines the relationship between altmetrics and citations (Holmberg et al., 2019; Sugimoto et al., 2017) on the premise that two indicators of impact should be correlated. Since altmetrics accumulate 4 The fact that not all citations are critical, and that scientist’s reason to cite can go beyond the merits of the work itself highlights the need for contextualized, multidimensional, and well-curated analyses. 5 Data providers seized the opportunity to provide aggregated indicators such as the Altmetric Attention Score which lumps together diverse mentions and interactions not based on any clear criteria (Thelwall, 2021). Critiques of this method seem not to have prevented its application. 6 faster than citations, it was suggested that altmetrics could be used as predictors of future citations. This led to research that involved counting the number of times scholarly outputs were tweeted, shared on Facebook, or mentioned in blogs and news media to obtain direct indicators of impact. However, the evidence shows that altmetric counts of scholarly outputs are poor predictors of later citations (Costas et al., 2015). A second line explores the use of altmetrics to assess the societal impact of research. Yet, empirical studies show that high altmetric counts are not correlated with peer review assessment of societal impact 6 (Bornmann et al., 2019; Kassab et al., 2020; Ravenscroft et al., 2017). There are two reasons why these findings are not a surprise. First, there are no grounds for a straight relationship between citations and altmetrics since not all relevant scientific research (e.g. blue-sky research) will necessarily lead to societal impact. Second, attention and engagement with science do not necessarily translate into societal impact. The qualitative assessment methods literature spells out the underlying reasons: the difficulty of linking specific scholarly outputs to observed societal impact (attribution problem), the often long time lag between research and its specific impact on society — temporality issue — (Spaapen & Van Drooge, 2011), and that societal impact encompasses social, cultural, environmental and economic aspects — multiple dimensions (Bornmann, 2013). Although the qualitative approaches to measuring societal impact are diverse and focus on different processes creating value, there is agreement about the need to move from attribution to contribution since societal impact is context-dependent and requires interactions between scientists and stakeholders (Smit & Hessels, 2021). Research with societal impact can take the form of a product (e.g. evidence, methods, technology, models), knowledge use (i.e. adoption of knowledge by stakeholders), or benefits based on its application and use (e.g. changes to practice, awareness, health, economic impact) (Bornmann, 2013; De Jong et al., 2014; Woolley & Molas6 The exception is Mendeley for which mild-medium correlations are found when compared with peer review scores from the UK Research Excellence Framework (Akella et al., 2021; Thelwall et al., 2023). Still, one can argue that this mainly reflect academic use. 7 Gallart, 2023). This stream of work offers a new lens allowing better use of altmetric data (Robinson-García et al., 2018). For instance, the SIAMPI approach (Spaapen & Van Drooge, 2011) defines interactions as productive if the knowledge produced is scientifically robust and socially relevant. The authors operationalize this by distinguishing between direct (people to people), indirect (interaction mediated through a carrier such as scholarly output), and financial (research contract, financial contribution) interactions. Another example is Morton’s (2015) ‘contributions approach’ which defines a process view of societal impact based on the notion of research uptake (i.e. engagement with research), research use (e.g. discussion of, sharing, adaptations to, application of the research to inform policy or action), and research impact (research resulting in changes to knowledge, awareness, attitudes, or practice). Despite the existence of empirical evidence and qualitative assessment methods that demonstrate the inadequacy of applying bibliometric conventions to altmetrics, most altmetrics research continues to do so. To assess the extent of this trend, we queried the Scopus database 7 and retrieved all Altmetric papers (articles and reviews) published between October 2010 — the year of the Altmetrics Manifesto — and 2024. We systematically screened the abstracts, methods, and results sections of each paper to check for any evidence of output or process-based approaches. This resulted in our classifying the studies into one of three groups: output-based (counting altmetric data on papers as direct indicators of impact, akin to citations), process-based (applying network and interaction approaches to examine patterns of engagement, dissemination, and use), and other (e.g. using altmetric data in fields such as digital marketing or education). Figure 1 shows that the vast majority of altmetrics research applies an output-based approach 7 (TITLE-ABS-KEY (altmetric*) OR TITLE-ABS-KEY (“social media metric*”)) AND PUBYEAR > 2010 AND PUBYEAR < 2024 AND (LIMIT-TO (SRCTYPE, "j”)) AND (LIMIT-TO (DOCTYPE, "ar”) OR LIMIT-TO (DOCTYPE, "re”)) AND (LIMIT-TO (LANGUAGE, "English”)). The search query retrieved 1,493 papers. After manual revision, 109 papers that were neither articles nor reviews were removed, resulting in a final sample of 1,392 papers. 8 (82%), while only 2% of papers adopt a process lens. The remaining 16% corresponds to the use of social media data unrelated to the scope of this study. Figure 1. Time trends in altmetrics research by analytical approach 3. IMPROVING CONSTRUCT CLARITY IN ALTMETRICS 3.1. Aligning ontology and epistemology As explained above, there is no basis for applying bibliometric conventions and citation theory to capture a broader notion of impact. This mismatch is evidence of the need to rethink our understanding and use of altmetrics which demands a better clarification of altmetrics as a construct. Constructs are conceptual abstractions adopted or invented for a specific scientific purpose (Venkatraman & Grant, 1986). As Suddaby (2010, p. 346) notes: “Clear constructs are simply robust categories that distill phenomena into sharp distinctions that are comprehensible to a community of researchers.” Because construct clarity provides a basis for theory and practice, we argue that it is crucial to align ontology and epistemology so that knowledge generated 9 through altmetrics reflects the phenomena altmetrics seeks to observe. Ontology concerns the basic assumptions about the type of things that can exist, their conditions of existence and what counts as relevant objects, entities and processes in a given domain. Epistemology, however, deals with how valid knowledge about these objects, entities and processes can be accessed and generated (Chalmers, 1999; Kilduff et al., 2011). Our proposed shift from a bibliometric model to a process view draws on Thompson’s (2011) framework to provide construct clarity and contrast the classical perspectives that perceive reality as an entity or as a process. An entity lens views reality (in our case altmetrics) as comprising objects with stable properties whose meaning does not change across multiple situations. This applies to how citations are seen in bibliometrics and their consideration as direct indicators of scientific impact. However, a process lens emphasizes that reality is dynamic, emergent, and evolving and that interactions constitute rather than simply modifying reality (Rescher, 2006). Rather than considering things (numbers of mentions, shares, or comments to scholarly outputs) as meaningful and indicative by themselves, altmetrics should be about understanding happenings (the interaction, engagement with, and use of). Figure 2 depicts how the misalignment between ontology (top half) and epistemology (bottom half) in altmetrics research leads to ontological drift, specifically the fallacy of reification. This fallacy attributes entitative properties to a construct that is more adequately understood as a process (quadrants 1 and 3). Grounded in an entity perspective borrowed from citation theory, altmetrics research treats diverse forms of online engagement and interactions with scholarly outputs as direct indicators of the broader impact of research. That is, there is an assumption that some specific societal impact can be attributed to mentions, comments, shares, downloads, or other forms of engagement and interactions with scholarly outputs. This ontological drift is manifested in the assumption that online engagement with science by diverse stakeholders is mediated by shared norms, values, and epistemic conventions similar to 16 of socially relevant knowledge requires new tools to measure broader notions of impact 10 (European Commission et al., 2017; Rafols et al., 2024). Since many activities and interactions with science are digital, altmetrics research adopting a process view can provide information on interactions, exchanges, dialogues, and the ways that different stakeholders engage with and use research knowledge. Note also that the greater openness rhetoric which underpins claims of production of socially relevant knowledge triggers some important challenges (Mirowski, 2018). For instance, preprints make scientific work available faster and foster critique and uptake of new ideas but also facilitate the diffusion of unverified and methodologically flawed research (see Caulfield et al., 2021 for the COVID-19 case). Also, current open access is more a profitable business for a few commercial publishers than a way to provide societal access to science (Butler et al., 2023). Use of altmetrics as process indicators allows us to examine whether open access is achieving its goals or reinforcing inequalities and hampering diversity. 4.2. The science communication model(s) Besides the greater expectations about scientific work bringing value to society, scientists are increasingly being encouraged by funders and their institutions to take a more active role in communicating science and establishing a dialogue with public audiences (NASEM, 2017; Weingart, 2017). As a result of digitalization, open access, and the rise of social media, science communication can no longer be understood as a linear and top-down process (deficit model) in which scientists generate new knowledge, science journalists operate as brokers, and the public passively consumes the information (Bucchi, 2017; Scheufele, 2014). This transformation of the science-society interface has democratized content creation and dissemination and paved the way 10 Concern over the “gamification” of societal impact measures as has happened with citations is well-founded (Holmberg et al., 2019). However, these arguments are related more to reform of the science reward system, research culture, and responsible use of indicators than total avoidance of quantitative data (Balboa et al., 2024). Peer review as a quality control mechanism is subject to bias (Horrobin, 1990), and qualitative assessments can lack objectivity and be hindered by time and resource constraints (Bornmann, 2013). 17 to dialogue, participatory, and co-creation models (Trench, 2008). However, it raises questions about how general audiences can identify reliable scientific information and how science communication can foster a shared understanding of scientific findings and support a dialogue to enact societal change. Robust socially relevant knowledge does not arise simply from scientific evidence but is spawned by continual interaction and conversation between fact-finding and meaning-making (Jasanoff, 2010). Although the science, technology and innovation community has so far paid little attention to science communication, recent studies show, for example, the utility of a network perspective and content analysis for unraveling the communities involved in anti-vaccine movements (Van Schalkwyk et al., 2020), and scientists’ engagement in the climate change debate (Walter et al., 2019). Likewise, science communication outputs — press releases, news items — are being analyzed quantitatively and becoming relevant to understanding online science communication (Groves et al., 2015; Orduña-Malea & Costas, 2023). Thus, altmetrics have the potential to monitor aspects related to the science communication processes. They can be relevant for policy to foster more informed and meaningful online interactions with science (i.e. diminish the effects of misinformation, fake news, and uncritical use of sources) (Díaz-Faes, 2023). 5. CONCLUDING REMARKS This study contributes to overcoming the seclusion of data-driven research in quantitative science studies (Rafols, 2019) by addressing the need for a conceptual framework in altmetrics research. Since construct clarity is crucial for theory and practice (Suddaby, 2010), we examine the misalignment between what altmetrics measure and how their meaning is interpreted and applied in practice. We also discuss construct validity to provide a basis for translating a shift towards viewing altmetrics as process-based metrics into research practice. The Altmetrics Manifesto promised a change by expanding the notion of impact. However, the conceptual lens underpinning altmetrics research has remained grounded in the principles of 18 bibliometrics. We argue that altmetrics research uncritically borrowed citation theory, treating interactions and exchanges on social media and online platforms as if they were similar to those mediated by the norms and values ruling scientific work. This leads to the underlying assumption that scientific and societal impact are somehow equivalent, despite the fact that the latter is context dependent and difficult to attribute to specific scholarly outputs. This has resulted in a form of ontological misalignment (fallacy of reification) in which altmetrics are assigned entitative properties as if they directly indicate societal impact, rather than recognizing that they capture processual aspects of engagement that may precede or contribute to societal impact. We have suggested that aligning ontology and epistemology requires a redefinition of altmetrics as process indicators. This shift is supported by the policy and evaluation context which needs quantitative evidence to provide information on the complex and pluralistic interactions and exchanges related to the potential antecedents of societal impact and science communication dynamics (Rushforth & Hammarfelt, 2023; Science Europe, 2022). Bornmann & Leydesdorff (2014) note that altmetrics are in a similar emerging state to that of bibliometric indicators in the 1970s. It could be argued that as a field, we are still navigating the development of a toolbox of methods for altmetrics while having embarked on this phase with no consensus on altmetrics as a construct. In developing altmetrics as process-based indicators to support assessment of societal impact, we suggest a mixed-method approach, in which process altmetrics adapt to and account for each platform’s specific characteristics and affordances in order to hint at relevant sciencesociety interactions and engagement patterns that can be explored further qualitatively. Author contributions AADF: Conceptualization, Data curation, Investigation, Formal Analysis, Methodology, Visualization, Funding acquisition, Writing-original draft, Writing-review & editing ZH: Investigation, Data curation, Formal analysis, Writing-review & editing 19 Competing interests The authors have no competing interests to disclose. Acknowledgments Adrián A. Díaz-Faes acknowledges support from research project PID2020-112837RJ-I00, funded by MCIN/AEI/10.13039/501100011033. We are grateful to Rodrigo Costas for insightful discussions, and to Maddie Hare and Pablo D’Este for their invaluable feedback on an earlier version of this paper. A preliminary version of this work was presented at NWB2024 in Reykjavík. References Aagaard, K., Norn, M. T., & Stage, A. K. (2022). How mission-driven policies challenge traditional research funding systems [version 1; peer review: 2 approved with reservations, 1 not approved]. F1000Research, 11(949). https://doi.org/10.12688/f1000research.123367.1 Akella, A. P., Alhoori, H., Kondamudi, P. R., Freeman, C., & Zhou, H. (2021). Early indicators of scientific impact: Predicting citations with altmetrics. Journal of Informetrics, 15(2), 101128. https://doi.org/https://doi.org/10.1016/j.joi.2020.101128 Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories. SAGE Open, 9, 1–17. https://doi.org/10.1177/2158244019829575 Alperin, J. P., Fleerackers, A., Riedlinger, M., & Haustein, S. (2024). Second-order citations in altmetrics: A case study analyzing the audiences of COVID-19 research in the news and on social media. Quantitative Science Studies, 5(2), 366–382. https://doi.org/10.1162/qss_a_00298 Araujo, R. F. (2020). Communities of attention networks: introducing qualitative and conversational perspectives for altmetrics. Scientometrics, 124(3), 1793–1809. 20 https://doi.org/10.1007/s11192-020-03566-7 Balboa, L., Gadd, E., Méndez, E., Pölönen, J., Stroobants, K., & Tóth-Czifra, E. (2024). The role of scientometrics in the pursuit of responsible research assessment. LSE Blog. https://blogs.lse.ac.uk/impactofsocialsciences/2024/09/04/the-role-of-scientometrics-inthe-pursuit-of-responsible-research-assessment/ Beck, S., Brasseur, T. M., Poetz, M., & Sauermann, H. (2022). Crowdsourcing research questions in science. Research Policy, 51(4), 104491. https://doi.org/https://doi.org/10.1016/j.respol.2022.104491 Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233. Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903. Bornmann, L., & Daniel, H. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80. Bornmann, L., Haunschild, R., & Adams, J. (2019). Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). Journal of Informetrics, 13(1), 325–340. Bornmann, L., & Leydesdorff, L. (2014). Scientometrics in a changing research landscape: bibliometrics has become an integral part of research quality evaluation and has been changing the practice of research. EMBO Reports, 15(12), 1228–1232. Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37(1), 34–36. Bucchi, M. (2017). Credibility, expertise and the challenges of science communication 2.0. 21 Public Understanding of Science, 26(8), 890–893. Butler, L.-A., Matthias, L., Simard, M.-A., Mongeon, P., & Haustein, S. (2023). The oligopoly’s shift to open access: How the big five academic publishers profit from article processing charges. Quantitative Science Studies, 4(4), 778–799. https://doi.org/10.1162/qss_a_00272 Caulfield, T., Bubela, T., Kimmelman, J., & Ravitsky, V. (2021). Let’s do better: public representations of COVID-19 science. FACETS, 6, 403–423. https://doi.org/10.1139/facets-2021-0018 Chalmers, A. F. (1999). What is this thing called science? (3rd ed.). Open University Pres. Costas, R., de Rijcke, S., & Marres, N. (2021). “Heterogeneous couplings”: Operationalizing network perspectives to study science-society interactions through social media metrics. Journal of the Association for Information Science and Technology, 72(5), 595–610. https://doi.org/10.1002/asi.24427 Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019. https://doi.org/10.1002/asi.23309 Cronin, B. (1981). The need for a theory of citation. Journal of Documentation, 37(1), 16–24. https://doi.org/10.1108/eb026703 Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A., & Sheikh, A. (2011). The case study approach. BMC Medical Research Methodology, 11(1), 100. https://doi.org/10.1186/1471-2288-11-100 De Jong, S., Barker, K., Cox, D., Sveinsdottir, T., & Van den Besselaar, P. (2014). Understanding societal impact through productive interactions: ICT research as a case. Research Evaluation, 23(2), 89–102. https://doi.org/10.1093/reseval/rvu001 22 Díaz-Faes, A. A. (2023). Science communication and public policy: Improving science society interactions – opportunities and challenges. In V. Pavone, J. Molas, J. Brandts, & R. Serrano (Eds.), Science for Public Policy (CSIC, pp. 76–79). Editorial CSIC. https://digital.csic.es/handle/10261/337975 Díaz-Faes, A. A., Bowman, T. D., & Costas, R. (2019). Towards a second generation of ‘social media metrics’: Characterizing Twitter communities of attention around science. PLOS ONE, 14(5), e0216408. https://doi.org/doi.org/10.1371/journal.pone.0216408 European Commission, B.-I. U., for Research, D.-G., Innovation, of Technology, G. U., University, K., for Economics, L. I. C., University, L., of North Texas, U., of Sheffield (UK), U., Peters, I., Frodeman, R., Wilsdon, J., Bar-Ilan, J., Lex, E., & Wouters, P. (2017). Next-generation metrics – Responsible metrics and evaluation for open science. Publications Office. https://doi.org/doi/10.2777/337729 Fang, Z., Costas, R., & Wouters, P. (2022). User engagement with scholarly tweets of scientific papers: a large-scale and cross-disciplinary analysis. Scientometrics. https://doi.org/10.1007/s11192-022-04468-6 Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. SAGE Publications. Gilbert, G. N. (1977). Referencing as Persuasion. Social Studies of Science, 7(1), 113–122. https://doi.org/10.1177/030631277700700112 Goffman, E. (1959). The presentation of the self in everyday life. Anchor. Groves, T., Figuerola, C. G., & Quintanilla, M. Á. (2015). Ten years of science news: A longitudinal analysis of scientific culture in the Spanish digital press. Public Understanding of Science, 25(6), 691–705. https://doi.org/10.1177/0963662515576864 Hare, M., Krause, G., MacKnight, K., Bowman, T. D., Costas, R., & Mongeon, P. (2024). Do 23 you cite what you tweet? Investigating the relationship between tweeting and citing research articles. Quantitative Science Studies, 5(2), 332–350. https://doi.org/10.1162/qss_a_00296 Haustein, S. (2016). Grand challenges in altmetrics: heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423. https://doi.org/10.1007/s11192-016-19109 Haustein, S. (2019). Scholarly Twitter metrics. In G. Wolfgang, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer Handbook of science and technology indicators (pp. 729– 760). Springer. https://link.springer.com/chapter/10.1007/978-3-030-%0A02511-3_28 Haustein, S., Bowman, T. D., & Costas, R. (2016). Interpreting “altmetrics”: viewing acts on social media through the lens of citation and social theories. In C. R. Sugimoto (Ed.), Theories of Informetrics and Scholarly Communication (pp. 372–406). De Gruyter Berlin, Boston. Holmberg, K., Bowman, S., Bowman, T., Didegah, F., & Kortelainen, T. (2019). What Is Societal Impact and Where Do Altmetrics Fit into the Equation? Journal of Altmetrics. https://doi.org/10.29024/joa.21 Horrobin, D. F. (1990). The Philosophical Basis of Peer Review and the Suppression of Innovation. JAMA, 263(10), 1438–1441. https://doi.org/10.1001/jama.1990.03440100162024 Jasanoff, S. (2010). A New Climate for Society. Theory, Culture & Society, 27(2–3), 233–253. https://doi.org/10.1177/0263276409361497 Kassab, O., Bornmann, L., & Haunschild, R. (2020). Can altmetrics reflect societal impact considerations?: Exploring the potential of altmetrics in the context of a sustainability science research center. Quantitative Science Studies, 1(2), 792–809. https://doi.org/10.1162/qss_a_00032 24 Kilduff, M., Mehra, A., & Dunn, M. B. (2011). From Blue Sky Research to Problem Solving: A Philosophy of Science Theory of New Knowledge Production. Academy of Management Review, 36(2), 297–317. https://doi.org/10.5465/amr.2009.0164 Larrue, P. (2021). The design and implementation of mission-oriented innovation policies. OECD Science, Technology and Industry Policy Papers No 100. https://doi.org/https://doi.org/https://doi.org/10.1787/3f6c76a4-en Leckert, M. (2021). (E-) Valuative Metrics as a Contested Field: A Comparative Analysis of the Altmetricsand the Leiden Manifesto. Scientometrics, 126(12), 9869–9903. https://doi.org/10.1007/s11192-021-04039-1 Liberatore, A., Bowkett, E., MacLeod, C. J., Spurr, E., & Longnecker, N. (2018). Social Media as a Platform for a Citizen Science Community of Practice. Citizen Science: Theory and Practice. https://doi.org/10.5334/cstp.108 MacRoberts, M. H., & MacRoberts, B. R. (2018). The mismeasure of science: Citation analysis. Journal of the Association for Information Science and Technology, 69(3), 474– 482. https://doi.org/https://doi.org/10.1002/asi.23970 Martin, B. R., & Irvine, J. (1983). Assessing Basic Research: Some Partial Indicators of Scientific Progress in Radio Astronomy. Research Policy, 12(2), 61–90. https://doi.org/10.1016/0048-7333(83)90005-7 Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. University of Chicago Press. Merton, R. K. (1988). The Matthew Effect in Science, II: Cumulative Advantage and the Symbolism of Intellectual Property. Isis, 79(4), 606–623. https://doi.org/10.1086/354848 Mirowski, P. (2018). The future(s) of open science. Social Studies of Science, 48(2), 171–203. https://doi.org/10.1177/0306312718772086 Moed, H. F. (2005). Citation analysis in research evaluation. Springer. 25 Moed, H. F., & Garfield, E. (2004). In basic science the percentage of “authoritative” references decreases as bibliographies become shorter. Scientometrics, 60(3), 295–303. https://doi.org/10.1023/B:SCIE.0000034375.39385.84 Mongeon, P. (2018). Using social and topical distance to analyze information sharing on social media. Proceedings of the Association for Information Science and Technology, 55(1), 397–403. https://doi.org/https://doi.org/10.1002/pra2.2018.14505501043 Morton, S. (2015). Progressing research impact assessment: A ‘contributions’ approach. Research Evaluation, 24(4), 405–419. https://doi.org/10.1093/reseval/rvv016 NASEM. (2017). Communicating Science Effectively: A Research Agenda. Washington. https://doi.org/https://doi.org/10.17226/23674. Newman, M. E. J. (2004). Coauthorship networks and patterns of scientific collaboration. Proceedings of the National Academy of Sciences of the United States of America, 101(suppl 1), 5200–5205. https://doi.org/10.1073/pnas.0307545100 Norström, A. V, Cvitanovic, C., Löf, M. F., West, S., Wyborn, C., Balvanera, P., Bednarek, A. T., Bennett, E. M., Biggs, R., de Bremond, A., Campbell, B. M., Canadell, J. G., Carpenter, S. R., Folke, C., Fulton, E. A., Gaffney, O., Gelcich, S., Jouffray, J.-B., Leach, M., … Österblom, H. (2020). Principles for knowledge co-production in sustainability research. Nature Sustainability, 3(3), 182–190. https://doi.org/10.1038/s41893-019-0448-2 Orduña-Malea, E., & Costas, R. (2023). A Scientometric-inspired framework to analyze EurekAlert! Press releases. In I. Broer, S. Lemke, A. Mazarakis, I. Peters, & Z.-W. Christian (Eds.), The science-media interface: on the relation between internal and external science communication (pp. 1–27). De Gruyter. Orduña-Malea, E., & Lopezosa, C. (2024). Uncovering the potential of Twitch as a source for social media metrics. First Monday, 29(1 SE-). https://doi.org/10.5210/fm.v29i1.13214 Pearce, W., Holmberg, K., Hellsten, I., & Nerlich, B. (2014). Climate Change on Twitter: