The societal impact of Open Science–a scoping review

1 The societal impact of Open Science–a scoping review Keywords: societal impact, social impact, open science, citizen science, participatory research, open access Nicki Lisa Cole*, Eva Kormann, Thomas Klebel, Simon Apartis and Tony Ross-Hellauer *Author for correspondence ([email protected]). †Present address: Open and Reproducible Research Group, Sandgasse 34, 8010, Graz, Austria Abstract Open Science (OS) aims, in part, to drive greater societal impact of academic research. Government, funder and institutional policies state that it should further democratise research and increase learning and awareness, evidence-based policy-making, the relevance of research to society’s problems, and public trust in research. Yet, measuring the societal impact of OS has proven challenging and synthesised evidence of it is lacking. This study fills this gap by systematically scoping the existing evidence of societal impact driven by OS and its various aspects, including Citizen Science (CS), Open Access (OA), Open/FAIR Data (OFD), Open Code/Software, and others. Using the PRISMA Extension for Scoping Reviews and searches conducted in Web of Science, Scopus, and relevant grey literature, we identified 196 studies that contain evidence of societal impact. The majority concern CS, with some focused on OA, and only a few addressing other aspects. Key areas of impact found are education and awareness, climate and environment, and social engagement. We found no literature documenting evidence of the societal impact of OFD and limited evidence of societal impact in terms of policy, health, and trust in academic research. Our findings demonstrate a critical need for additional evidence and suggest practical and policy implications. 1. Introduction Recent decades have seen increased efforts, on the part of research funders, institutions and governmental organisations to foster, monitor and demonstrate the impacts of funded research beyond the academy (1–3). Within Europe, both the United Kingdom (UK) (4) and the Netherlands assess the societal impact of research alongside other quality criteria (1), and the European Commission (EC) has placed considerable focus on societal impact in recent funding Framework Programmes (1). The Horizon Europe (HE) Framework Programme includes a focus on ensuring that funded research addresses European Union (EU) policy priorities and global challenges, delivers benefits and impact through research and innovation missions, and strengthens the uptake of research and innovation in society (5). Similar policy approaches are taken in numerous other countries (6). For example, Austria aims to foster the societal impact of research through its “open innovation” policy and funding for Citizen Science and participatory research, while the Netherlands offers funding for societal engagement in research through participatory and community-based models (6). Beyond Europe, societal impact of research is monitored by the Australian government (1), and in the USA, federal legislation urges higher education institutions (HEIs) and funders to create “broader impacts” of research, including “societal benefits” (7) and the US Office of Science and Technology Policy (OSTP) includes a dedicated Science and Society team that aims to “ensur[e] all of America can participate in, contribute to, and benefit from science and technology” (8). Science policy in India aims to drive societal impact with the encouragement of science-society connections in a variety of ways, including funding for community-based research (6). The adoption of Open Science (OS) policies by research institutions and funders globally aligns with the aim of fostering societal impact. OS is both a set of practices and a research reform movement that aims to make academic research (henceforth, ‘research’) more transparent, inclusive and accessible (9). It includes diverse practices like Open Access (OA) to research publications and Open and FAIR Data (OFD); the creation and use of Open Code and Software; practising process transparency through pre-registration (specifying a 2 research plan in advance and publishing it in a publicly accessible registry (See 10)) and Open Methods; and evaluation transparency through Open Evaluation (e.g., Open Peer Review); and Citizen Science (CS), cocreation, participatory research, and collaboration (11,12). Collectively, through these practices, OS aims to make research and the knowledge it generates freely accessible and useful outside of academia, to make research processes more collaborative and efficient, to create an open infrastructure system to support and enable open practices and free access, and to create new ways of assessing the value of research which break with traditional metrics (13). Implicit in these aims is the belief that OS can yield greater societal impact as compared with ‘closed’ research. UNESCO’s definition of OS includes the concept of benefitting society, and in its Recommendation on Open Science, it asserts that increasing openness should “enhanc[e] the social impact of science and increas[e] the capacity of society as a whole to solve complex interconnected problems” (11). The European Commission’s (EC) policy on OS includes the assertion that OS makes research “more responsive to society’s needs” (14). This open approach to research and innovation is intended to best support the pursuit of solutions to “societal challenges”, or in other words, achieve maximum societal benefit. Yet, to date, there is limited evidence as to whether OS policies and practices are achieving this goal. While ‘research impact’ is a vague concept that lacks a coherent and consistent definition (15), some attempts have been made to define societal impact through literature reviews on the topic (1,4). As stated in a literature review from the Ludwig Boltzmann Gesellschaft (LBG), “Societal impact [...] focuses on the effects and changes that research activities unfold beyond academia in other areas of life such as society, culture, public services, health or the environment,” and can include changes in practice, policy and legislation, and to awareness, understanding and individual knowledge and skills (1). A review by Bornmann does not result in a concise definition, but lists similar aspects to the LBG review and includes economic impacts (4). Critical to the LBG definition of societal impact is that it is demonstrable; e.g. there is evidence that research outputs are used in policy-making or that they inform the improvement of healthcare delivery, among others. Adding nuance, the authors assert that societal impact may be instrumental, like the examples listed in the prior sentence, conceptual (e.g. changes to awareness, understanding or perspective), or attitudinal or cultural (e.g. behavioural changes). It can also take the form of capacity building, such as long-term impacts that manifest through knowledge, skills gain, or the development of relationships between diverse stakeholders (1). Additionally, Bornmann notes that societal impact “is not a short-term phenomenon”, but rather “only becomes apparent in the distant future” (4). Other “hallmarks” of it include that it can be either anticipated or unanticipated, within or outside of the intended area, geographically limited or global (4). Extending this prior work within the PathOS project 1 , for which this review was conducted, we define impact (generally) as “long-lasting, elementary and wide-spread change” and understand that it can be “direct or indirect, intended or unintended, [and] relate to behavioural and/or systemic changes” (16). Further, we conceptualise societal impact as a composite of multiple things. This includes and is not limited to 1) social impact (contribution to community welfare, quality of life, behaviour, practices and activities of and relationships among and between people and groups), 2) cultural impact (contribution to understanding of ideas and reality, values and beliefs), 3) political impact (contribution to how policy makers act and how policies are constructed, and to governance and administration of society), 4) environmental impact (contribution to the management of the environment, for example, natural resources, environmental pollution, climate and meteorology), and 5) health impact (contribution to public health, life expectancy, prevention of illnesses and disease, community safety) (16). We consider economic impact from OS separate from societal impact, and note that our colleagues investigated it, and academic impact, through separate scoping reviews (17). We recognise that measuring societal impact is difficult, primarily because of challenges related to causality, but also due to a host of other issues (2). For example, as Wehn et al. (18) point out in their review of impact 1 PathOS (Open Science Impact Pathways) is a Horizon Europe project aiming to collect concrete evidence of Open Science effects and study the pathways of Open Science practices, from input to output, outcome and impact. https://pathos-project.eu/ 3 assessment procedures for CS, there is a lack of standardisation for assessing impact. We envision the impact process as a sequence of events: from inputs to a given system (e.g., more OS practices within academia) to immediate outputs (more OA literature, more CS projects), to further impacts beyond the initial system (e.g., increased trust in research or use of research outputs in policy) (16). The fundamental problem when trying to identify a causal effect of OS on societal impact is that one must compare two situations and study how impacts change: one where the increase in OS practices takes place, and one where it does not. Unfortunately, only in rare circumstances is it possible to observe both situations (e.g., in carefully controlled experiments). Methods to estimate causal effects absent of controlled experiments exist, but require careful reasoning and sometimes strong assumptions. For this reason, we sometimes need to acknowledge that a causal effect cannot be identified, and restrain ourselves from drawing too strong conclusions for policy or advice. While lots of work has been done by institutions, funders and even publishers to measure progress in the uptake or implementation of OS, much less exists to systematically monitor its societal impact, likely due in part to the difficult nature of establishing causality. In a review focused on broad impacts of OA (academic, economic and societal) by Tennant et al. (19), the discussion of societal impact is largely speculative in nature, centring the argument that OA results in increased public engagement with research outputs, absent any evidence to support this claim. Much research exists based on altmetrics, which are designed to measure the presence of published research outside of academia and are taken as an indicator of societal impact (20–22). Some studies measure the difference in altmetrics (a composite of citations in policy documents, mentions in news or blog posts, social media attention, references in Wikipedia, and readership) between closed and OA publications to indicate the societal impact of OA specifically. We consider these later in this paper. However, altmetrics are questioned as indicators of societal impact for various aspects, e.g., for not giving information on who is engaging or limited coverage (23). There are some studies that investigate the societal impact of CS broadly, like a survey conducted by Von Goenner et al. (24) that found evidence for societal impact stemming from participation, including knowledge and skills acquisition, increased self-efficacy, and a greater interest in science, and some reviews that document the societal impact of CS in focused ways. For example, Aristeidou and Herodotou reviewed research focused on, and documented, changes in learning, awareness and science literacy stemming from online CS (25). Bonney et al. (26) conducted a review of literature focused on similar impacts specific to their connection to data collection and processing within CS projects and presented similar results. Walker et al.’s (27) review documented a wide range of both positive and negative impacts derived specifically from CS conducted within water science, including democratisation of science, benefits to resource monitoring and management, increased awareness and scientific literacy, increases in social, political and human capital, as well as negative impacts to livelihood and health and safety, among others. Seeking to foster standardisation of how the impact of CS is studied, Wehn et al. (18) propose categorising it as impact in society, economy, environment, science and technology, and/or governance (we interpret society, environment, and governance as aspects of societal impact) and offer a wide range of indicators that can be used to measure CS-driven impact. Yet, we note that the evidence of societal impact and methods for measuring it discussed here are limited to CS, and do not necessarily indicate ways of documenting and measuring societal impact of OS as a whole. Responding to this gap in the literature, we follow the PRISMA Extension for Scoping Reviews methodology (PRISMA-Scr) (28) to systematically scope, critically appraise, consolidate and valorise evidence from the existing literature that demonstrates societal impact of OS generally and its various aspects, including OA, Open and FAIR/Data, Open Methods, Open Code/Software, Open Evaluation, and CS. We pose the primary research question (RQ1): What evidence exists in the literature regarding the effect of OS on the societal impact of research? In addition, we pose the following secondary research questions: ● SRQ1: What types of positive or negative, direct or indirect societal impact are observed? ● SRQ2: What kinds of mechanisms produce them? ● SRQ3: What specific enabling and/or inhibiting factors (drivers and barriers) are associated with these impacts? ● SRQ4: What knowledge gaps emerge from this analysis? 4 2. Methods Following identification of the above research questions, the study proceeded in four steps: identify relevant studies, select eligible studies, extract data from relevant studies, and summarise and report the results. The study protocol was pre-registered on 31st October 2022 (29) and an addendum detailing the grey literature and snowball search procedures was published on 29th June 2023 (30), both of which provide deeper methodological detail and are published on the Open Science Framework (OSF) platform. For any changes to what was set out in these documents, see Supplement 1. 2.1. Identifying relevant studies We sought to identify all studies presenting evidence (positive or negative) regarding the direct or indirect societal impacts of OS, across all categories of OS practices and types of impact. A search was first conducted for published peer-reviewed literature in the general cross-disciplinary databases Web of Science (WoS) (all databases) and Scopus published between January 2000 and 8th November 2022 (the date of both searches). Search strings were constructed iteratively via keyword/synonym identification and pilot testing. Terms used for types of impact were refined by the team through prioritising keywords for impact labels applicable generally (engag*, educat*, trust, etc.). We next assessed, given our resources, the maximum manageable number of titles we could screen, and then added new issue-specific keywords (health*, environment*/climat*, covid*/coronavirus) one by one, trialling the search to determine if our maximum number of titles was reached as we went. Since these latter keywords were intended as a supplement to the more general impact keywords, our aim was not to be exhaustive (e.g., including farming or emergencies, for example, would still leave out other domains where science has societal impact and hence potentially OS impact). The COVID-19 pandemic was perhaps the major emergency of our time when we conceived this study, with much discourse on the impact of OS in responding to it. We hence decided to prioritise this as a potential area of societal impact. The keywords ultimately used to compose the following search strings, beyond OS terms, include: ● Societal impact ● Trust ● Education/understanding ● Engagement ● Government policy ● Sustainable Development Goals ● Environment/climate ● Health ● COVID ● Participation Search in both databases took place on 8th Nov 2022 using the following query details: Web of Science (All Databases) – 6478 results (TI= ("open scien*" OR "science 2.0" OR "open data" OR "FAIR data" OR "open access" OR ("open code" OR "open software" OR "open tool*") OR “open method*” OR "citizen science" OR "open peer review" OR "open metric*" ) OR AB= ("open scien*" OR "science 2.0" OR "open data" OR "FAIR data" OR ("open code" OR "open software" OR "open tool*") OR “open method*” OR "citizen science" OR "open peer review" OR "open metric*" OR “open access publ*” OR “open access paper*” OR “open access journal*” OR “open access book*”) ) AND TS =((impact* OR effect* OR outcome*) AND (engag* OR educat* OR trust OR polic* OR (sdg OR "sustainable development goal*") OR (gender* OR diversit*) OR participat* OR health* OR (environment* OR climat*) OR (covid* OR coronavirus*))) 5 Timespan: 2000-2022. Databases: WOS, BCI, BIOSIS, CCC, DIIDW, KJD, MEDLINE, RSCI, SCIELO Search language = English Scopus – 6793 results TITLE-ABS ("open scien*" OR "science 2.0" OR "open data" OR "FAIR data" OR ("open access" W/1 publ* OR paper* OR journal* OR book*) OR ("open code" OR "open software" OR "open tool*") OR “open method*” OR "citizen science" OR "open peer review" OR "open metric*") OR TITLE ("open access") AND TITLE-ABS-KEY ((impact* OR effect* OR outcome*) AND (engag* OR educat* OR trust OR polic* OR (sdg OR "sustainable development goal*") OR (gender* OR diversit*) OR participat* OR health* OR (environment* OR climat*) OR (covid* OR coronavirus*))) AND (PUBYEAR > 1999) AND ( LIMIT-TO ( LANGUAGE,"English" ) ) In the second phase of this study, we used “snowball search” to analyse citations to and from included studies, as well as a systematic grey literature search of websites of relevant OS stakeholders (e.g., EC, OECD, UNESCO, etc.), to identify a further 1742 potentially relevant studies. Detailed documentation for both searches, code for the snowball search, and data are included in the data package shared with this paper (31). 2.2. Selection of eligible studies The searches of WoS and Scopus yielded 13,271 total results. Title and abstract screening were guided by the PRISMA-ScR checklist (see Supplement 2) and mapped using the PRISMA-P chart (Figure 1). The following inclusion criteria were used: ● Articles on the societal impact of OS (including OA 2 , Open/FAIR Data 3 , Open Methods, Open Code/Software, CS, Open Evaluation) ● Conducted internationally or nationally ● Published from 1 January 2000 until the date of search ● Text in English ● Full-text available ● Study is either a research article, review article, conference paper, or other peer-reviewed output, or a grey literature study from a recognised stakeholder ● Study reports evidence of OS societal impact ● All methodologies (quantitative, qualitative, mixed, etc.) are eligible These criteria were applied in both title/abstract and full-text screening phases. Following an initial screening pass of titles to remove obvious false positives, followed by merging and de-duplication, 4,514 records remained from the original search of peer-reviewed literature. Two researchers then conducted title/abstract screening, with the first researcher coding either ‘yes’, ‘no’ or ‘unsure’ for inclusion, and the second researcher then reviewing all entries judged ‘unsure’ to decide inclusion. At this stage, we also recorded the aspects of Open Science (OA, OFD, etc.) to which the included studies were most relevant. 2 We did not explicitly include preprints as part of our definition of OA published materials, nor did we specifically include them as a separate category. However, some of the evidence that met our inclusion criteria included discussion of preprints, which we include in the OA subsection of our results. 3 We excluded Open Government Data (OGD) from this study because our focus is on the societal impact of OS practices within academic research. We understand OGD to be data made open by government ministries and offices. In contrast, OFD as we use it refers to data made open by academic researchers. We include here academic research conducted at government funded organisations, like NASA and CERN. 6 Figure 1. PRISMA-P flow diagram. *Indicates that there is a data set available at http://doi.org/10.5281/zenodo.10559446 (31) for this step of the process (grey literature records are only available after deduplication with n = 40). 7 After this, 453 of 472 studies sought were retrieved for full-text screening. All reasonable efforts were made to obtain full-texts, including inter-library loan and emailing authors. Full-texts were imported into a shared Zotero folder. Following full-text screening by one researcher, 153 total studies remained for inclusion from the initial search. In the snowballing and grey literature phase, 265 studies remained after title/abstract screening, and 43 after full-text screening. Hence, a total of 196 relevant studies were identified for inclusion in this Scoping Review. 2.3. Extracting the data Data extraction for studies included from the initial search was conducted using a collaborative Excel file shared via Microsoft Teams and carried out according to the data extraction form illustrated in Table 1. The studies were assigned to individual co-authors for extraction based on the provisional assignment to which aspect of OS they were primarily relevant. Intermittent checks on data extraction quality were performed by the lead author and feedback discussed within the team. Later, screening and data extraction for the snowballed and grey literature were conducted using the same inclusion criteria and extraction form, but carried out in the Systematic Review Facility (SyRF) online platform (32). 4 2.4. Summarising and reporting the results Data extraction results were collated within two Excel files shared on the Microsoft Teams platform and categorised by aspect(s) of OS (one for the initial database search, one for grey literature and snowballed sources). Co-authors were then assigned to summarise and report results narratively in a shared Google document. For aspects of OS that had many papers, co-authors also summarised the data by societal impact aspect. The team then collaborated to refine this initial narrative and to present it in the form of this paper. 3. Results 3.1. Overview We found 196 papers to be in scope (153 from the original academic literature search, and 43 from grey literature and the academic snowball search). Of these, the vast majority provided evidence of the societal impact of CS (163 papers, 83.2% of OS type instances (Figure 2)), across a wide variety of types of societal impact (see Figure 3). Twenty-eight papers demonstrated the societal impact of OA, with impacts including public engagement with scientific literature, use in policy-making, and health-related outcomes. Beyond OA, our search revealed limited evidence of the societal impact of OS. We identified three papers that speak to the impacts of OS in general and two that demonstrate the public health impacts of Open Code/Software. We found no literature with evidence of societal impact from Open Methods, Open Evaluation, or Open/FAIR Data, despite several papers suggesting to do so (see the discussion section). 4 We became aware of SyRF while completing the first phase of this study. After assessing that using it would not alter our methods or results, we decided to implement it for the second phase due to an interest in trialling a dedicated and free-to-use review platform. 8 Figure 2. Number of papers by type of OS (% of all papers). Figure 3. Number of papers by type of impact (% of all papers). Education and awareness (112 or 57.1% of papers) and climate and environment (96 or 49.0% of papers) were by far the most commonly evidenced types of impact within our data. Other common types of impact evidenced by our study include social engagement (between citizens and scientists/other stakeholders, with scientific/project outcomes, and with the broader community) (63 or 32.1% of papers), and policy and governance (50 or 25.5% of papers). Less common but also present in the literature are evidence of impacts in terms of equity and empowerment (36 or 18.4% of papers), health (33 or 16.8% of papers), trust in and attitudes toward research (14 or 7.1% of papers), and privacy/ethics (1 or 0.5% of papers). Looking at the trends within OS aspects (see Table 2), we found that the majority of papers within CS demonstrate impact in terms of education and awareness (112 or 68.7% of papers), and climate and environment (96 or 58.9% of papers). Frequently, these impacts overlap, with studies demonstrating impacts in education and awareness that pertain to climate and environmental topics. The literature shows that CS also creates impact through fostering social engagement (40 or 24.5% of papers), in the realms of policy and governance (45 or 27.6% of papers) and health (29 or 17.8% of papers), fostering equity and empowerment (36 or 22.1% of papers), and by creating trust in research and impacting attitudes to it (12 or 7.4% of papers). We found no literature with rigorous evidence of societal impact in terms of diversity or gender. Papers that demonstrate the societal impact of OA publishing primarily show this in terms of engagement (with OA texts) (22 or 78.6% of papers), but also through policy and governance (5 or 17.9% of papers), and health (2 or 7.1% of papers). One paper provides evidence in terms of privacy/ethics and we found no papers with evidence of OA impact on climate and environment, or education and awareness. In what follows, we present detailed findings within the OS aspects of CS, OA, Open Code/Software, and OS General. 9 3.2. Societal impact of Citizen Science 1. Education and awareness As shown in Table 2, the greatest number of papers within CS provide evidence of impact in terms of education and awareness. These impacts were studied across a range of CS projects and programs, from those in educational settings (from primary school through university) to crowd-sourcing, to communitybased initiatives, and across the globe. Most studies in this category used a preand post-test methodology (typically surveys, but sometimes also interviews) to evaluate changes to participants’ level of subject knowledge, understanding of science and the scientific process, scientific thinking, and/or scientific skills. 5 Figure 4. Number and percentage of studies by type of educational impact. The majority of these papers demonstrate societal impact by documenting changes in CS participants’ subject knowledge and skills associated with the topic of the program or project in question (79 papers, see Table 3). Others demonstrated changes in participants’ general scientific knowledge and skills (33 papers), changes in participants’ interest in studying science or pursuing a scientific career (13 papers) 6 , changes in community knowledge and/or awareness where the project or program was situated (10 papers), and changes in communication and organising skills (2 papers). Nearly all of this evidence indicates positive changes (see Table 3), though a rare few show no impact or mixed results. Shinbrot et al. (33) found a limited impact on knowledge development in an environmental CS program in Mexico. While Vitone et al. (34) established a positive relationship between participation and interest in science and the project subject matter, they found no correlation between participation and subject matter learning. Raddick et al. (35) and Meschini et al. (36) found no learning gain when CS participation was brief and limited in the nature of participation (in an online galaxy classification project and a tourism-based CS program, respectively). Though Meschini et al. (36) found higher levels of environmental awareness three years after participating in a tourism-based CS program, they did not find evidence of specific knowledge retention. And, while Jordan et al. (37) found increases in subject learning, they did not find a gain in scientific knowledge. Further, both Derrien et al. (38) and Stewart et al. (39) found no impact of participation on interest in pursuing a scientific career. The findings of some studies indicate elements of CS initiatives that lead to positive impacts. Mady et al. (40) found that higher degrees of participation in an ornithological CS program led to greater increases in knowledge, and that these were highest when participants were involved in data collection. Similarly, both Phillips (41) and Ballard et al. (42) found that hands-on experience in the research process and interaction with project materials fostered learning, while sustained, long-term participation was found by Bedessem et al. (43) to result in increased scientific skills and by Kloetzer et al. (44), to positive subject learning outcomes. 5 In this section we present summarised results due to the high volume of literature discussed. Table 3 details all literature described herein with references. 6 Throughout this paper we use the term ‘research’ to refer to academic research rather than ‘science’ to be inclusive of academic disciplines and research fields that may not be considered ‘scientific’. In this section, we use the term ‘science’ because we are reporting trends in the literature, and ‘science’ is the term used in this literature. 16 environmental impact of CS is influenced by whether or not the project or programme responded to a community need for data (79,125,138,140,141,143–146) and the extent to which policymakers and administrators are willing to accept this data and have mechanisms in place for using it (24,140,143). Cutting across impact types, a project or programme being driven by community need is an enabling factor for impact (45,126,138,143,148,151,166,168). Evidence for other OS aspects is more limited, but some findings pertaining to OA suggest that the type of OA (Green vs. Gold) (195,196), the specific social media platform or website (e.g., 198,199,203), and clear signalling of OA status (212) are factors which influence social engagement with OA outputs (possibly in interaction with research fields). While our findings demonstrate a wide variety of societal impacts derived from OS practices, they also illuminate considerable knowledge gaps (SRQ4). Strikingly, the evidence we gathered is concentrated around CS, and further, mostly focused on impacts derived from participation in CS, rather than those derived from the research generated by CS (though some evidence of this does exist). Elucidating the challenges of assessing the impact of CS, Wehn et al. (18) point out that those leading CS projects may find it challenging to measure mediumand long-term impacts due to the disconnect between the timeline of such work and project funding structures. Assessing impacts necessarily comes after a project has ended, which means the funding linked to the project has ceased to be available. This aligns with the conceptualisations of societal impact offered by the LBG (1) and Bornmann (2), which emphasise the longterm nature of impacts and the process of tracking them. Additionally, Wehn et al. (18) found through their review of CS impact assessment studies that project priorities and a lack of assessment competences within the project team are barriers to this kind of work. Further, as mentioned above, our review returned limited evidence of the societal impact of OA and other OS aspects. We imagine that Wehn et al.’s (18) observations extend to assessing the societal impact of projects that produce other OS outputs, such as Open Code or Software, as well. From this, we might suggest that once broader frameworks for assessment of OS impact are in place (discussed below), funding instruments may make specific provisions for projects to ensure consistent data collection to facilitate longer-term impact assessment. Also striking is the sheer absence of evidence of societal impact derived from Open/FAIR Data within the surveyed literature. Throughout this study, we considered 250 texts focused on OFD (after title screening) that represented a diversity of research areas and aims but found that any claims of societal impact were speculative rather than based on observed and documented usage. For example, possible impacts included potential privacy violations (227,228), improvements in health research (229–231), or better monitoring of SDGs (232), yet evidence to back these claims was not presented. It is important to note, as stated in our Methods section, that we excluded Open Government Data (OGD) from our study. Our study focused on OS practices within academic research, and therefore societal impact from OGD was out of scope. We note, however, that there does appear to already be substantial literature focused on the societal and economic impact of this type of open data (we caught much of it in our initial search of the academic literature). Considering the methodologies deployed to study it may prove instructive for new research into the societal impact of Open/FAIR (academic) data. While case studies are often employed due to the complexity of impact assessment (See, e.g., 233,234), attempts have also been made to quantify the societal impact of OGD, for example, by the Open Data Barometer 8 , which mainly uses expert surveys on topics such as environmental impact or the inclusion of marginalised groups as indicators for societal impact. Overall, it appears that the evidence included in this study is concentrated in areas where establishing evidence of OS societal impact is less challenging due to established methodologies or datasets. The majority of our evidence is generated through CS projects and programmes and focused on learning impacts because there are established methods for conducting preand post-test surveys with participants and communities and these can be done with participants from any CS initiative. Additionally, there is considerable evidence of climate and environmental impact from CS because CS is an established approach to responding to problems that fall within these realms, by, for example, generating needed but missing monitoring data or pushing back on community-level environmental injustices. Similarly, there are established methods and 8 https://opendatabarometer.org/ 17 workflows for tracing OA publication references, online engagement with them, and online interactions about them; therefore, numerous studies can harness and make use of altmetrics data (questionable though the veracity of societal impact as measured by this indicator may be). Much more challenging is tracing the usage and societal impact of OFD and Open Code/Software. A lack of consistent referencing practices for these resources across academic disciplines and research fields makes it extremely challenging to understand usage and impact within academia, and the societal impact that may stem from research that uses these resources. And, while one might be able to classify those who view and download open resources based on IP address or other user details, this would still be several steps away from creating evidence of use and societal impact. In addition, delineating which contributions to Open Source projects come from academia is also difficult since (especially larger OS projects) often are the outcome of contributions from diverse contributors. Many researchers no doubt contribute to Linux, for example, but it was started by someone who did not continue in academia beyond their master's degree (235). The big tech players also invest heavily in OS development. Even projects which are used intensively within research (e.g., Sci-Kit Learn in Machine Learning 9 ) are often developed by non-academics (Sci-Kit Learn was developed at Google Summer of Code by a non-academic (236)). That separating out contributions from the academic research community, as opposed to others, is problematic may also explain why we did not identify any distinct evidence on this within our reviewed literature. Our study reveals that knowledge gaps also exist around causation. While some of the evidence included here is causal, i.e., there is an established causal relationship between an OS practice and a type of societal impact, the majority of the evidence included in this study is correlational. For example, among all the included studies on the societal impact of OA, only two out of 28 used a research design permitting causal claims, while all others were observational in nature. More research on causal relationships between OS interventions, activities, outcomes and impact is therefore needed to meet the institutional and governmental desire to monitor the impact of OS (See Klebel and Traag (237) on how to incorporate causal thinking into empirical studies on science). Yet, establishing causality in this sense is not necessarily about establishing linearity. As Wehn et al. (18) point out, establishing causality may involve both “intermediary outcomes and impacts within a given domain” as well as “between outcomes in different domains.” Similarly, Coulson et al. (238) documented through research that “pathways to change and impact are opened by enabling the step from awareness to action,” pointing to a step-wise approach fostering societal impact and to tracking it in a causal way (239). Further, Wehn et al. (18) caution against creating “impact silos” in assessment work, where a lack of awareness exists between interrelated interventions, outcomes, and impacts that may occur across various domains (e.g., between social changes and environmental impacts). We emphasise, though, that this work should not only be done on an ad-hoc project basis, for reasons established by Wehn et al. (18). While some projects may have the funding, capacity and time to establish evidenced-based indicators of societal impact, there is need for more broad-based impact monitoring frameworks to do this work. At present, infrastructures for OS monitoring whether national (e.g., French Open Science Monitor 10 ) or international (e.g., COKI’s Open Access Dashboard 11 ) seem to focus primarily on monitoring the uptake of OS practices. Yet, monitoring whether more researchers are publishing more OA, or sharing more research data, while important, does not answer the original aims motivating the implementation of OS – whether scientific publications are used by more diverse audiences (e.g., academic periphery, lay publics, industry); whether OFD is being used to fuel scientific or economic innovation; whether fostering transparency and sharing is raising standards of quality in research. OS monitoring frameworks and infrastructures must, as OS become mainstream, begin to answer such questions. The PathOS project, for which this review was conducted, aims to contribute to this work by establishing evidence-based, causal impact pathways for OS through modelling and case study implementation. We situate the intended contributions of this project alongside those of others who have already demonstrated 9 https://scikit-learn.org/ 10 https://www.ouvrirlascience.fr/the-open-science-monitor/ 11 https://open.coki.ac/) 18 that it is possible, for example, to link CS outcomes to SDG indicator monitoring (240), for co-created community level indicators to measure both short and long-term impacts of CS initiatives (238), and that there are evidenced-based principles for assessing CS impact (18) that can reasonably be extended to assessing the societal impact of OS broadly speaking. In anticipating and contributing towards the further development of frameworks and infrastructures, we especially look forward to the work of the UNESCO working group on OS monitoring (241). Though there is considerable work to be done in this area, we believe that the results of our study fill an important gap in the literature. Though most of the evidence that we found demonstrates short-term rather than long-term impact, our findings validate and expand upon the more focused and subject-specific reviews of societal impacts driven by CS (25–27), and build on Tennant’s (19) prior review of OA by demonstrating that evidence of societal impact from OA publishing remains lacking, eight years later. And, though the evidence outside of CS is limited, we also demonstrate that multiple aspects of OS can contribute to the same categories of societal impact. We note that this study, while intended to be a wide-reaching synthesis of published evidence of societal impact of OS, does have some limitations. Included studies are limited by language and (possibly) publication venue (due to the use of exclusive academic databases for the initial search). The parameters of our search did not overtly include other OS practices, like preprints, preregistration, open analysis, and open collaboration, therefore we may have missed evidence of societal impact stemming from these. In addition, our search string — although covering widely used general terms for types of impact (e.g., trust, education, etc.) – included only some keywords related to distinct domains or issues (e.g., health, climate, COVID) but not others such as farming or emergencies. We acknowledge that these pragmatic choices (made with the aim of keeping the number of included titles within a manageable amount given available resources) might mean that our search strategy failed to capture all impact studies (if they did not use more general terms for impact within their titles or abstracts). We further note that both qualitative research and arts and humanities have low representation within the corpus of literature included in this study, therefore evidence of societal impact stemming from OS within these realms may have been missed. We recognize that publication bias toward positive results is a known problem within scientific research, and therefore expect that we may be missing evidence of null or negative societal impact. And importantly, we acknowledge our authorship team’s collective positionality as white Europeans has shaped our research process such that our conceptualization of societal impact and evidence of it may not be as robust and nuanced as it could be. 5. Conclusion In sum, there is considerable evidence within academic and grey literature of the societal impact of OS, but it is almost entirely derived from studies focused on the impact of CS, and heavily concentrated on providing evidence of impact in terms of education and awareness, climate and environment, and social engagement. A few studies focused on OA, Open Code/Software, and OS general also show some positive (and some negative) societal impacts, but the veracity of societal impact as measured by altmetrics – the majority of the OA literature, is questionable. We are also able to conclude that certain mechanisms and enabling factors lead to societal impact from OS, while certain inhibiting factors get in the way of it. The results of this study will prove instructive to academic research institutions, funders, publishers, science policymakers, researchers, educators and the general public. There is clear evidence that CS produces a wide variety of beneficial societal impacts, and evidence that signalling OS practices and deploying Open Code/Software in response to societal needs also produces impact. Therefore, investing in these practices is a wise choice for leaders and researchers who wish to foster the societal impact of scientific research. For educators, the evidence that CS fosters learning outcomes and interest in science suggests that the integration of CS within educational settings across age groups is a productive practice. For the general public, in particular people, groups and communities who wish to generate solutions to problems they experience, our findings suggest that CS is a pathway to do so. CS projects and programmes need not be top-down, created by researchers, but can originate at the grassroots and have impact, as our evidence indicates (for example (148,151). 19 Our findings indicate that additional research is needed to study the societal impact of OS beyond CS, and that more precise and in-depth research is needed to truly establish the societal impact of OA. To date, to our knowledge, wide scale surveys of the use of OS resources by the general public in nations around the world have not been conducted. Such an approach could provide missing foundational knowledge of which societal actors are using OS resources in which ways and might identify disparities in use that have implications in terms of equity. We also believe that building on large scale quantitative research with indepth qualitative research with users of OS resources could prove instructive in illuminating causal relationships in OS pathways to impact. Acknowledgments The authors gratefully thank Vincent Traag for his valuable comments on earlier drafts of this article and the two anonymous reviewers who provided invaluable constructive feedback on the first submitted version. Any errors remain entirely the authors’ own. Funding Statement This work was supported by the project PathOS, funded by the European Commission under the Horizon Europe programme (grant no. 101058728) The Know-Center is funded within COMET—Competence Centers for Excellent Technologies—under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry for Digital and of Economic Affairs, the Austrian Research Promotion Agency (FFG), and by the province of Styria. The COMET programme is managed by the FFG. Data Accessibility Datasets and additional materials supporting this article are published on Zenodo, doi:10.5281/zenodo.10559446 (http://doi.org/10.5281/zenodo.10559446). Competing Interests We have no competing interests. Authors' Contributions NLC: Data curation, formal analysis, investigation, methodology, project administration, supervision, validation, visualisation, writing (original draft), writing (editing and review) EK: Data curation, formal analysis, investigation, software, validation, visualisation, writing (original draft), writing (editing and review) TK: Conceptualization, data curation, formal analysis, investigation, methodology, project administration, software, validation, visualisation, writing (editing and review) SA: Formal analysis, investigation, writing (original draft), writing (editing and review) TRH: Conceptualization, formal analysis, funding acquisition, investigation, methodology, supervision, writing (original draft), writing (editing and review) 20 References 1. What is societal impact of research? A literature review [Internet]. Vienna, Austria: Ludwig Boltzmann Gesellschaft; 2021 Apr [cited 2023 Dec 14] p. 19. Available from: https://ois.lbg.ac.at/wpcontent/uploads/sites/24/2022/01/Literature-Review_Societal-Research-Impact.pdf 2. Bornmann L. Measuring the societal impact of research. EMBO Rep [Internet]. 2012 Aug [cited 2023 Dec 14];13(8):673–6. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3410397/ 3. Bührer S, Feidenheimer A, Walz R, Lindner R, Beckert B, Wallwaey E. Concepts and methods to measure societal impacts – an overview. (74). 4. Bornmann L. What is societal impact of research and how can it be assessed? a literature survey. J Am Soc Inf Sci Technol [Internet]. 2013 [cited 2024 Feb 14];64(2):217–33. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.22803 5. Directorate-General for Research and Innovation (European Commission), Nixon J. Study to support the monitoring and evaluation of the framework programme for research and innovation along key impact pathways: indicator methodology and metadata handbook [Internet]. LU: Publications Office of the European Union; 2022 [cited 2023 Dec 14]. Available from: https://data.europa.eu/doi/10.2777/44653 6. Wehn U, Hepburn L. Guidance for the implementation of the UNESCO Open Science Recommendation re. ‘Opening science to society’ [Internet]. Zenodo; 2022 Nov [cited 2024 Apr 29]. Available from: https://zenodo.org/records/7464276 7. America COMPETES Reauthorization Act of 2010 [Internet]. 5116 Jan 4, 2011. Available from: https://www.govinfo.gov/app/details/PLAW-111publ358 8. The White House [Internet]. [cited 2024 Feb 14]. Science and Society | OSTP. Available from: https://www.whitehouse.gov/ostp/ostps-teams/science-and-society/ 9. Field SM. Charting the Constellation of Science Reform [Internet]. [Groningen, Netherlands]: University of Groningen; 2022 [cited 2024 Jan 17]. Available from: https://osf.io/udfw4 10. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci [Internet]. 2018 Mar 13 [cited 2024 May 16];115(11):2600–6. Available from: https://www.pnas.org/doi/10.1073/pnas.1708274114 11. UNESCO. UNESCO Recommendation on Open Science [Internet]. Paris, France: UNESCO; 2021 Nov [cited 2022 Sep 19] p. 34. Available from: https://unesdoc.unesco.org/ark:/48223/pf0000379949.locale=en 12. What is open science? [Internet]. [cited 2023 Dec 14]. Available from: https://www.cos.io/open-science 13. Fecher B, Friesike S. Open Science: One Term, Five Schools of Thought. In: Bartling S, Friesike S, editors. Opening Science [Internet]. Cham: Springer International Publishing; 2014 [cited 2020 May 14]. p. 17–47. Available from: http://link.springer.com/10.1007/978-3-319-00026-8_2 14. European Commission Fact Sheet on Open Science [Internet]. European Commission, DirectorateGeneral for Research and Innovation; 2019 [cited 2023 Dec 14] p. 3. Available from: https://researchand-innovation.ec.europa.eu/system/files/2019-12/ec_rtd_factsheet-open-science_2019.pdf 15. Reed MS, Ferré M, Martin-Ortega J, Blanche R, Lawford-Rolfe R, Dallimer M, et al. Evaluating impact from research: A methodological framework. Res Policy [Internet]. 2021 May 1 [cited 2024 Jan 17];50(4):104147. Available from: https://www.sciencedirect.com/science/article/pii/S0048733320302225 16. Dekker R, Karasz I, Stoy L. PathOS - D1.1 Open Science Intervention Logic [Internet]. Zenodo; 2023 Apr [cited 2023 Aug 24]. Available from: https://zenodo.org/record/7801286 17. Klebel T, Cole NL, Tsipouri L, Kormann E, Karasz I, Liarti S, et al. PathOS - D1.2 Scoping Review of Open Science Impact [Internet]. Zenodo; 2023 May [cited 2023 Aug 24]. Available from: https://zenodo.org/record/7883699 18. Wehn U, Gharesifard M, Ceccaroni L, Joyce H, Ajates R, Woods S, et al. Impact assessment of citizen science: state of the art and guiding principles for a consolidated approach. Sustain Sci [Internet]. 2021 Sep [cited 2023 Feb 23];16(5):1683–99. Available from: https://link.springer.com/10.1007/s11625-02100959-2 19. Tennant JP, Waldner F, Jacques DC, Masuzzo P, Collister LB, Hartgerink ChrisHJ. The academic, economic and societal impacts of Open Access: an evidence-based review. F1000Research [Internet]. 21 2016 Apr 11 [cited 2023 Feb 5];5:632. Available from: https://f1000research.com/articles/5-632/v1 20. Adie E, Roe W. Altmetric: enriching scholarly content with article‐level discussion and metrics. Learn Publ [Internet]. 2013 Jan [cited 2023 Dec 18];26(1):11–7. Available from: https://onlinelibrary.wiley.com/doi/10.1087/20130103 21. Priem J, Groth P, Taraborelli D. The Altmetrics Collection. Ouzounis CA, editor. PLoS ONE [Internet]. 2012 Nov 1 [cited 2023 Dec 18];7(11):e48753. Available from: https://dx.plos.org/10.1371/journal.pone.0048753 22. Holmberg K, Bowman S, Bowman T, Didegah F, Kortelainen T. What Is Societal Impact and Where Do Altmetrics Fit into the Equation? J Altmetrics [Internet]. 2019 Dec 18 [cited 2023 Aug 14];2(1):6. Available from: https://journalofaltmetrics.org/articles/10.29024/joa.21/ 23. Thelwall M. The pros and cons of the use of altmetrics in research assessment. 2 [Internet]. 2020 May 12 [cited 2024 Jan 23]; Available from: https://wlv.openrepository.com/handle/2436/623132 24. Von Gönner J, Herrmann TM, Bruckermann T, Eichinger M, Hecker S, Klan F, et al. Citizen science’s transformative impact on science, citizen empowerment and socio-political processes. Socio-Ecol Pract Res [Internet]. 2023 Mar [cited 2023 Aug 14];5(1):11–33. Available from: https://link.springer.com/10.1007/s42532-022-00136-4 25. Aristeidou M, Herodotou C. Online Citizen Science: A Systematic Review of Effects on Learning and Scientific Literacy. Citiz Sci Theory Pract [Internet]. 2020 Apr 7 [cited 2023 Feb 23];5(1):11. Available from: http://theoryandpractice.citizenscienceassociation.org/articles/10.5334/cstp.224/ 26. Bonney R, Phillips TB, Ballard HL, Enck JW. Can citizen science enhance public understanding of science? Public Underst Sci [Internet]. 2016 Jan [cited 2023 Feb 23];25(1):2–16. Available from: http://journals.sagepub.com/doi/10.1177/0963662515607406 27. Walker D, Smigaj M, Tani M. The benefits and negative impacts of citizen science applications to water as experienced by participants and communities. WIREs Water [Internet]. 2021 Jan [cited 2023 Feb 23];8(1). Available from: https://onlinelibrary.wiley.com/doi/10.1002/wat2.1488 28. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med [Internet]. 2018 Oct 2 [cited 2023 Apr 4];169(7):467–73. Available from: https://www.acpjournals.org/doi/10.7326/M18-0850 29. Ross-Hellauer T, Klebel T, Pitelis A. Protocol for Scoping Reviews of (1) Academic, (2) Societal and (3) Economic impacts of Open Science. 2022 Oct 31 [cited 2022 Nov 2]; Available from: https://osf.io/gm57c 30. Cole NL, Klebel T, Ross-Hellauer T. Addendum to Protocol for Scoping Reviews of (1) Academic, (2) Societal and (3) Economic impacts of Open Science: PROTOCOL FOR GREY LITERATURE SEARCH AND INDEXING OF SOURCES, AND FOR SCREENING AND DATA CHARTING IN THE SYRF PLATFORM [Internet]. Open Science Framework; 2023 [cited 2024 Jan 22]. Available from: https://osf.io/https://osf.io/3b6xj 31. Kormann E, Cole NL, Klebel T, Apartis S, Ross-Hellauer T. Data from ‘The societal impact of Open Science - a scoping review’ [Internet]. Zenodo; 2024 [cited 2024 Jan 24]. Available from: https://zenodo.org/records/10559446 32. Bahor Z, Liao J, Currie G, Ayder C, Macleod M, McCann SK, et al. Development and uptake of an online systematic review platform: the early years of the CAMARADES Systematic Review Facility (SyRF). BMJ Open Sci [Internet]. 2021 Mar 31 [cited 2024 Feb 14];5(1). Available from: http://access.portico.org/stable?au=phzmqrt8kdx 33. Shinbrot XA, Jones KW, Solomon J, Escobedo MR. Examining the influence of citizen science participation on individual volunteers in the global South: A case study of hydrologic monitors in Veracruz, Mexico. J Environ Educ [Internet]. 2022 Jan 2 [cited 2023 Feb 23];53(1):6–21. Available from: https://www.tandfonline.com/doi/full/10.1080/00958964.2021.1984193 34. Vitone T, Stofer K, Steininger MS, Hulcr J, Dunn R, Lucky A. School of Ants goes to college: integrating citizen science into the general education classroom increases engagement with science. J Sci Commun [Internet]. 2016 Jan 21 [cited 2023 Feb 23];15(01):A03. Available from: https://jcom.sissa.it/archive/15/01/JCOM_1501_2016_A03 35. Raddick MJ, Prather EE, Wallace CS. Galaxy zoo: Science content knowledge of citizen scientists. Public Underst Sci [Internet]. 2019 Aug [cited 2023 Feb 23];28(6):636–51. Available from: http://journals.sagepub.com/doi/10.1177/0963662519840222 36. Meschini M, Prati F, Simoncini GA, Airi V, Caroselli E, Prada F, et al. Environmental Awareness 22 Gained During a Citizen Science Project in Touristic Resorts Is Maintained After 3 Years Since Participation. Front Mar Sci [Internet]. 2021 Feb 19 [cited 2023 Feb 23];8:584644. Available from: https://www.frontiersin.org/articles/10.3389/fmars.2021.584644/full 37. Jordan RC, Gray SA, Howe DV, Brooks WR, Ehrenfeld JG. Knowledge Gain and Behavioral Change in Citizen-Science Programs: Citizen-Scientist Knowledge Gain. Conserv Biol [Internet]. 2011 Dec [cited 2023 Feb 23];25(6):1148–54. Available from: https://onlinelibrary.wiley.com/doi/10.1111/j.15231739.2011.01745.x 38. Derrien MM, Zuidema C, Jovan S, Bidwell A, Brinkley W, López P, et al. Toward Environmental Justice in Civic Science: Youth Performance and Experience Measuring Air Pollution Using Moss as a Bio-Indicator in Industrial-Adjacent Neighborhoods. Int J Environ Res Public Health [Internet]. 2020 Oct 5 [cited 2023 Aug 14];17(19):7278. Available from: https://www.mdpi.com/1660-4601/17/19/7278 39. Stewart K, Castaneda DI, Azadeh Bolhari PE. Work in Progress: Citizen scientists’ description of an engineer. In 2020. 40. Mady RP, Phillips TB, Bonter DN, Quimby C, Borland J, Eldermire C, et al. Engagement in the Data Collection Phase of the Scientific Process is Key for Enhancing Learning Gains. Citiz Sci Theory Pract [Internet]. 2023 Mar 29 [cited 2023 Aug 14];8(1):14. Available from: https://theoryandpractice.citizenscienceassociation.org/article/10.5334/cstp.594/ 41. Phillips TB, Ballard HL, Lewenstein BV, Bonney R. Engagement in science through citizen science: Moving beyond data collection. Sci Educ [Internet]. 2019 May [cited 2023 Aug 14];103(3):665–90. Available from: https://onlinelibrary.wiley.com/doi/10.1002/sce.21501 42. Ballard HL, Dixon CGH, Harris EM. Youth-focused citizen science: Examining the role of environmental science learning and agency for conservation. Biol Conserv [Internet]. 2017 Apr [cited 2023 Feb 23];208:65–75. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0006320716302051 43. Bedessem B, Torres AC, Fontaine C, Deguines N. Science learning in biodiversity citizen science: Inputs from the analysis of online social interactions within a contributory project for pollinators’ monitoring. Biol Conserv [Internet]. 2022 Dec 1 [cited 2023 Aug 14];276:109807. Available from: https://www.sciencedirect.com/science/article/pii/S0006320722003603 44. Kloetzer L, Lorke J, Roche J, Golumbic Y, Winter S, Jõgeva A. Learning in Citizen Science. In: Vohland K, Land-Zandstra A, Ceccaroni L, Lemmens R, Perelló J, Ponti M, et al., editors. The Science of Citizen Science [Internet]. Cham: Springer International Publishing; 2021 [cited 2023 Aug 14]. p. 283–308. Available from: https://link.springer.com/10.1007/978-3-030-58278-4_15 45. Dickinson JL, Shirk J, Bonter D, Bonney R, Crain RL, Martin J, et al. The current state of citizen science as a tool for ecological research and public engagement. Front Ecol Environ [Internet]. 2012 Aug [cited 2023 Feb 23];10(6):291–7. Available from: http://doi.wiley.com/10.1890/110236 46. Frigerio D, Puehringer-Sturmayr V, Neuböck-Hubinger B, Gegendorfer G, Kotrschal K, Hirschenhauser K. Monitoring public awareness about the endangered northern bald ibis: a case study involving primary school children as citizen scientists. PeerJ [Internet]. 2019 Sep 11 [cited 2023 Feb 23];7:e7569. Available from: https://peerj.com/articles/7569 47. Adamou A, Georgiou Y, Paraskeva-Hadjichambi D, Hadjichambis ACh. Environmental Citizen Science Initiatives as a Springboard towards the Education for Environmental Citizenship: A Systematic Literature Review of Empirical Research. Sustainability [Internet]. 2021 Dec 11 [cited 2023 Feb 23];13(24):13692. Available from: https://www.mdpi.com/2071-1050/13/24/13692 48. Aivelo T, Huovelin S. Combining formal education and citizen science: a case study on students’ perceptions of learning and interest in an urban rat project. Environ Educ Res [Internet]. 2020 Mar 3 [cited 2023 Aug 14];26(3):324–40. Available from: https://www.tandfonline.com/doi/full/10.1080/13504622.2020.1727860 49. Allen BL. Strongly Participatory Science and Knowledge Justice in an Environmentally Contested Region. Sci Technol Hum Values [Internet]. 2018 Nov [cited 2023 Aug 14];43(6):947–71. Available from: http://journals.sagepub.com/doi/10.1177/0162243918758380 50. Araújo JL, Morais C, Paiva JC. Student participation in a coastal water quality citizen science project and its contribution to the conceptual and procedural learning of chemistry. Chem Educ Res Pract [Internet]. 2022 [cited 2023 Feb 23];23(1):100–12. Available from: http://xlink.rsc.org/?DOI=D1RP00190F 51. Asingizwe D, Poortvliet PM, Van Vliet AJH, Koenraadt CJM, Ingabire CM, Mutesa L, et al. What do people benefit from a citizen science programme? Evidence from a Rwandan citizen science 23 programme on malaria control. Malar J [Internet]. 2020 Dec [cited 2023 Aug 14];19(1):283. Available from: https://malariajournal.biomedcentral.com/articles/10.1186/s12936-020-03349-8 52. Ballard HL, Robinson LD, Young AN, Pauly GB, Higgins LM, Johnson RF, et al. Contributions to conservation outcomes by natural history museum-led citizen science: Examining evidence and next steps. Biol Conserv [Internet]. 2017 Apr [cited 2023 Feb 6];208:87–97. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0006320716303512 53. Branchini S, Meschini M, Covi C, Piccinetti C, Zaccanti F, Goffredo S. Participating in a Citizen Science Monitoring Program: Implications for Environmental Education. Bernardi G, editor. PLOS ONE [Internet]. 2015 Jul 22 [cited 2023 Feb 23];10(7):e0131812. Available from: https://dx.plos.org/10.1371/journal.pone.0131812 54. Bremer S, Haque MM, Aziz SB, Kvamme S. ‘My new routine’: Assessing the impact of citizen science on climate adaptation in Bangladesh. Environ Sci Policy [Internet]. 2019 Apr [cited 2023 Feb 23];94:245–57. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1462901118309365 55. Brossard D, Lewenstein B, Bonney R. Scientific knowledge and attitude change: The impact of a citizen science project. Int J Sci Educ [Internet]. 2005 Jan [cited 2023 Feb 23];27(9):1099–121. Available from: http://www.tandfonline.com/doi/abs/10.1080/09500690500069483 56. Carson S, Rock J, Smith J. Sediments and Seashores - A Case Study of Local Citizen Science Contributing to Student Learning and Environmental Citizenship. Front Educ [Internet]. 2021 Jun 14 [cited 2023 Feb 23];6:674883. Available from: https://www.frontiersin.org/articles/10.3389/feduc.2021.674883/full 57. Chase SK, Levine A. Citizen Science: Exploring the Potential of Natural Resource Monitoring Programs to Influence Environmental Attitudes and Behaviors. Conserv Lett [Internet]. 2018 [cited 2023 Aug 14];11(2):e12382. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/conl.12382 58. Christoffel R. How Participation in Citizen Science Projects Impacts Individuals. In: THIRTEEN How Participation in Citizen Science Projects Impacts Individuals [Internet]. University of California Press; 2020 [cited 2023 Feb 23]. p. 185–8. Available from: https://www.degruyter.com/document/doi/10.1525/9780520960473-017/html 59. Cronje R, Rohlinger S, Crall A, Greg N. Does Participation in Citizen Science Improve Scientific Literacy? A Study to Compare Assessment Methods. Appl Environ Educ Commun. 2011 Jul 1;10:135– 45. 60. Damman S, Helness H, Grindvoll ILT, Sun C. Citizen science to enhance evaluation of local wastewater treatment – a case study from Oslo. Water Sci Technol [Internet]. 2019 May 15 [cited 2023 Feb 23];79(10):1887–96. Available from: https://iwaponline.com/wst/article/79/10/1887/67735/Citizenscience-to-enhance-evaluation-of-local 61. Dem ES, Rodríguez-Labajos B, Wiemers M, Ott J, Hirneisen N, Bustamante JV, et al. Understanding the relationship between volunteers’ motivations and learning outcomes of Citizen Science in rice ecosystems in the Northern Philippines. Paddy Water Environ [Internet]. 2018 Oct [cited 2023 Feb 23];16(4):725–35. Available from: http://link.springer.com/10.1007/s10333-018-0664-9 62. Diprose G, Greenaway A, Moorhouse B. Making Visible More Diverse Nature Futures through Citizen Science. Citiz Sci Theory Pract [Internet]. 2022 Feb 24 [cited 2023 Feb 23];7(1):6. Available from: http://theoryandpractice.citizenscienceassociation.org/articles/10.5334/cstp.442/ 63. Ekman K. INFORMAL LEARNING IN AN ONLINE CITIZEN SCIENCE COMMUNITY. EDULEARN19 Proc [Internet]. 2019 [cited 2023 Feb 23];5144–51. Available from: https://library.iated.org/view/EKMAN2019INF 64. English PB, Richardson MJ, Garzón-Galvis C. From Crowdsourcing to Extreme Citizen Science: Participatory Research for Environmental Health. Annu Rev Public Health [Internet]. 2018 Apr 1 [cited 2023 Feb 23];39(1):335–50. Available from: https://www.annualreviews.org/doi/10.1146/annurevpublhealth-040617-013702 65. Forrester TD, Baker M, Costello R, Kays R, Parsons AW, McShea WJ. Creating advocates for mammal conservation through citizen science. Biol Conserv [Internet]. 2017 Apr [cited 2023 Aug 14];208:98–105. Available from: https://linkinghub.elsevier.com/retrieve/pii/S000632071630249X 66. Greving H, Bruckermann T, Schumann A, Straka TM, Lewanzik D, Voigt-Heucke SL, et al. Improving attitudes and knowledge in a citizen science project about urban bat ecology. Ecol Soc [Internet]. 2022 [cited 2023 Feb 23];27(2):art24. Available from: https://www.ecologyandsociety.org/vol27/iss2/art24/ 67. Groulx M, Brisbois MC, Lemieux CJ, Winegardner A, Fishback L. A Role for Nature-Based Citizen 24 Science in Promoting Individual and Collective Climate Change Action? A Systematic Review of Learning Outcomes. Sci Commun [Internet]. 2017 Feb [cited 2023 Feb 23];39(1):45–76. Available from: http://journals.sagepub.com/doi/10.1177/1075547016688324 68. Hadjichambi D, Hadjichambis ACh, Adamou A, Georgiou Y. A systematic literature review of K-12 environmental Citizen Science (CS) initiatives: Unveiling the CS pedagogical and participatory aspects contributing to students’ environmental citizenship. Educ Res Rev [Internet]. 2023 May 1 [cited 2023 Aug 14];39:100525. Available from: https://www.sciencedirect.com/science/article/pii/S1747938X23000180 69. Haywood BK, Parrish JK, Dolliver J. Place-based and data-rich citizen science as a precursor for conservation action: Citizen Science and Conservation Action. Conserv Biol [Internet]. 2016 Jun [cited 2023 Feb 23];30(3):476–86. Available from: https://onlinelibrary.wiley.com/doi/10.1111/cobi.12702 70. Hiller SE, Kitsantas A. The Effect of a Horseshoe Crab Citizen Science Program on Middle School Student Science Performance and STEM Career Motivation: Citizen Science Career Motivation. Sch Sci Math [Internet]. 2014 Oct [cited 2023 Aug 14];114(6):302–11. Available from: https://onlinelibrary.wiley.com/doi/10.1111/ssm.12081 71. Hollow B, Roetman PEJ, Walter M, Daniels CB. Citizen science for policy development: The case of koala management in South Australia. Environ Sci Policy [Internet]. 2015 Mar [cited 2023 Feb 23];47:126–36. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1462901114001993 72. Hoover E. ‘We’re not going to be guinea pigs;’ Citizen Science and Environmental Health in a Native American Community. J Sci Commun [Internet]. 2016 Jan 21 [cited 2023 Feb 23];15(01):A05. Available from: https://jcom.sissa.it/archive/15/01/JCOM_1501_2016_A05 73. Hsu CH, Chang YM, Liu CC. Can Short-Term Citizen Science Training Increase Knowledge, Improve Attitudes, and Change Behavior to Protect Land Crabs? Sustainability [Internet]. 2019 Jul 18 [cited 2023 Feb 23];11(14):3918. Available from: https://www.mdpi.com/2071-1050/11/14/3918 74. Isley CF, Fry KL, Sharp EL, Taylor MP. Bringing citizen science to life: Evaluation of a national citizen science program for public benefit. Environ Sci Policy [Internet]. 2022 Aug [cited 2023 Feb 23];134:23– 33. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1462901122001022 75. Johnson MF, Hannah C, Acton L, Popovici R, Karanth KK, Weinthal E. Network environmentalism: Citizen scientists as agents for environmental advocacy. Glob Environ Change [Internet]. 2014 Nov [cited 2023 Feb 23];29:235–45. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0959378014001733 76. Kelly R, Fleming A, Pecl GT, von Gönner J, Bonn A. Citizen science and marine conservation: a global review. Philos Trans R Soc B Biol Sci [Internet]. 2020 Dec 21 [cited 2023 Feb 23];375(1814):20190461. Available from: https://royalsocietypublishing.org/doi/10.1098/rstb.2019.0461 77. Kermish-Allen R, Peterman K, Bevc C. The utility of citizen science projects in K-5 schools: measures of community engagement and student impacts. Cult Stud Sci Educ [Internet]. 2019 Sep [cited 2023 Feb 23];14(3):627–41. Available from: http://link.springer.com/10.1007/s11422-017-9830-4 78. Kerr S. Citizen science and deep mapping for climate communication: A report on CHICC. J Community Archaeol Herit [Internet]. 2022 Mar 15 [cited 2023 Feb 23];1–8. Available from: https://www.tandfonline.com/doi/full/10.1080/20518196.2022.2051139 79. Kleitou P, Rees S, Cecconi F, Kletou D, Savva I, Cai LL, et al. Regular monitoring and targeted removals can control lionfish in Mediterranean Marine Protected Areas. Aquat Conserv Mar Freshw Ecosyst [Internet]. 2021 Oct [cited 2023 Feb 23];31(10):2870–82. Available from: https://onlinelibrary.wiley.com/doi/10.1002/aqc.3669 80. Kobori H, Dickinson JL, Washitani I, Sakurai R, Amano T, Komatsu N, et al. Citizen science: a new approach to advance ecology, education, and conservation. Ecol Res [Internet]. 2016 Jan [cited 2023 Aug 14];31(1):1–19. Available from: http://doi.wiley.com/10.1007/s11284-015-1314-y 81. Lakomý M, Hlavová R, Machackova H. Open Science and the Science-Society Relationship. Society [Internet]. 2019 Jun 15 [cited 2023 Feb 6];56(3):246–55. Available from: http://link.springer.com/10.1007/s12115-019-00361-w 82. Land-Zandstra AM, Devilee JLA, Snik F, Buurmeijer F, van den Broek JM. Citizen science on a smartphone: Participants’ motivations and learning. Public Underst Sci [Internet]. 2016 Jan [cited 2023 Feb 23];25(1):45–60. Available from: http://journals.sagepub.com/doi/10.1177/0963662515602406 83. Locritani M, Merlino S, Abbate M. Assessing the citizen science approach as tool to increase awareness on the marine litter problem. Mar Pollut Bull [Internet]. 2019 Mar [cited 2023 Feb 23];140:320–9. 25 Available from: https://linkinghub.elsevier.com/retrieve/pii/S0025326X19300335 84. Lüsse M, Brockhage F, Beeken M, Pietzner V. Citizen science and its potential for science education. Int J Sci Educ [Internet]. 2022 May 3 [cited 2023 Feb 23];44(7):1120–42. Available from: https://doi.org/10.1080/09500693.2022.2067365 85. Lynch-O’Brien LI, Babchuk WA, Dauer JM, Heng-Moss T, Golick D. Transference of Citizen Science Program Impacts: A Theory Grounded in Public Participation in Scientific Research. Diversity [Internet]. 2021 Jul 25 [cited 2023 Feb 23];13(8):339. Available from: https://www.mdpi.com/14242818/13/8/339 86. Marchante E, Marchante H. Engaging Society to Fight Invasive Alien Plants in Portugal—One of the Main Threats to Biodiversity. In: Castro P, Azeiteiro UM, Bacelar-Nicolau P, Leal Filho W, Azul AM, editors. Biodiversity and Education for Sustainable Development [Internet]. Cham: Springer International Publishing; 2016 [cited 2023 Feb 23]. p. 107–22. (World Sustainability Series). Available from: http://link.springer.com/10.1007/978-3-319-32318-3_8 87. Marks L, Laird Y, Trevena H, Smith BJ, Rowbotham S. A Scoping Review of Citizen Science Approaches in Chronic Disease Prevention. Front Public Health [Internet]. 2022 May 9 [cited 2023 Feb 6];10:743348. Available from: https://www.frontiersin.org/articles/10.3389/fpubh.2022.743348/full 88. Meixner T, Berkowitz AR, Downey AE, Pillich J, LeVea R, Smith BK, et al. Rapid Assessment and Long-Term Monitoring of Green Stormwater Infrastructure with Citizen Scientists. Sustainability [Internet]. 2021 Nov 12 [cited 2023 Feb 23];13(22):12520. Available from: https://www.mdpi.com/20711050/13/22/12520 89. Merenlender AM, Crall AW, Drill S, Prysby M, Ballard H. Evaluating environmental education, citizen science, and stewardship through naturalist programs. Conserv Biol [Internet]. 2016 Dec [cited 2023 Aug 14];30(6):1255–65. Available from: https://onlinelibrary.wiley.com/doi/10.1111/cobi.12737 90. Nursey-Bray M, Palmer R, Pecl G. Spot, log, map: Assessing a marine virtual citizen science program against Reed’s best practice for stakeholder participation in environmental management. Ocean Coast Manag [Internet]. 2018 Jan [cited 2023 Feb 23];151:1–9. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0964569117305665 91. Peter M, Diekötter T, Kremer K. Participant Outcomes of Biodiversity Citizen Science Projects: A Systematic Literature Review. Sustainability [Internet]. 2019 May 15 [cited 2023 Feb 23];11(10):2780. Available from: https://www.mdpi.com/2071-1050/11/10/2780 92. Peter M, Diekötter T, Kremer K, Höffler T. Citizen science project characteristics: Connection to participants’ gains in knowledge and skills. Silva D de P, editor. PLOS ONE [Internet]. 2021 Jul 15 [cited 2023 Feb 23];16(7):e0253692. Available from: https://dx.plos.org/10.1371/journal.pone.0253692 93. Peter M, Diekötter T, Höffler T, Kremer K. Biodiversity citizen science: Outcomes for the participating citizens. Mascarenhas A, editor. People Nat [Internet]. 2021 Apr [cited 2023 Feb 23];3(2):294–311. Available from: https://onlinelibrary.wiley.com/doi/10.1002/pan3.10193 94. Peters M, Eames C, Hamilton D. The use and value of citizen science data in New Zealand. J R Soc N Z [Internet]. 2015 Jul 3 [cited 2023 Feb 23];45(3):151–60. Available from: http://www.tandfonline.com/doi/full/10.1080/03036758.2015.1051549 95. Popa CL, Dontu SI, Savastru D, Carstea EM. Role of Citizen Scientists in Environmental Plastic Litter Research—A Systematic Review. Sustainability [Internet]. 2022 Oct 15 [cited 2023 Feb 23];14(20):13265. Available from: https://www.mdpi.com/2071-1050/14/20/13265 96. Santori C, Keith RJ, Whittington CM, Thompson MB, Van Dyke JU, Spencer R. Changes in participant behaviour and attitudes are associated with knowledge and skills gained by using a turtle conservation citizen science app. Wehi P, editor. People Nat [Internet]. 2021 Feb [cited 2023 Feb 23];3(1):66–76. Available from: https://onlinelibrary.wiley.com/doi/10.1002/pan3.10184 97. Schaefer T, Kieslinger B, Fabian CM. Citizen-Based Air Quality Monitoring: The Impact on Individual Citizen Scientists and How to Leverage the Benefits to Affect Whole Regions. Citiz Sci Theory Pract [Internet]. 2020 Mar 3 [cited 2023 Feb 23];5(1):6. Available from: http://theoryandpractice.citizenscienceassociation.org/articles/10.5334/cstp.245/ 98. Schläppy ML, Loder J, Salmond J, Lea A, Dean AJ, Roelfsema CM. Making Waves: Marine Citizen Science for Impact. Front Mar Sci [Internet]. 2017 May 16 [cited 2023 Feb 6];4:146. Available from: http://journal.frontiersin.org/article/10.3389/fmars.2017.00146/full 99. Schneiderhan-Opel J, Bogner FX. The Relation between Knowledge Acquisition and Environmental Values within the Scope of a Biodiversity Learning Module. Sustainability [Internet]. 2020 Mar 6 [cited 32 23];58(8):1179–202. Available from: https://onlinelibrary.wiley.com/doi/10.1002/tea.21697 192. Jordan RC, Ballard HL, Phillips TB. Key issues and new approaches for evaluating citizen-science learning outcomes. Front Ecol Environ [Internet]. 2012 Aug [cited 2023 Feb 23];10(6):307–9. Available from: http://doi.wiley.com/10.1890/110280 193. Araujo AC, Vanin AA, Nascimento DP, Gonzalez GZ, Costa LOP. What are the variables associated with Altmetric scores? Syst Rev [Internet]. 2021 Dec [cited 2023 Feb 6];10(1):193. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-021-01735-0 194. Haneef R, Ravaud P, Baron G, Ghosn L, Boutron I. Factors associated with online media attention to research: a cohort study of articles evaluating cancer treatments. Res Integr Peer Rev [Internet]. 2017 Dec [cited 2023 Aug 14];2(1):9. Available from: http://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-017-0033-z 195. Clayson PE, Baldwin SA, Larson MJ. The open access advantage for studies of human electrophysiology: Impact on citations and Altmetrics. Int J Psychophysiol [Internet]. 2021 Jun 1 [cited 2023 Feb 6];164:103–11. Available from: https://www.sciencedirect.com/science/article/pii/S016787602100101X 196. Long H, Drown L, Amin ME. The effect of open access on scholarly and societal metrics of impact in the ASHA Journals [Internet]. OSF Preprints; 2022 [cited 2023 Jul 5]. Available from: https://osf.io/2ufte/ 197. Yu X, Meng Z, Qin D, Shen C, Hua F. The long-term influence of Open Access on the scientific and social impact of dental journal articles: An updated analysis. J Dent [Internet]. 2022 Apr [cited 2023 Feb 6];119:104067. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0300571222001245 198. Cho J. Altmetrics of Highly Cited Research Papers in Social Science. Ser Rev [Internet]. 2021 Jan 2 [cited 2023 Feb 6];47(1):17–27. Available from: https://www.tandfonline.com/doi/full/10.1080/00987913.2021.1882652 199. Cho J. Altmetrics analysis of highly cited academic papers in the field of library and information science. Scientometrics [Internet]. 2021 Sep 1 [cited 2023 Aug 14];126(9):7623–35. Available from: https://doi.org/10.1007/s11192-021-04084-w 200. Dehdarirad T, Didegah F, Didegah A. Social media visibility of open access versus non-open access articles: A case study of Life Sciences & Biomedicine. In: 17TH INTERNATIONAL CONFERENCE ON SCIENTOMETRICS & INFORMETRICS (ISSI2019), VOL I [Internet]. 2019 [cited 2023 Feb 6]. p. 762–9. Available from: https://research.chalmers.se/en/publication/515327 201. McKiernan EC, Bourne PE, Brown CT, Buck S, Kenall A, Lin J, et al. How open science helps researchers succeed. Rodgers P, editor. eLife [Internet]. 2016 Jul 7 [cited 2022 Nov 18];5:e16800. Available from: https://doi.org/10.7554/eLife.16800 202. Wang X, Liu C, Mao W, Fang Z. The open access advantage considering citation, article usage and social media attention. Scientometrics [Internet]. 2015 May [cited 2023 Feb 6];103(2):555–64. Available from: http://link.springer.com/10.1007/s11192-015-1547-0 203. Maleki A. PubMed and ArXiv vs. Gold Open Access: Citation, Mendeley, and Twitter Uptake of Academic Articles of Iran. In Istanbul; 2015 [cited 2023 Feb 6]. Available from: https://www.semanticscholar.org/paper/PubMed-and-ArXiv-vs.-Gold-Open-Access%3ACitation%2C-of-Maleki/cd8c83a9eeb3abe88b2c1cb5acc905cd2de85071 204. Hadad S, Aharony N. Open Access Advantages as a Function of the Discipline: Mixed-methods Study. J Acad Librariansh [Internet]. 2023 Jul [cited 2023 Aug 14];49(4):102746. Available from: https://linkinghub.elsevier.com/retrieve/pii/S009913332300085X 205. Tai TC, Robinson JPW. Enhancing Climate Change Research With Open Science. Front Environ Sci [Internet]. 2018 Oct 11 [cited 2023 Feb 23];6:115. Available from: https://www.frontiersin.org/article/10.3389/fenvs.2018.00115/full 206. Dehdarirad T, Karlsson K. News media attention in Climate Action: latent topics and open access. Scientometrics [Internet]. 2021 Sep [cited 2023 Aug 14];126(9):8109–28. Available from: https://link.springer.com/10.1007/s11192-021-04095-7 207. Schultz T. All the research that’s fit to print: Open access and the news media. Quant Sci Stud [Internet]. 2021 Nov 5 [cited 2023 Aug 14];2(3):828–44. Available from: https://direct.mit.edu/qss/article/2/3/828/102385/All-the-research-that-s-fit-to-print-Open-access 208. Teplitskiy M, Lu G, Duede E. Amplifying the impact of open access: Wikipedia and the diffusion of science. J Assoc Inf Sci Technol [Internet]. 2017 Sep [cited 2023 Feb 23];68(9):2116–27. Available from: 33 https://onlinelibrary.wiley.com/doi/10.1002/asi.23687 209. Taylor M. An altmetric attention advantage for open access books in the humanities and social sciences. Scientometrics [Internet]. 2020 Dec [cited 2023 Feb 23];125(3):2523–43. Available from: https://link.springer.com/10.1007/s11192-020-03735-8 210. Wei M, Noroozi Chakoli A. Evaluating the relationship between the academic and social impact of open access books based on citation behaviors and social media attention. Scientometrics [Internet]. 2020 Dec [cited 2023 Feb 6];125(3):2401–20. Available from: https://link.springer.com/10.1007/s11192020-03678-0 211. Snijder R. Revisiting an open access monograph experiment: measuring citations and tweets 5 years later. Scientometrics [Internet]. 2016 Dec [cited 2023 Feb 6];109(3):1855–75. Available from: http://link.springer.com/10.1007/s11192-016-2160-6 212. Li H, Liu L, Wang X. The open access effect in social media exposure of scholarly articles: A matchedpair analysis. J Informetr [Internet]. 2021 Aug [cited 2023 Feb 6];15(3):101154. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1751157721000250 213. Ozaygen A, Montgomery L, Neylon C, Huang K, Pyne R, Emery C, et al. More Readers in More Places: The benefits of open access for scholarly books [Internet]. Zenodo; 2020 [cited 2023 Oct 5]. Available from: https://zenodo.org/record/4014905 214. Fleerackers A, Chtena N, Pinfield S, Alperin JP, Barata G, Oliveira M, et al. Making science public: a review of journalists’ use of Open Science research. F1000Research [Internet]. 2023 May 18 [cited 2023 Aug 14];12:512. Available from: https://f1000research.com/articles/12-512/v1 215. Vilkins S, Grant WJ. Types of evidence cited in Australian Government publications. Scientometrics [Internet]. 2017 Dec [cited 2023 Feb 23];113(3):1681–95. Available from: http://link.springer.com/10.1007/s11192-017-2544-2 216. Zong Q, Huang Z, Huang J. Can open access increase LIS research’s policy impact? Using regression analysis and causal inference. Scientometrics [Internet]. 2023 Aug 1 [cited 2023 Aug 14];128(8):4825–54. Available from: https://doi.org/10.1007/s11192-023-04750-1 217. Besançon L, Peiffer-Smadja N, Segalas C, Jiang H, Masuzzo P, Smout C, et al. Open science saves lives: lessons from the COVID-19 pandemic. BMC Med Res Methodol [Internet]. 2021 Jun 5 [cited 2023 Feb 6];21(1):117. Available from: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874021-01304-y 218. Hardisty DJ, Haaga DAF. Diffusion of treatment research: does open access matter? J Clin Psychol [Internet]. 2008 Jul [cited 2023 Feb 23];64(7):821–39. Available from: https://onlinelibrary.wiley.com/doi/10.1002/jclp.20492 219. Davis PM, Walters WH. The impact of free access to the scientific literature: a review of recent research. J Med Libr Assoc JMLA [Internet]. 2011 Jul [cited 2023 Aug 14];99(3):208–17. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3133904/ 220. Marshall Z, Brunger F, Welch V, Asghari S, Kaposy C. Open Availability of Patient Medical Photographs in Google Images Search Results: Cross-Sectional Study of Transgender Research. J Med Internet Res [Internet]. 2018 Feb 26 [cited 2023 Feb 6];20(2):e70. Available from: http://www.jmir.org/2018/2/e70/ 221. Zong Q, Huang Z, Huang J. Do open science badges work? Estimating the effects of open science badges on an article’s social media attention and research impacts. Scientometrics [Internet]. 2023 Jun 1 [cited 2023 Aug 14];128(6):3627–48. Available from: https://doi.org/10.1007/s11192-023-04720-7 222. Rosman T, Bosnjak M, Silber H, Koßmann J, Heycke T. Open science and public trust in science: Results from two studies. Public Underst Sci [Internet]. 2022 Nov [cited 2023 Feb 6];31(8):1046–62. Available from: http://journals.sagepub.com/doi/10.1177/09636625221100686 223. Song H, Markowitz DM, Taylor SH. Trusting on the shoulders of open giants? Open science increases trust in science for the public and academics. J Commun [Internet]. 2022 Aug 3 [cited 2023 Aug 14];72(4):497–510. Available from: https://academic.oup.com/joc/article/72/4/497/6623479 224. Bokonda PL, Ouazzani-Touhami K, Souissi N. Open Data Kit: Mobile Data Collection Framework For Developing Countries. Int J Innov Technol Explor Eng [Internet]. 2019 Oct 30 [cited 2023 Feb 23];8(12):4749–54. Available from: https://www.ijitee.org/portfolio-item/L35831081219/ 225. Kobayashi S, Falcón L, Fraser H, Braa J, Amarakoon P, Marcelo A, et al. Using Open Source, Open Data, and Civic Technology to Address the COVID-19 Pandemic and Infodemic. Yearb Med Inform [Internet]. 2021 Aug [cited 2023 Feb 23];30(01):038–43. Available from: http://www.thieme- 34 connect.de/DOI/DOI?10.1055/s-0041-1726488 226. Wehn U, Ajates R, Fraisl D, Gharesifard M, Gold M, Hager G, et al. Capturing and communicating impact of citizen science for policy: A storytelling approach. J Environ Manage [Internet]. 2021 Oct [cited 2023 Feb 23];295:113082. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0301479721011440 227. Zipper SC, Stack Whitney K, Deines JM, Befus KM, Bhatia U, Albers SJ, et al. Balancing Open Science and Data Privacy in the Water Sciences. Water Resour Res [Internet]. 2019 Jul [cited 2023 Feb 23];55(7):5202–11. Available from: https://onlinelibrary.wiley.com/doi/10.1029/2019WR025080 228. O’Callaghan E, Douglas HM. #MeToo Online Disclosures: A Survivor-Informed Approach to Open Science Practices and Ethical Use of Social Media Data. Psychol Women Q [Internet]. 2021 Dec 1 [cited 2023 Feb 23];45(4):505–25. Available from: https://doi.org/10.1177/03616843211039175 229. Alvarez-Romero C, Martinez-Garcia A, Ternero Vega J, Díaz-Jimènez P, Jimènez-Juan C, Nieto-Martín MD, et al. Predicting 30-Day Readmission Risk for Patients With Chronic Obstructive Pulmonary Disease Through a Federated Machine Learning Architecture on Findable, Accessible, Interoperable, and Reusable (FAIR) Data: Development and Validation Study. JMIR Med Inform [Internet]. 2022 Jun 2 [cited 2023 Aug 14];10(6):e35307. Available from: https://medinform.jmir.org/2022/6/e35307 230. Golob JL, Lugogo N, Lauring AS, Lok AS. SARS-CoV-2 vaccines: a triumph of science and collaboration. JCI Insight [Internet]. 2021 May 10 [cited 2023 Feb 23];6(9):e149187. Available from: https://insight.jci.org/articles/view/149187 231. Karaa A, MacMullen LE, Campbell JC, Christodoulou J, Cohen BH, Klopstock T, et al. Community Consensus Guidelines to Support FAIR Data Standards in Clinical Research Studies in Primary Mitochondrial Disease. Adv Genet [Internet]. 2022 Mar [cited 2023 Feb 23];3(1):2100047. Available from: https://onlinelibrary.wiley.com/doi/10.1002/ggn2.202100047 232. Mubea K, Killough B, Seidu O, Kimani J, Mugambi B, Kamara S. Africa Regional Data Cube (ARDC) is Helping Countries in Africa Report on the Sustainable Development Goals (SDGS). In: IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium [Internet]. Waikoloa, HI, USA: IEEE; 2020 [cited 2023 Feb 23]. p. 3379–82. Available from: https://ieeexplore.ieee.org/document/9324156/ 233. Rajão R, Jarke J. The materiality of data transparency and the (re)configuration of environmental activism in the Brazilian Amazon. Soc Mov Stud [Internet]. 2018 May 4 [cited 2023 Feb 23];17(3):318– 32. Available from: https://www.tandfonline.com/doi/full/10.1080/14742837.2018.1459297 234. Van Belle JP, Hlabano M. Building Urban Resilience in the Face of Severe Drought through the Innovative Use of Open Data. In: 2019 4th MEC International Conference on Big Data and Smart City (ICBDSC) [Internet]. Muscat, Oman: IEEE; 2019 [cited 2023 Feb 23]. p. 1–7. Available from: https://ieeexplore.ieee.org/document/8645586/ 235. CHM [Internet]. 2024 [cited 2024 May 16]. Linus Torvalds. Available from: https://computerhistory.org/profile/linus-torvalds/ 236. scikit-learn [Internet]. [cited 2024 May 16]. About us. Available from: https://scikitlearn/stable/about.html 237. Klebel T, Traag V. Introduction to causality in science studies [Internet]. OSF; 2024 [cited 2024 Feb 19]. Available from: https://osf.io/4bw9e 238. Coulson S, Woods M, Hemment D, Scott M, Polvora A, Nascimento S. Report and Assessment of Impact and Policy Outcomes Using Community Level Indicators: H2020 Making Sense Report. 2017 [cited 2024 Apr 29]; Available from: https://www.research.ed.ac.uk/en/publications/report-andassessment-of-impact-and-policy-outcomes-using-communi 239. Coulson S, Woods M, Eu MS. Citizen Sensing: An Action-Orientated Framework for Citizen Science. Front Commun [Internet]. 2021 Apr 16 [cited 2024 Apr 29];6. Available from: https://www.frontiersin.org/articles/10.3389/fcomm.2021.629700 240. Ajates R, Hager G, Georgiadis P, Coulson S, Woods M, Hemment D. Local Action with Global Impact: The Case of the GROW Observatory and the Sustainable Development Goals. Sustainability [Internet]. 2020 Jan [cited 2024 Apr 29];12(24):10518. Available from: https://www.mdpi.com/20711050/12/24/10518 241. Badolato AM. ‘Building an Open Science Monitoring Framework with open technologies’ : UNESCO workshop – 19/12/23 [Internet]. Ouvrir la Science. 2024 [cited 2024 May 16]. Available from: https://www.ouvrirlascience.fr/building-an-open-science-monitoring-framework-with-open- 35 technologies-unesco-workshop-19-12-23 Tables Table 1. Categories extracted from included studies in the data charting process. Heading Description Author Name of author/s Date Date article sourced Title of study Title of the article or study Publication year Year that the article was published Publication type Journal, website, conference, etc. DOI/URL Unique identifier Exclusion Out of scope, non-English, duplicate Justification If a study was deemed to be out of scope, a justification had to be provided. Study details and design (if applicable) Type of study, empirical or review, etc. Notes on methods used in study (whether qualitative or quantitative, which population demographics studied, etc.) Types of data sources included Detail the data sources Study aims Overview of the main objectives of the study Relevance to which aspect of Open Science Open Access, Open/FAIR Data, Open Methods, Citizen Science, Open Evaluation, Open Science General Relevance to which aspect(s) of societal impact Engagement, participation, education, trust, policy, sustainable development goals, gender, diversity, health, climate/environment, COVID-1912 Key findings Noteworthy results of the study that contribute to the scoping review question(s) 12 Societal impact categories were amended throughout the initial data extraction process as it became clear that they did not adequately capture what we were seeing in the literature. Engagement was amended to ‘social engagement’, participation was removed for redundancy with engagement, education was changed to ‘education and awareness’, and equity, empowerment and privacy/ethics were added as additional categories. The full list of categories used in data charting and data analysis is available as Supplement 3 to this paper. 36 Coverage Optional field to note any relevant information about the level of coverage of the study, e.g., only specific countries, disciplines, demographics covered Confidence assessment Optional field to note any concerns about reliability/generalisability of findings (e.g., conflict of interest, potential biases, small sample sizes, or other methodological issues) within the study 37 Table 2. Type of impact by OS type, with number of papers coded per intersection (N) and % of papers within the OS type category. OS type Climate and environment Education and awareness Equity and empowerment Health Policy and governance Privacy/ethics Social engagement Trust and attitudes towards research Citizen Science 58.9% (96) 68.7% (112) 22.1% (36) 17.8% (29) 27.6% (45) 0.0% (0) 24.5% (40) 7.4% (12) Open Access 0.0% (0) 0.0% (0) 0.0% (0) 7.1% (2) 17.9% (5) 3.6% (1) 78.6% (22) 0.0% (0) Open Code 0.0% (0) 0.0% (0) 0.0% (0) 100.0% (2) 0.0% (0) 0.0% (0) 0.0% (0) 0.0% (0) Open Science general 0.0% (0) 0.0% (0) 0.0% (0) 0.0% (0) 0.0% (0) 0.0% (0) 33.3% (1) 66.7% (2) 38 Table 3. Literature with evidence of the impact of CS on education and awareness. Positive impact No impact Negative impact Total studies Changes in subject knowledge and/or related skills Adamou et al. 2021 (47); Aivelo and Houvelin 2020 (48); Allen 2018 (49); Araujo et al. 2022 (50); Aristeidou and Herodotou 2020 (25); Asingizwe et al. 2020 (51); Ballard et al. 2017 (52); Ballard et al. 2017 (42); Branchini et al. 2015 (53); Bremer et al. 2019 (54); Brossard et al. 2005 (55); Carson et al. 2021 (56); Chase and Levine 2018 (57); Christoffel 2020 (58); Cronje et al. 2011 (59); Damman et al. 2019 (60); Dem et al. 2018 (61); Derrien 2020 (38); Diprose et al. 2022 (62); Ekman 2019 (63); English et al. 2018 (64); Forrester et al. 2017 (65); Greving et al. 2022 (66); Groulx et al. 2017 (67); Hadjichambi et al. 2023 (68); Haywood et al. 2016 (69); Hiller and Kitsantas 2014 (70); Hollow et al. 2015 (71); Hoover 2016 (72); Hsu et al. 2019 (73); Isley et al. 2022 (74); Johnson et al. 2014 (75); Jordan et al. 2011 (37); Kelly et al. 2020 (76); Kermish-Allen et al. 2019 (77); Kerr 2022 (78); Kleitou et al. 2021 (79); Kloetzer et al. 2021 (44); Kobori et al. 2016 (80); Lakomy et al. 2019 (81); Land-Zandstra et al. 2016 (82); Locritani et al. 2019 (83); Luesse et al. 2022 (84); Lynch-O’Brien et al. 2021 (85); Mady et al. 2023 (40); Marchante and Marchante 2016 (86); Marks et al. 2022 (87); Meixner et al. 2021 (88); Merenlender et al. 2016 (89); Meschini et al. 2021 (36); Nursey-Bray et al. 2018 (90); Peter et al. 2019 (91); Peter et al. 2021 (92); Peter et al. 2021 (93); Peters et al. 2015 (94); Phillips et al. 2019 (41); Popa et al. 2022 (95); Santori et al. 2021 (96); Schaefer et al. 2020 (97); Schlaeppy et al. 2017 (98); Schneiderhan-Opel and Bogner 2020 (99); Schuttler et al. 2018 (100); Seamans 2018 (101); Seifert et al. 2016 (102); Shaw 2017 (103); Silva et al. 2016 (104); Stepenuck and Green 2015 (105); Turrini et al. 2018 (106); Van Haeften et al. 2021 (107); Varaden et al. 2021 (108); Von Gönner et al. 2023 (24); Walker et al. 2021 (109); Meschini et al. 2021 (36); Raddick et al. 2019 (35); Shinbrot et al. 2022 (33); Vitone et al. 2016 (34) none 79 39 Walker et al. 2021 (27); Williams et al. 2021 (110); Zarybnicka et al. 2017 (111); Zhang et al. 2023 (112) Change in general scientific knowledge and skills Anderson et al. 2020 (113); Aivelo and Huovelin 2020 (48); Ballard et al. 2017 (52); Ballard et al. 2017 (42); Bedessem et al. 2022 (43); Carson et al. 2021 (56); Cho et al. 2021 (114); Christoffel 2020 (58); Conrad and Hilchey 2011 (115); Cronje et al. 2011 (59); da Silva and Heaton 2017 (116); Dickinson et al. 2012 (45); English et al. 2018 (64); Grossberndt et al. 2021 (117); Haywood et al. 2016 (69); Hiller and Kitsantas 2015 (118); Hoekstra et al. 2020 (119); Isley et al. 2022 (74); Johnson et al. 2014 (75); Kloetzer et al. 2021 (44); Lewis and Carson 2021 (120); Luesse et al. 2022 (84); Mady et al. 2023 (40); Merenlender et al. 2016 (89); Peter et al. 2021 (93); Phillips et al. 2019 (41); Price and Lee 2013 (121); Ross-Hellauer et al. 2022 (122); Trumbull et al. 2000 (123); Walker et al. 2021 (109); Walker et al. 2021 (27); Zarybnicka et al. 2017 (111) Jordan et al. 2011 (37) none 33 Change in interest in studying science or pursuing career in science Ballard et al. 2017 (42); Cho et al. 2021 (114); Hiller and Kitsantas 2014 (70); Johnson et al. 2014 (75); Koomen et al. 2019 (124); Luesse et al. 2022 (84); Mahajan et al. 2021 (125); Rosas et al. 2022 (126); Seifert et al. 2016 (102); Vitone et al. 2016 (34); Wallace and Bodzin 2019 (127) Derrien et al. 2020 (38); Stewart et al. 2020 (39) none 13 Change in community knowledge or awareness Asingizwe et al. 2020 (51); Ballard et al. 2017 (42); Costa et al. 2022 (128); Frigerio et al. 2019 (46); Johnson et al. 2014 (75); Mahajan et al. 2022 (129); Schaefer et al. 2020 (97); Shinbrot et al. 2022 (33); Stepenuck and Green 2015 (105); Walker et al. (2021) (27) none none 10 Change in communication and organising skills Bonney et al. 2016 (26); Kloetzer et al. 2021 (44) none none 2 40 Table 4. Literature with evidence of the impact of CS on climate and environment. Positive impact No impact Negative impact Total studies Awareness, attitudes and values Adamou et al. 2021 (47); Ballard et al. 2017 (42); Branchini et al. 2015 (53); Bremer et al. 2019 (54); Carson et al. 2021 (56); Chase and Levine 2018 (57); Cronje et al. 2011 (59); English et al. 2018 (64); Evans et al. 2005 (153); FernandezGimenez et al. 2008 (149); Grossberndt et al. 2021 (117); Groulx et al. 2017 (67); Haywood et al. 2016 (69); Hsu et al. 2019 (73); Johnson et al. 2014 (75); Kelemen-Finan et al. 2018 (154); Kerr 2022 (78); Kleitou et al. 2021 (79); Kloetzer et al. 2021 (44); Lynch-O’Brien et al. 2021 (85); Mahajan et al. 2021 (125); Mahajan et al. 2022 (129); Marchante and Marchante 2016 (86); Meschini et al. 2021 (36); Ostermann-Miyashita et al. 2021 (131); Popa et al. 2022 (95); Sandhaus et al. 2018 (150); Schneiderhan-Opel and Bogner 2020 (99); Schuttler et al. 2018 (100); Shinbrot et al. 2022 (33); Stepenuck and Green 2015 (105); Toomey and Domroese 2013 (155); Torres et al. 2023 (156); Walker et al. 2021 (109); Walker et al. 2021 (27); West et al. 2020 (152) Forrester et al. 2017 (65) none 37 Behaviour change Adamou et al. 2021 (47); Day et al. 2022 (135); Deguines et al. 2020 (132); Evans et al. 2005 (153); Fulton et al. 2019 (140); Gotor et al. 2021 (133); Grossberndt et al. 2021 (117); Groulx et al. 2017 (67); Hadjichambi et al. 2023 (68); Haywood et al. 2016 (69); Hodgkinson et al. 2022 (157); Lewandowski and Oberhauser 2017 (134); Lynch-O’Brien et al. 2021 (85); Mahajan et al. 2021 (125); Mahajan et al. 2022 (129); Marchante and Marchante 2016 (86); Peter et al. 2019 (91); Popa et al. 2022 (95); Rodriguez et al. 2019 (136); Sandhaus Jordan et al. 2011 (37); Popa et al. 2022 (95) none 28 41 et al. 2018 (150); Santori 2021 (96); Spellman et al. 2021 (158); Stepenuck and Green 2015 (105); Toomey and Domroese 2013 (155); Vasiliades et al. 2021 (159); Walker et al. 2021 (109); Walker et al. 2021 (27) Conservation Aceves-Bueno et al. 2015 (137); Ballard et al. 2017 (52); Ballard et al. 2017 (42); Chiaravalloti et al. 2022 (138); Crow et al. 2020 (160); Day et al. 2022 (135); Earp and Liconti 2020 (145); Fulton et al. 2019 (140); Haywood et al. 2022 (69); Hsu et al. 2019 (73); Hyder et al. 2015 (147); Johnson et al. 2014 (75); Mwango’mbe et al. 2021 (141); Pecorelli et al. 2019 (161); Sandhaus et al. 2018 (150); Santori et al. 2021 (96); Schlaeppy et al. 2017 (98); Skrbinsek et al. 2019 (162); Soroye et al. 2022 (142); Zhang et al. 2023 (112) none none 20 Biodiversity Branchini et al. 2015 (53); Carson et al. 2021 (56); Deguines et al. 2020 (132); Dem et al. 2018 (61); Earp and Liconti 2022 (145); Fraisl et al. 2020 (163); Hyder et al. 2015 (147); Jordan et al. 2011 (37); Kelemen-Finan et al. 2018 (154); Kleitou et al. 2021 (79); Lee et al. 2021 (164); Lynch-O’Brien et al. 2021 (85); Marchante and Marchante 2016 (86); Peter et al. 2021 (92); Peter et al. 2021 (93); Schlaeppy et al. 2017 (98); Shaw 2017 (103); Soroye et al. 2022 (142); Zarybnicka et al. 2017 (111) none none 19 Pollution Ballard et al. 2017 (52); Brooks et al. 2019 (144); Dhillon 2017 (148); Earp and Liconti 2020 (145); English et al. 2018 (64); Gray et al. 2017 (146); Grossberndt et al. 2021 (117); Hodgkinson et al. 2022 (157); Hyder et al. 2015 (147); LandZandstra et al. 2016 (82); Mahajan et al. 2021 (125); Mahajan none none 18 48 OA ● Research field (STEM vs SSH) ● Type of OA (Gold vs Green) ● Social media/web platform ● Clearly signalling OA status in social media posts ● Country-level economic status Policy and governance No evidence found CS ● Lack of official recognition of CS data ● Lack of systems in place to integrate CS data ● Political interest ● Corporate interest/lobbying Health CS ● Responding to a problem/community need ● Directly involving community in the project/program OCS ● Dissemination of open tools No evidence found Empowerment and equity No evidence found No evidence found Trust in and attitudes toward research OS general ● Awareness of OS among the general public No evidence found Figure and table captions Figure 1. PRISMA-P flow diagram. Figure 2. Number of papers by type of OS (% of all papers). Figure 3. Number of papers by type of impact (% of all papers). Figure 4. Number and percentage of studies by type of educational impact. Figure 5. Number and percentage of studies by type of environmental impact. Table 1. Categories extracted from included studies in the data charting process. Table 2. Type of impact by OS type, with number of papers coded per intersection (N) and % of papers within the OS type category. Table 3. Literature with evidence of the impact of CS on education and awareness. Table 4. Literature with evidence of the impact of CS on climate and environment. Table 5. Literature with evidence of the impact of CS on policy and governance. Table 6. Direct and indirect impacts evidenced in the literature, with OS type indicated ( ). Table 7. Enabling and inhibiting factors for societal impact of OS.