Societal impact of research and its assessment: Organisations' perspectives

SOCIETAL IMPACT OF RESEARCH AND ITS ASSESSMENT Organisations’ perspectives 1 Societal impact of research and its assessment: Organisations’ perspectives AUTHORS: Reetta Muhonen, Maria Pietilä, Tommi Kärkkäinen, Janne Pölönen PUBLISHER AND YEAR OF PUBLICATION: Federation of Finnish Learned Societies, 2025 SERIES: Web publications of the Federation of Finnish Learned Societies 25. ISSN: 2242-8011 ISBN: 978-952-7589-24-3 DOI: 10.5281/zenodo.17809546 LICENCE: CC-BY 2 ABSTRACT Objective Federation of Finnish Learned Societies (Tieteellisten seurain valtuuskunta, TSV) launched a project in January 2023 to promote the assessment of the societal impact of research in line with responsible assessment practices. The aim of the project was to provide information on impact assessment, taking into account the specific characteristics of different research disciplines and types of organisations, and to promote a culture of responsible assessment by highlighting the importance of different researcher profiles. The first part of the project focused on researchers' views on impact assessment and its results were published in a report in March 2025. This report focuses on the second part of the project, which concentrates on organisation-level views on impact assessment in Finnish universities, state research institutes and universities of applied sciences. In addition to the above, an overview of the impact assessment practices of four case countries was carried out. Materials The results presented in the report are based on a variety of data: a survey, interviews and document material. We have also used publicly available data from the Finnish National Agency for Education’s statistics service Vipunen. Main findings and conclusions Practices for assessing the societal impact of research vary from one type of organisation to another, depending on the core functions of each organisation, the nature of its research activities and its resources. Organisations' reasons for impact assessment relate to legislative and funder requirements, funding, strategic direction, performance development, selfassessment, and societal expectations. Organisations also seek to highlight the multiple impacts of research on society by monitoring and assessing their research activities. Universities, state research institutes and universities of applied sciences want stronger national cooperation in developing the assessment of the societal impact of research. The desire for collaboration is particularly related to sharing good practices and learning together, rather than building binding assessment frameworks. The systematic development of impact assessment databases is also considered important. The desire for national cooperation is also related to reducing the burden and financial costs of assessment work. In developing responsible assessment, it is important to identify the roles of the different actors in the research community and to clarify the impact objectives. There is also a need for coordinated cooperation between different actors in the research community. 3 In developing assessment practices, it is important to remain sensitive to the diversity of impact. However, it is equally important to recognise that not all impacts can be measured or directly attributed to individual research activities. 4 TABLE OF CONTENTS Abstract .......................................................................................................................................... 2 Preface ........................................................................................................................................... 6 1. INTRODUCTION ........................................................................................................................ 8 2. DATA ........................................................................................................................................ 10 2.1 International Examples ........................................................................................................ 10 2.2 Universities .......................................................................................................................... 10 2.3 State research institutes ...................................................................................................... 11 2.4 Universities of Applied Sciences .......................................................................................... 12 2.5 External Research Funding in Universities, State Research Institutes and Universities of Applied Sciences ....................................................................................................................... 13 3. ASSESSMENT OF THE SOCIETAL IMPACT OF RESEARCH IN FOUR COUNTRIES ........... 18 3.1 General Features of Assessment ........................................................................................ 18 3.2 Assessment Implementation ................................................................................................ 20 3.2.1 Groups Responsible for Assessments........................................................................... 20 3.2.2 Definitions of the Societal Impact of Research .............................................................. 21 3.2.3 Data and Evidence of Impact ........................................................................................ 22 3.2.4 Impact Assessment Criteria .......................................................................................... 24 3.3 Summary of Observations ................................................................................................... 25 4. CHALLENGES OF ASSESSING THE SOCIETAL IMPACT OF RESEARCH ........................... 27 5. ASSESSING THE SOCIETAL IMPACT OF RESEARCH IN UNIVERSITIES ............................ 29 5.1 Background ......................................................................................................................... 29 5.2 Definition of the Societal Impact of Research ...................................................................... 31 5.3 Assessment Criteria ............................................................................................................ 33 5.4 Assessment Indicators ........................................................................................................ 36 5.5 Assessment Support Materials ............................................................................................ 38 5.6 The Role of Parties Outside Academia in Assessments ...................................................... 38 5.7 Assessment Development Needs ........................................................................................ 39 6. ASSESSING THE SOCIETAL IMPACT OF RESEARCH IN STATE RESEARCH INSTITUTES42 6.1 Background ......................................................................................................................... 42 6.2 Societal Impact of Research ................................................................................................ 44 6.3 Monitoring and Assessing the Societal Impact of Research ................................................. 47 6.4 The Challenges of Assessing Societal Impact ..................................................................... 49 6.5 Development Needs in Assessment .................................................................................... 50 5 7. ASSESSING THE SOCIETAL IMPACT OF RDI ACTIVITIES AT UNIVERSITIES OF APPLIED SCIENCES ................................................................................................................................... 54 7.1 Background ......................................................................................................................... 54 7.2 Societal Impact of RDI Activities .......................................................................................... 55 7.3 Reasons for Assessing Societal Impact ............................................................................... 56 7.4 Monitoring and Assessment of the Societal Impact of RDI Activities .................................... 58 7.5 Challenges of RDI Activities and their Assessment .............................................................. 60 7.6 Development Needs in Assessment .................................................................................... 61 8. VIEWS for NATIONAL DEVELOPMENT WORK ....................................................................... 63 9. CONCLUSIONS AND NEXT STEPS ........................................................................................ 65 References ................................................................................................................................... 68 ANNEXES..................................................................................................................................... 73 Annex 1. University survey: Assessing the societal impact of research in research assessment exercises. .................................................................................................................................. 73 Annex 2. Interview outline: Assessing the impact of research in state research institutes. ......... 76 Annex 3. Interview outline: Assessment of RDI activities/research in universities of applied sciences. ................................................................................................................................... 78 6 PREFACE The Federation of Finnish Learned Societies (TSV) is committed to promoting responsible research assessment. The study on the societal impact assessment is part of the long-term development work that TSV has undertaken to support responsible assessment practices. Examples of this work include the Publication Forum for the assessment of scientific publication channels, the national recommendation on researcher assessment (Working Group for Responsible Assessment of Researchers 2020) and the related FIN-CAM (Finnish Career Assessment Matrix) development work. In 2022, TSV joined the international Coalition for Advancing Research Assessment (CoARA). TSV coordinates Finland’s national CoARA chapter. Responsibility of research and assessment practices is one of the priorities of the TSV's strategy for 2024–2030. The strategy aims to reform the culture of assessment in cooperation with the whole scientific and research community. This means developing assessment practices that emphasise ethics, openness, diversity, impact and wellbeing at work. The aim is also to recognise and reward the voluntary work of researchers and their contribution to the research community and society. Strengthening the community-based assessment of publication channels, scientific quality and societal impact is also key. In January 2023, TSV launched a project to develop and strengthen practices for assessing the societal impact of research, taking into account the principles of responsible assessment. The first report of the project (Muhonen et al. 2025) dealt with researchers' views on the assessment of the societal impact of research. This second report focuses on the results of the project, which are related to the theme of societal impact of research from the perspective of Finnish universities, state research institutes and universities of applied sciences. The report also examines the practices of impact assessment in four European case countries. One of the key principles of responsible assessment (cf. CoARA) is that the assessment criteria and practices are developed in cooperation with research organisations and researchers. The results of the study support this objective by increasing understanding of the opportunities for and challenges of assessing the societal impact of research in different organisations that carry out and assess research. The organisational approach is important because it helps to understand the relationship between the specific characteristics of organisations and the societal impact of research and its assessment. This report focuses on the specific characteristics of organisations, in particular their statutory duties, the research funding profile and how they support the activities of their research staff. The issues related to assessment are also linked to the 7 differences in disciplines, which have been discussed in the report of the first part of the project (Muhonen et al. 2025). The research report was written by Project Director Reetta Muhonen, D.Soc.Sc., docent; Project Researcher Maria Pietilä, D.Soc.Sc; Project Researcher Tommi Kärkkäinen, MA; and Secretary General Janne Pölönen, Lic.Phil. We would like to thank all those who responded to the university survey and those who participated in interviews with universities of applied sciences and state research institutes. We also thank Elina Pylvänäinen for coordinating and implementing the university survey. Finally, we would like to thank the experts who helped us to map the assessment frameworks of our case countries: Gemma Derrick, Jon Holm, Emanuel Kulczycki and Ludo Waltman. 8 1. INTRODUCTION Universities, universities of applied sciences and state research institutes are mainly publicly funded organisations, each with their own statutory duties and their own relationship with research and society. Traditionally, the main objective of university research has been the advancement of science. At universities of applied sciences, research encompasses research, development and innovation (RDI) activities aimed at supporting practical problem solving and regional and industrial development. In state research institutes, research is sectorspecifically focused on each institute's own administrative branch and other expert and official duties. Research institutes serve a wide range of administrative branches, the rest of the public sector, businesses and the third sector. (Hakala et al. 2003; Universities Act 558/2009; Universities of Applied Sciences Act 932/2014; Late & Puuska 2014; Ministry of Education and Culture 2025.) The Universities Act (558/2019) and the Universities of Applied Sciences Act (932/2014) require higher education institutions to assess their activities and their impact. The legislation also requires universities and universities of applied sciences to participate in regular external audits of their activities and quality assurance systems and to publish the results of these audits. External audits of higher education institutions in Finland are carried out by the Finnish Education Evaluation Centre (FINEEC), which carries out assessments of higher education institutions, thematic and system assessments, accreditation of technology degree programmes and assessments of fields of education (FINEEC 2025). FINEEC has also assessed societal impact by focusing on the impact quality systems. Finnish universities have assessed their research activities separately. The research assessment exercises have typically been carried out every 5–6 years. They have been carried out by an external group of experts and have focused on departmental, subject or unit level. (Wang et al. 2014.) The university of applied science sector does not have similar wellestablished practices for assessing RDI activities, but each university of applied science has conducted and targeted assessments in their own way. State research institutes are part of the central government. Each research institute is governed by its own law. Unlike universities and universities of applied sciences, state research institutes have no obligation to assess their research and its impact (see, for example, Act on the Finnish Environment Institute 1069/2009; Act on the Geological Survey of Finland 167/2011; the Food Authority Act 371/2018). Instead, the societal impact of activities, including research, is monitored and assessed as part of the performance management process between the institute and the respective ministry. In performance-based management, the performance of agencies and institutes is regularly assessed against 15 Figure 2. Share external sources (%) accounted for of all external research funding in universities in 2012– 2022. Source: Vipunen – Education Statistics Finland. In 2022, the main sources of extra-budgetary research funding for state research institutes were companies and ministries (22% of external funding each), the EU (21%; the various EU funding instruments are not classified for research institutes in Vipunen), the Research Council of Finland (16%) and Business Finland (10%) (see Figure 3). A special characteristic of research institutes is that the funding is concentrated in two large institutes: VTT Technical Research Centre of Finland (whose funding accounted for 44% of the research funding of research institutes in 2022) and Natural Resources Institute Finland LUKE (whose funding accounted for 22% of the research funding of research institutes in 2022). It should be noted that there are differences not only between types of organisations, but also within the same type of organisation: universities, state research institutes and UASs differ in size, research volume and main sources of funding. For example, the different amounts of overall research funding for research institutes are also influenced by the nature of the research carried out, such as whether expensive infrastructure is needed to carry out the research. 0 % 10 % 20 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 % 2012 2014 2016 2018 2020 2022 Research Council of Finland Business Finland Ministries Municipalities Domestic funds and foundations Businesses EU Framework Programme funding ERDF, ESF and other EU funding Other sources 16 Figure 3. Share external sources (%) accounted for of external research funding in state research institutes in 2012–2022. Source: Vipunen – Education Statistics Finland. Two sources of funding are particularly prominent in UASs: ministries and ERDF, ESF and other EU funding (Figure 4). In 2022, funding from ministries accounted for 37% of external research funding and ERDF, ESF and other EU funding for 36% of external research funding. ERDF refers to the European Regional Development Fund and ESF to the European Social Fund. The most common other sources of funding were companies (6% of external research funding) and EU Framework Programme funding (6%). 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 2012 2014 2016 2018 2020 2022 Research Council of Finland Business Finland Ministries Municipalities Domestic funds and foundations Businesses EU Other sources 17 Figure 4. Share external sources (%) accounted for of external research funding in universities of applied sciences in 2012–2022. Source: Education Statistics Finland, Vipunen. The structure of research funding for universities, state research institutes and UASs reflects their different roles in the research and innovation system. University research funding is primarily provided by the Research Council of Finland, which funds academic, peer reviewed scientific research. The funding structure of state research institutes is diverse, including both academic and applied research funding bodies. The funding of UASs is primarily provided by the EU's regional and structural policy funding instruments, which aim, for example, to reduce disparities in development and wellbeing between regions. 0 % 10 % 20 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 % 2012 2014 2016 2018 2020 2022 Research Council of Finland Business Finland Ministries Municipalities Domestic funds and foundations Businesses EU Framework Programme funding ERDF, ESF and other EU funding Other sources 18 3. ASSESSMENT OF THE SOCIETAL IMPACT OF RESEARCH IN FOUR COUNTRIES In this chapter, we present and compare the practices for assessing the societal impact of research in four countries: the UK, Poland, Norway and the Netherlands. The countries were selected to provide a broad picture of different assessment models and the role of societal impact as part of the assessments. Since the 2010s, focusing on the societal impact of research has become more common in the countries’ research assessment practices. In the Netherlands, impact was already part of the assessment framework in 2003– 2009. 3.1 General Features of Assessment In the UK, research is assessed nationally with the Research Excellence Framework (REF) (UKRI 2023b). REF assessments have been published in 2014 and 2021. The next assessment is scheduled for publication in 2029. The assessment is carried out by four UK higher education funding organisations (Scottish Funding Council; Higher Education Funding Council for Wales; Department for the Economy, Northern Ireland; Research England), with Research England coordinating the assessment (UKRI 2023b). REF assessments are discipline-specific and targeted at higher education institutions (REF 2012; REF 2019b). The REF Steering Group is responsible for defining the assessment domains, while the main and sub-panels define the criteria for the elements (REF 2019b; UKRI 2025). The elements and criteria are the same for all those being assessed. The elements of assessment are the outputs and impact of research as well as the research environment, each of which is given its own weighting in the assessment. In 2021, the weighting of impact was 25%, five percentage points higher than in 2014 (REF 2012; REF 2019b). The weighting of research outputs was 60%, five percentage points lower than before. The impact percentage is due to remain the same in the 2029 assessment (REF 2023, 11, 13). The results of the assessment have an impact on the funding of higher education institutions, so it is a summative assessment. The REF, together with the Higher Education Innovation Fund framework, forms the UK's quality-based funding for higher education institution research (UKRI 2023a). The results determine the basic funding allocated to higher education institutions. In addition to this, the Research Councils under UK Research and Innovation (UKRI) provide project-specific funding. In Poland, the national assessment of research is carried out every four years by a research assessment commission appointed by the Minister of Science and Higher Education (the Law on Higher Education and Science, 2018, 139–141). The Ministry of Science and Higher 19 Education, supported by the Commission for the Evaluation of Science, has designed the assessment process to be common to all those being assessed (Commission for the Evaluation of Science 2021; Kulczycki & Korytkowski 2021). Assessment focuses on higher education institutions, institutes of the Polish Academy of Sciences and other research institutes by discipline (the Law on Higher Education and Science 2018, 139–141). The three main areas of assessment are scientific or artistic quality, the economic results of scientific research or development, and the impact of the research on society and the economy. The domains have been given their own weighting in the assessments. Impact corresponds to around 20% of the assessment result (Minister of Education and Science 2022). The results of assessment determine the quality category (A+, A, B+, B, C) of the research of the assessed subject, which affects the public funding the higher education institution receives (the Law on Higher Education and Science 2018, 205-206). In addition to this, sufficiently good results (A+, A, B+) are a condition for maintaining the title of a university and the right to train doctoral students (the Law on Higher Education and Science, 2018, 6–7, 107–109). The research quality category is part of the research activities of a higher education institution, including the number of doctoral researchers and research staff and research costs (the Law on Higher Education and Science 2018, 205–206). This category is part of the higher education funding algorithm. Another key component of the algorithm is teaching activity, which includes the number of students and the costs associated with students. In Norway, research is assessed by discipline, with one discipline being assessed at a time (the Research Council of Norway 2024a). The Research Council of Norway has been responsible for the assessments since the 1990s. Assessments focus on the research carried out in universities, public sector research institutes and the health sector. The Research Council of Norway defines a general assessment protocol for each discipline, the details of which are negotiated with the research organisations (the Research Council of Norway 2024a; 2024c). Therefore, the organisations involved in the assessment have a say in how it is carried out. The domains of assessment have varied over the years. In the 2017 Evaluation of the Humanities in Norway, the domains included organisation, management and strategy, access to and use of resources, research production and quality, recruitment and training, networking of researchers, impact on teaching and societal impact (the Research Council of Norway 2017, 19). In the 2024 Evaluation of Life Sciences in Norway, the areas of focus were research production, quality and integrity, relevance to institutional and sectoral purposes, and relevance to society (the Research Council of Norway 2024b, 42–45). The assessments provide information on the state of research and higher education in Norway. This information helps the parties involved in the assessment to understand their own activities, supports the advisory work of the Research Council of Norway and provides information to those responsible for the development of research and higher education (the Research 20 Council of Norway 2024a). The results of the assessment do not affect the funding of higher education institutions. In the Netherlands, research is assessed every six years according to the Strategy Evaluation Protocol (SEP) framework (VSNU, KNAW & NWO 2020). The SEP framework and its implementation are the responsibility of the Association of Universities of the Netherlands (VSNU), the Royal Netherlands Academy of Arts and Sciences (KNAW) and the Dutch Research Council (NWO). SEP assessments are national and focus on universities, research institutes of the Dutch Research Council and institutes of the Royal Netherlands Academy of Arts and Sciences. The first SEP was carried out in 2003–2009 and the current review for 2021–2027 is the fourth in the 21st century. Publicly funded research organisations are obliged to assess their research activities every six years. Although there is no legal obligation for universities to use the SEP framework, the use of the model is based on an agreement between VSNU, NWO and KNAW and is well established as a national practice. In practice, all universities follow the SEP protocol as a research assessment framework. (VSNU, KNAW & NWO 2003; 2009; 2014; 2020.) The three main assessment criteria of the 2021–2027 evaluation framework are the quality, societal relevance and viability of research. Assessments under the SEP framework focus on how the practices and outputs of the assessed unit contribute to the achievement of the unit's goals and strategy. The SEP framework is used for formative assessment. The aim of the assessments is to help the assessed units to understand and develop their own activities and to make visible the quality and societal relevance of their research (VSNU, KNAW & NWO 2020, 7). 3.2 Assessment Implementation 3.2.1 Groups Responsible for Assessments In all the countries reviewed, assessments are carried out by groups of experts who must have a good understanding of the research being assessed. The Dutch SEP framework specifies that, in addition to discipline-specific knowledge, the experts should have the competence to assess the applicability and relevance of research to society (VSNU, KNAW & NWO 2020, 43). The Norwegian guidelines for assessments in law, life sciences, health and medicine, science and mathematics, and engineering are similar (the Research Council of Norway 2021, 56–57; the Research Council of Norway 2022a, 10; 2022b, 10; 2022c, 10; 2024b, 46). In Poland, on the other hand, the law states that members of the assessment committee must have experience of active involvement in the application of research and a significant track record in this area (the Law on Higher Education and Science 2018). With regard to the REF 21 framework in the UK, additional assessors have been invited, where necessary, to join assessment panels with a particular focus on impact (REF 2019b, 4–5). Similar to the policy in Poland, they must have professional experience in the use, application and benefits of research. Assessors appointed to the sub-panels will undertake either one of the following roles: a. To assess the impact element of submissions and develop the impact subprofiles, alongside sub-panel members. These will be people with professional experience of making use of, applying or benefiting from academic research. (REF 2019b, 5). 3.2.2 Definitions of the Societal Impact of Research The assessment systems have largely similar definitions of the societal impact of research, i.e. impact is understood in terms of effects and benefits beyond the academic community. In the UK REF framework, societal impact is defined as: For the purposes of the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. (REF 2019a, 68) The 2021 assessment of the discipline of law published by the Research Council of Norway used the UK's REF definition of societal impact almost in its original form (the Research Council of Norway 2021, 73). The assessment of social science, published earlier in 2018, also cites and refers to the REF. The definition of, and model for, societal impact in the Research Council’s evaluations is derived from the 2014 Research Excellence Framework (REF) in the United Kingdom. In the REF, societal impact is defined as ‘any effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. (the Research Council of Norway 2018, 11–12) The difference between the Norwegian social science review and the UK REF is that, unlike in Norway, the REF considered the impact of research on education as part of the societal impact of research (REF 2019b, 54; the Research Council of Norway 2018). In the 2024 Norwegian life sciences assessment and the 2024 ongoing assessments of medicine and health sciences, natural sciences and mathematics, the concept of societal relevance has been used. The assessment of societal relevance is described as follows: The committee assesses the quality, scale and relevance of contributions targeting specific economic, social or cultural target groups, of advisory reports on policy, of contributions to public debates, and so on. [...] The committee is also invited to assess the societal impact of research based on case studies submitted by the administrative units and/or other relevant data presented to the committee. (the Research Council of Norway 2022a, 9; 2022b, 9; 2022c, 9) 22 In Poland, the guidelines for research assessment define the societal impact of research as the impact that research has on different sectors of society (Commission for the Evaluation of Science 2021). In the Dutch Strategy Evaluation Protocol (SEP), societal impact is part of the societal relevance of research, and a distinction is made between impact, societal interaction and uptake of research. The societal relevance of the unit’s research in terms of impact, public engagement and uptake of the unit’s research is assessed in economic, social, cultural, educational or any other terms that may be relevant. (VSNU, KNAW & NWO 2020, 8) 3.2.3 Data and Evidence of Impact In all countries, the assessments use descriptions of the societal impact of research provided by the assessed units. In Poland and the UK, impact is reported through impact case studies. In Norway and the Netherlands, the impact case studies are part of the self-assessment reports. In Norway, in addition to self-assessment reports and impact case studies, the assessors use national register data for their assessments in life sciences, medicine and health sciences, mathematics and natural sciences (e.g. the Research Council of Norway 2022a, 16; 2022b, 16; 2022c, 16). In the Netherlands, on the other hand, the assessed units are encouraged to include statistical information in their self-assessment reports, such as the number of popular publications, the number of citations they have received and the financial support the studies have received from social groups (VSNU, KNAW & NWO 2020, 19). In all assessment models, the description of impact concerns the activities that contribute to an increase in impact and the impacts achieved. In addition to this, the assessed units are encouraged to link their descriptions of impact to their research or research outputs so that the assessors are aware of the research activity to which the impact relates. The UK and Norwegian assessments also include instructions to describe the stakeholders who benefit from the research and how they benefit from the research or activities of the unit (REF 2019a, 97; the Research Council of Norway 2018; 2021). For example, the REF instructed that the case studies be realised as follows: A clear explanation of the process or means through which the research led to, underpinned or made a contribution to the impact [...] Where the submitted unit’s research was part of a wider body of research that contributed to the impact [...] Details of the beneficiaries […] Details of the nature of the impact – how they have benefitted, been affected or impacted on. (REF 2019a, 97). Evidence was also requested for the claims made in the qualitative descriptions in order to verify impact. To this end, the UK REF assessment includes a request for information from external sources, such as stakeholder testimonies or links to documents, to be included in the impact case studies to verify the reported impact. 23 All the material required to make a judgement should be included in the case study template (REF3) – no further reading should be required. URLs should only be included for the purpose of verifying or corroborating claims made in the submission. […] Extracts from corroborating statements may be included within the case studies, where appropriate. (REF 2019b, 58). In Poland, the guidance on evidence of impact is of a general nature: A detailed characterisation of no more than 5 proofs of impact of the scientific activity (maximum 500 characters in spaces for each proof for each language version) and, in the case of the characterisation referred to in the fourth indent of point e, no more than 5 proofs of impact resulting from the scientific activity of another entity created by the entity for the purpose of commercialisation (the characterisation may include the website address where in the date of placing the impact description in the POL-on System, a given proof of impact is available). (Commission for the Evaluation of Science 2021) In the Evaluation of Social Sciences in Norway, evidence was requested in a very similar way to the REF (the Research Council of Norway 2018), which also aligned with the 2021 law assessment: This section should list sources that could corroborate key claims made about the impact of the unit’s research (reports, reviews, web links or other documented sources of information in the public domain, users/beneficiaries who could be contacted to corroborate claims, etc.) (the Research Council of Norway 2021) In Norway's 2024 Evaluation of Life Sciences, the requirement for evidence is not mentioned in the assessment guidelines (the Research Council of Norway 2024b). However, the selfassessment guidelines for the units request that they list five to ten of the most significant publications or other outputs for stakeholders and to describe their impact (the Research Council of Norway, n.d.). In the Netherlands, the inclusion of "factual evidence", such as statistical data to support arguments about impact, was encouraged. This narrative argument is, wherever possible, supported by factual evidence (where appropriate, the unit can use quantitative indicators). The choice of indicators accordingly depends on the exact argument for which they should provide evidence. The research unit selects its indicators based on the argument which it wants to develop. Other sources of robust data may include benchmarking against peer research units as well as case studies highlighting its most distinctive and societally relevant accomplishment(s). (VSNU, KNAW & NWO 2020, 19) It was requested that statistical data be combined in one or more of three categories: research outputs to societal stakeholders (e.g. popular publications, patents, videos, exhibitions and software), use of research outputs by stakeholders (e.g. collaborative projects, 24 commissioned studies, citations) and recognition by stakeholders (e.g. financial support received, involvement in organisations, awards) (VSNU, KNAW & NWO 2020, 32). 3.2.4 Impact Assessment Criteria The criteria for assessing the systems repeatedly mentioned the significance and reach of impact. In the REF assessment, they are defined as: Significance will be understood as the degree to which the impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or wellbeing of the beneficiaries. (REF 2019b, 52) Reach will be understood as the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact. Reach will be assessed in terms of the extent to which the potential constituencies, number or groups of beneficiaries have been reached; it will not be assessed in purely geographic terms, nor in terms of absolute numbers of beneficiaries. The criteria will be applied wherever the impact occurred, regardless of geography or location, and whether in the UK or abroad. (REF 2019b, 52) The same definition was used in several Norwegian assessments. For example, in the humanities assessment, the criteria definitions are almost identical to those used in the REF assessment (the Research Council of Norway 2017). The Norwegian social science assessment report presents the same criteria, the assessment of which concerns a) assessing the importance of the proposed change (major/minor), b) assessing the scope of the proposed change (global, national, regional, local), c) giving priority to more recent examples of impact (impacts and research should be from the last 10–15 years) (the Research Council of Norway 2018). The Polish assessments also use significance and scope as criteria, which are defined in much the same way as in the REF assessments. In Poland, however, the guidance on significance is to assess whether the research has provided a solution to a long-standing societal problem (Commission for the Evaluation of Science 2021). Polish assessments also differ from others in that they instruct the assessed units to describe whether and how multidisciplinary research contributed to the impact. There are also differences in the criteria used in the Norwegian assessments compared to the REF. In several of the Norwegian assessments (humanities, social sciences, law, natural sciences, mathematics and engineering, life sciences and health and medicine), the impact of research has been assessed in the context of the state's long-term strategy for research and higher education (see the Research Council of Norway 2017; 2018; 2021; 2022a; 2022b; 2022c; 2024b). The strategy includes six thematic domains based on which the impact of the units was examined: oceans; climate, environment and clean energy; public sector reform and 31 5.2 Definition of the Societal Impact of Research Table 2 summarises the universities' responses to the survey’s question: How is the societal impact of research is defined in your organisation (e.g. in the most recent or forthcoming overall research assessment)? In some universities, the societal impact of research was defined in the assessment reports, while in others, it was reported to be more implicit in the policies guiding the assessment. In general, universities defined the societal impact of research as impact, change, benefit or added value reaching beyond the academic community. The definitions highlighted interaction with societal actors and the changes that research brings to society. Universities use the definition of the UK Research Excellence Framework, a European pioneer in impact assessment, in their definitions of impact: [...] impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. [...] Impact includes the reduction or prevention of harm, risk, cost or other negative effects. (REF 2019b, 68). The definitions of impact were also influenced by the definition of societal impact of research used by the Research Council of Finland: Societal impact refers to the ways in which research contributes to developments in society and to dealing with social issues. Impact is a complex phenomenon that arises in the interaction between research data and other factors, often over a long period of time. (Societal impact – Research Council of Finland, i.a.). It was also typical that the definition of one university was influenced by both the REF and the Research Council of Finland. There were differences between universities in the emphasis placed on the process of impact and the emphasis placed on the outputs at the end of the process, i.e. wider impact such as societal change. Table 2. How is the societal impact of research defined in your organisation (e.g. in the most recent or forthcoming research assessment exercise)? Responses from universities. In the most recent assessment of research, art and impact (RAI2018), societal impact was defined in the guidelines for external assessors as: “Panels are asked to assess the societal impact of the Units in terms of their influence on change, benefit or value added to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Impact also includes reduction or prevention of harm, risk, cost or other negative effects. Impact of the research and artistic activity may appear as or lead to: • Societal quality (e.g. interaction with and communicating results to external stakeholders, engagement in entrepreneurial activities); • Societal impact (e.g. influence on stakeholders or societal procedures); • Valorisation (e.g. activities aimed at making results available and suitable for application in products, processes and services, utilization of innovation potential); • Dissemination (e.g. activities aimed at making results widely known or 32 providing stakeholders a window to current research and novel results). Societal impact is demonstrated by case studies provided by the Units of Assessment. In the case studies, both quantitative and qualitative impact indicators shall be considered. Such indicators include but are not limited to expert tasks, popularised works, media visibility, activities and external funding resulting from collaboration with non-academic institutions (corporations, Tekes, EU), invention disclosures, patents, licenses, start-up companies, cooperation with the public, private and third sector outside the academia and involvement in training programs for leaders and executives. Panels are also asked to assess how the Unit is developing its strategy to support and enable impact of its activities. “ Our definition in English in the latest assessment: Societal impact referred to the interaction between the Unit and the wider societal audiences. Societal Impact of Research Activities: Societal relevance, what kind of impact the RC’s research has had on, e.g., the economy, health & wellbeing, public policy & services, technology, education and the environment. Indicate the impact on local, national and global level. How does the RC contribute to the Sustainable Development Goals of the United Nations (https://sdgs.un.org/goals)? As part of the assessment of research, the assessed units carried out an assessment of the impact of their research. In their self-assessments, units were instructed to consider not only their research findings but also the impact on education, technologies, society in general, culture, the environment and the economy. The ULAP RAE 2022–23 self-assessment report defined societal impact as: Societal impact is understood here broadly, and considered to emerge as a long-term outcome from research. Impact may arise, e.g., in form of leadership, influence, benefits or added value, and research-teaching linkages, but it may appear also as prevention of harm or negative effects. Research impact refers to the contribution of research to various issues and developments in society. Impact is the result of a combination of research information and other factors, often over a longer period of time. It is, by definition, a complex phenomenon. Our northern handprint is global. The University of Oulu is a northern, international science university. We are producing new scientific knowledge and solutions to build a more sustainable, intelligent and humane world. Our northern location gives us a special opportunity and responsibility to tackle issues in the Arctic. Our cutting-edge research is based on strong multidisciplinary expertise. Our high-level, cross-disciplinary research discoveries relate to creating and understanding the digital future and building a more sustainable world. provide positive evidence of such impacts and describe research activities linked to the impacts for review. Impact cases are commonly applied in the UK as part of the Research Excellent Framework (REF 2021). Accordingly, societal impact is defined as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia,” or as the “reduction or prevention of harm, risk, cost or other negative effects.” Such impacts may occur locally, regionally, nationally or internationally. An impact includes (but is not limited to) an effect on, change, or benefit to: • the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding; • of an audience, beneficiary, community, constituency, organisation, or individuals. In considering the impact of research towards the achievement of societal benefits it can refer to e.g. participation of underrepresented minorities, increased well-being of individuals, public engagement with science and technology, increased partnerships with industry or increased economic competitiveness. A high-scoring societal impact case clearly identifies beneficiaries of research and engages non-academic stakeholders. It clarifies links between research and claimed impact. It also provides evidence of the societal achievements of research, for example, in the form of testimonials from key stakeholders and beneficiaries. Societal impact was examined for each academic subject through case studies of their choice. The definition was deliberately left vague to allow different disciplines to highlight the areas and means of societal impact that are important to them. The following definition was used for the research assessment exercise: "Societal impact is defined here as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.” (Research Excellence Framework 2021). Under the umbrella concept of 33 societal impact, please consider a broader perspective to the chain of actions that aim for and finally, together with contextual factors, constitute the societal impact of the research presented here. You can elaborate both the direct and indirect, intended and unintended societal impact. If the societal impact of your research (i.e.longterm change) is not yet demonstrable, you can also focus on the societal outcomes (changes in behaviour, relationships, activities and actions) of research." The last (and first) research assessment exercise of the new Tampere University was carried out in 2022 (TAU RAE 2022). The Terms of Reference document defined societal impact as follows: "For this criterion the panel assesses the reach and significance of the research conducted in the unit in terms of society at large as well as its relevance and if it contributes to producing new knowledge or solutions or increasing understanding on different phenomena. The panel should also consider how results are disseminated and how well the unit is engaged with stakeholder networks. Please note that it is possible to reach the highest rating with local, national and/or international relevance." The Quality Manual defines the term as follows: "Societal impact refers to the impacts and influences that the University’s activities have on society. Societal interaction means collaborating with external stakeholders. In practice, societal interaction is most often an inseparable element of research and education." “Research makes an impact on society, among other things, by generating new research information and providing access to openly available research data and practical application environments. Collaboration with citizens, companies and local actors strengthen the University’s impact both locally and globally. Talented graduates and alumni are a proof to the societal impact of education." The University of Turku has not formally defined the societal impact of research at university level. In most contexts where the societal impact of research has been discussed or assessed, the official definition of the Research Council of Finland has been used. We have not explicitly defined what is referred to by research impact, but it can be divided into scientific impact, which takes place in the form of dissemination of research results, and what can be described as the ‘societal impact of research’, i.e. something that benefits society in a broad sense (the economy, culture, politics, health, the environment, quality of life, etc.) In our latest internal call for applications to appoint centres of excellence (ÅAU CoE 2023), the following criteria are emphasised in the assessment of applications: “... The units selected as CoEs are scientifically first-rate research communities that have capacity for renewal and high societal impact. The CoE programmes contribute to the renewal of science by supplying new research topics, new methods and approaches, and new research teams. Thanks to the long-term funding provided in collaboration with CoE host organisations, the funding instrument effectively works as an incentive for risk-taking and new initiatives in research. The primary assessment criteria are the scientific quality, promotion of scientific renewal, and scientific impact of the research plan” 5.3 Assessment Criteria As with the definitions of the societal impact of research, the criteria used by universities to assess impact reflect the UK REF's approach to assessment. The REF assessment criteria outline: ”The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society, culture, public policy or services, health, the environment or quality of life that were underpinned by excellent research conducted in the submitted unit” (REF 2019a, 7). A typical criterion was the relevance of the research or other outputs to a sector or stakeholder in society. The criteria also included the societal significance and reach of the research. With regard to the extent of impact, some universities instructed their assessment units to describe the extent to which the societal impact of research is expressed at regional, 34 national and international level. In addition to these general assessment dimensions, the importance of identifying and naming stakeholders as part of the impact assessment was emphasised. Units were instructed to describe who is affected by the research or activity and how they interact with stakeholders. The criteria also reflected the time span limitations of the assessment. Some universities specified that expected or potential future impact is not taken into account in the assessments. In other universities, assessments focus specifically on the capacity and potential of the unit to deliver societal impact in the future. Some guidelines stressed that the assessed unit should describe both actual and potential impact. Some universities also reported using criteria related to actions to support and promote impact. These universities drew attention to issues such as how the unit's strategy supports the promotion of impact, while others requested a broader assessment of how the unit supports impact work. Table 3. What criteria have been used to guide assessors in assessing the societal impact of research in the most recent overall research assessment? Responses from universities. “Panels are asked to assess the societal impact of the Units in terms of their influence on change, benefit or value added to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Impact also includes reduction or prevention of harm, risk, cost or other negative effects. Impact of the research and artistic activity may appear as or lead to: • Societal quality (e.g. interaction with and communicating results to external stakeholders, engagement in entrepreneurial activities); • Societal impact (e.g. influence on stakeholders or societal procedures); • Valorisation (e.g. activities aimed at making results available and suitable for application in products, processes and services, utilization of innovation potential); • Dissemination (e.g. activities aimed at making results widely known or providing stakeholders a window to current research and novel results). Societal impact is demonstrated by case studies provided by the Units of Assessment. In the case studies, both quantitative and qualitative impact indicators shall be considered. Such indicators include but are not limited to expert tasks, popularised works, media visibility, activities and external funding resulting from collaboration with non-academic institutions (corporations, Tekes, EU), invention disclosures, patents, licenses, start-up companies, cooperation with the public, private and third sector outside the academia and involvement in training programs for leaders and executives. Panels are also asked to assess how the Unit is developing its strategy to support and enable impact of its activities. “ Criteria in English: The panel will assess how the Unit positions its research vis-à-vis broader issues, extending also beyond academia: whether potential stakeholders and audiences have been identified, and which research questions or results are immediately relevant or could be relevant later. Other criteria, with different meanings in different disciplines, are the Unit’s activities on valorisation (activities aimed at making results available and suitable for application) and dissemination and communication (activities aimed at making results widely known or providing stakeholders and different actors in civil society a window to current research and novel results). The Unit’s approach to supporting and enabling the impact of its activities will also be considered. Guiding questions: Evaluate the impact of research: - Has the RC’s research produced significant new knowledge/innovations/solutions/patents relevant to the economy, health & wellbeing, public policy & services, technology, education or the environment? - How viable is the RC’s collaboration with various stakeholders from the private and public sectors? - Significance of impact on local, national and/or global level? Indicate - Current strengths, areas of development and recommendations. 35 In connection with the research assessment exercise, the assessed units (mainly faculties) of the University of Jyväskylä conducted a self-assessment, one domain of which (13 domains in total) was societal impact. Assessors were instructed to assess this section in relation to the resources of the units and other themes of assessment. The panel was given a reasonably free hand to assess impact with the following guidance: Assessment of societal impact of research (overall rating 1–5) You may consider the following topics: - Character and quality of research collaboration - Commercialisation and consultancy - Level of educational collaboration - Goals for public policy and public engagement. Guiding questions for panelists: Does the research unit (RU) consider the societal relevance of its research as an important aspect of its activities? · Is the research societally relevant and what kind of impacts does it have locally, nationally and/or globally? · What kind of impacts have the collaborations of the researchers at the RU with non-academic partners, e.g. policymakers, the civil society, business life, the environment and the health care system on the national and/or global scale? Rating: Rating criteria of the societal impact of research 6 Outstanding: The RU has outstanding societal interactions and impact and its research is of very high relevance in terms of new knowledge and solutions. The RU has identified relevant audiences and stakeholders and set up systematic activities to reach them. The outcomes provide very convincing evidence of societal impact. 5 Excellent: The RU has excellent societal interactions and impact and its research is of high relevance. The RU has wide and dynamic interaction with the society. The RU has identified audiences and stakeholders as well as activities to reach them. The outcomes provide convincing evidence of societal impact. 4 Very good: The RU has very good interactions with the society and its research is relevant for producing new knowledge and solutions for the society. There are activities to reach the society and proof of successful outcomes. 3 Good: The RU has established some interactions with society. For most parts, the research at the RU is relevant to society. The unit has some understanding of the role and positioning of their research in the society. 2 Fair: The RU is developing its interaction with society. The research at the RU has limited societal relevance but has the potential for making a wider impact on society. 1 Poor: The RU has little interaction with the society. For most parts, the RU itself does not perceive the societal relevance of its research. “Panels are asked to assess the societal impact of the Units in terms of their influence on change, benefit or value added to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Impact also includes reduction or prevention of harm, risk, cost or other negative effects. Impact of the research and artistic activity may appear as or lead to: • Societal quality (e.g. interaction with and communicating results to external stakeholders, engagement in entrepreneurial activities); • Societal impact (e.g. influence on stakeholders or societal procedures); • Valorisation (e.g. activities aimed at making results available and suitable for application in products, processes and services, utilization of innovation potential); • Dissemination (e.g. activities aimed at making results widely known or providing stakeholders a window to current research and novel results). Societal impact is demonstrated by case studies provided by the Units of Assessment. In the case studies, both quantitative and qualitative impact indicators shall be considered. Such indicators include but are not limited to expert tasks, popularised works, media visibility, activities and external funding resulting from collaboration with non-academic institutions (corporations, Tekes, EU), invention disclosures, patents, licenses, start-up companies, cooperation with the public, private and third sector outside the academia and involvement in training programs for leaders and executives. Panels are also asked to assess how the Unit is developing its strategy to support and enable impact of its activities. “ The assessors were asked to look at examples of the societal impact of the subjects and assess them qualitatively. Assessment criteria for societal impact Societal impact of research is assessed on a scale from 1–3: 3 Outstanding: At the UoA, there is clear understanding of the role and positioning of its research in society. The UoA has identified audiences and stakeholders as well as activities to reach them. The outcomes provide convincing evidence. 2 Adequate: There are activities and outcomes of societal impact, but not yet in a consistent manner. The UoA has started to develop understanding of the role and positioning of its research in society or identified audiences and stakeholders. 1 Weak: Audiences and stakeholders have not been identified and there is only little activity or outcomes. The UoA has not defined their role or positioning in society. RAE 2022 Terms of Reference: 2.3. Societal impact "For this criterion the panel assesses the reach and significance of the research conducted in the unit in terms of society at large as well as its relevance and if it 36 contributes to producing new knowledge or solutions or increasing understanding on different phenomena. The panel should also consider how results are disseminated and how well the unit is engaged with stakeholder networks. Please note that it is possible to reach the highest rating with local, national and/or international relevance." After this, all ratings 1–5 were defined, for example 5 Outstanding: "The research of the Unit is outstanding in terms of reach and significance. The research is highly relevant and provides new knowledge and solutions that significantly benefit the society at large, and profoundly increases understanding on different phenomena." In the most recent research assessment exercise, the guidelines sent to peer reviewers asked them to assess societal impact by asking the following question: 1) How do you estimate the strengths and weaknesses of societal impact of the department or unit under evaluation? In addition to this, the reviewers were encouraged to give comments and recommendations on societal impact. Grades for the impact of research were given based on the criteria described in Table 3. Typically, the grade was based on how relevant, significant, wide-ranging and influential the research was to stakeholders. The extent to which impact was actively and/or systematically promoted was also considered important. However, the specification of the terms used in the scoring, such as relevance and significance, was left partly open. The evidence requirement embedded in the UK REF (see Chapter 3) was also reflected in the way universities instructed their staff on how to describe impact. The criteria and scoring guidelines of several universities explicitly mentioned that the assessed units had to provide evidence or clear examples of impact, for example by linking the research activities aimed at promoting societal impact with the impact achieved. As with relevance, no specific criteria were defined for evidence. Some universities relied on the opinions of stakeholders to verify impact. 5.4 Assessment Indicators The survey included a question on "what quantitative or qualitative indicators does your organisation use to assess the societal impact of research as part of its overall research assessments". Some universities specified the precise indicators of impact they use, such as expertise, patents and external funding achieved. Other universities described indicators at a more general level, for example as activities related to the dissemination of research results and the promotion of impact, or as impact-related objectives. Based on the universities' responses, we classified the indicators used in the research assessment exercises into four categories (see Table 4): 1) activities related to impact, 2) achievements related to impact, 3) research outputs, and 4) strategic planning and organisational support. 37 Table 4. Impact assessment indicators used by universities. Indicator category Indicators Activities related to impact • expert positions • workshops • involving citizens and stakeholders in research • participation in public, private or third sector activities • interaction with stakeholders/participation in stakeholder networks • participation in entrepreneurial activities • participation in innovation networks and platforms • studies commissioned by stakeholders • joint projects with research institutes (EUR/year) • popularisation of studies • ways to promote the exploitation and dissemination of research results • ways to promote innovation • self-assessment of the extent and volume of social networking/cooperation activities (none, rarely, occasionally, often, constantly). Achievements related to impact • media visibility (e.g. website clicks and news coverage) • societal visibility • funding and activities as a result of cooperation with non-academic parties • complementary RDI funding • doubling business income • partnership, framework and cooperation agreements concluded (number/year) • Business Finland funding (co-creation and co-innovation project types, EUR/year) • number of projects leading from research to business • placement of doctors in working life • participation in management training. Research outputs • popular outputs • open science (publications, data, methods, software/code) • policy recommendations • training materials • joint publications with non-academic partners • joint publications with research institutes (number/year) • new operating models • invention and work registrations • patents and licences • technology transfer agreements • IP rights transfers (number/year and revenue EUR/year) • start-ups and spinoffs. Strategic planning and organisational support • support for impact efforts • developing strategies related to impact • shortand long-term objectives for impact • linking impact objectives to the university's strategy and executive plan • results of the stakeholder survey in accordance with the strategy. 38 5.5 Assessment Support Materials Self-assessment reports are an important part of the impact assessment of all the universities that responded to the survey. In the majority of universities, self-assessments included qualitative impact case studies. Typically, the assessed units chose one or more studies or projects based on which they described the promotion and manifestation of societal impact. The data used by some universities in the assessments consisted only of self-assessment reports or impact case studies. In the case of these universities, qualitative data were not supported by information from databases. The decision was said to be partly due to lack of databases. However, most universities used databases and software in addition to self-assessments to collect quantitative data on the different domains of the societal impact of research. Databases were used to search for quantitative information on, for example, research and business funding, national and international cooperation, joint publications, open access publications, projects, start-ups and spin-offs, patents, licences and other intellectual property rights transfer agreements, and the added value of the companies formed. The self-assessments and assessors utilised the universities' own research information systems and HR databases. In addition to this, publication analyses were carried out with the help of databases such as Altmetric.com and the Overton citation database. Other sources of data included annual internal and other surveys on the societal impact of universities, SAP financial figures, the national statistics service, media monitoring and various reports. Background material for the assessments, such as bibliometric analyses, was also requested from external providers. In addition, some universities mentioned that they use the Scival analysis tool, which uses the publication metrics of the Scopus database, to produce their data. The survey showed that information on the university's strategy, executive plan, staff, funding, publications, visits and doctoral degrees had also been provided to support the assessments. 5.6 The Role of Parties Outside Academia in Assessments The panels used in the research assessment exercises were mainly composed of other researchers. In the survey, we wanted to explore the extent to which stakeholders have been involved in the research assessment exercises in the role of assessors. According to the survey, the use of stakeholders in research assessment exercises is not typical of Finnish universities (see Chapter 3). 39 When we asked whether the organisation uses stakeholders outside academia to assess the societal impact of research, eight universities responded positively. However, when asked to elaborate on this response, it was found that stakeholders were involved in assessing the impact of the organisation in contexts other than research assessment exercises. Some respondents described the organisation's societal impact as being supported by a working group consisting mainly of external stakeholders. In some cases, there were mentions of an advisory board, whose members represent stakeholders and act, for example, as members of assessment panels, although not necessarily as part of the actual overall research assessments. Stakeholder networks were also frequently mentioned in the responses, which assess societal impact, for example, in the context of accreditation processes. Various councils and panels were also mentioned, which meet periodically to support the societal relevance and impact of teaching and research. 5.7 Assessment Development Needs Table 5 summarises the universities' responses to the questions on the need for national recommendations and their wishes for national development work on impact assessment. Ten of the universities that responded saw a need for national impact assessment recommendations, methods and approaches. At the same time, however, it was pointed out that it is important to respect the autonomy of universities when determining the method of assessments. Similarly, the need to take into account scientific differences in development work was highlighted. With regard to development work, the view was that it is important to develop the conditions for impact, such as promoting actions and channels for disseminating information. Table 5. Universities' views on the needs for national development of impact assessment. What is your organisation's view on the need for national recommendations, methodologies and approaches for assessing societal impact? What are your organisation's wishes for the national development of impact assessment? Yes Public debate will provide ideas on how each organisation could address the issue. We do not immediately see the merit of choosing common indicators at national level, especially as measurement itself is challenging. In addition, the impact varies between different organisations. Yes: It is a good idea for experts to propose recommendations for common use. Assessments have different objectives at different universities, depending on the goals of the organisations. The key is to deepen the understanding and debate on impact and its diversity. 40 Consequently, we need a variety of ways to perceive, monitor and evaluate. Recommendations are not the same as a national assessment framework. No: The current approach has served the University's strategic work well. No: There is no reason to link the assessment of societal impact assessment, for example, to a funding model, at least not in addition to the current criteria. On the other hand, strong cooperation in assessing impact is very welcome. Cooperation is needed! Yes: Similar issues are certainly faced by all universities, so it would make sense for assessments to be based on a uniform framework, at least to a certain extent. Recognising the diversity of impact is a vital prerequisite for a common framework to work. Yes: A national common approach to societal impact must take into account that the priorities of impact differ, especially between sectors. The objectives and forms of societal impact can also vary widely within disciplines. The hope is that national recommendations on societal impact assessment will support national impact assessment. Yes: Yes, because every university deals with these matters anyway. However, the recommendations should not be too restrictive, but some tools should be made available. Where can we find a common and universal set of criteria, the SDGs? It is difficult to define impact, are all forms of action considered impact? A common understanding of what is being talked about and how to monitor, measure it? The time span of impact. Other European trends and assessment of the impact of research. The impact of operations also includes another aspect, the impact of teaching, which is intertwined in the work of universities. Yes: Common methodologies and approaches could facilitate the work; on the other hand, the recommendations should be general enough to cover all disciplines If recommendations and models are to be developed, a working group should be selected with representatives from as many disciplines as possible. Yes For example, online courses for researchers and impact training for staff. A debate in the sectors of arts and culture with regard to societal impact. Yes: The problem of impact assessment is common to all universities, so at least the development work in this area should be coordinated. Uniform guidelines and practices would make it easier to carry out comparisons between universities, but given the underdeveloped nature of the subject area, it may not be possible to define them at present. This is an important and underdeveloped theme, so further development is needed. It would make sense for universities to work together to learn from experiments and good practices. 47 6.3 Monitoring and Assessing the Societal Impact of Research Some research institutes have undergone international assessments in which an external expert panel has assessed, among other things, the overall impact of the institute's activities, its collaborative relationships and its funding base. The assessments have typically been commissioned by the supervising ministry. However, research institutes do not have a wellestablished tradition of regular, organisation-wide assessments of research activities (cf. research assessment exercises in universities; Chapter 5). Reflecting on the impact of research and other activities of the institute is generally considered useful when considering the basis for the institute's existence. [...] what would happen if we did not exist as an institute? How would the situation develop and would what we're doing affect anything? Is our impact a kind of pseudo-impact, meaning that things would in any case go approximately as they are proceeding now? The reasons for the emphasis on the societal impact of research and the perceived need to monitor or assess impact stem from both the institutes' own needs and external demands. For example, the supervising ministry and funders have different requirements in terms of impact and how to demonstrate it. In the interviews, the topic was also linked to themes of sustainability, in particular the requirements for government agencies to report on sustainability4. The need to demonstrate the usefulness of institutional research and the value of the work done is all the more important in situations where government spending is being cut. Societal crises, such as the COVID-19 pandemic and Russia's war of aggression in Ukraine, are also seen as having underlined the broad societal relevance of research carried out by research institutes. I think this aspect of impact has been emphasised over the last couple of years, that the COVID-19 pandemic was probably a big thing that changed our society in many respects and taught us things. And that's how impact is seen as more important than ever. The internal needs to assess the societal impact of research are related to the institutes' own management and development. The use of research results was also seen as a major motivating factor in the work of researchers. 4 State research institutes are part of the central government. The State Treasury recommends that ministries, agencies and institutes prepare an annual report on their activities. The State Treasury maintains a common reporting framework (State Treasury 2024). 48 Research institutes monitor societal interaction and the impact of research in a variety of ways. Impact is monitored at different levels: the whole organisation (e.g. in the context of performance-based management), at unit or programme level, and at project level. As an example of monitoring at research programme level, one research institute has designed impact pathways for research programmes that are systematically followed. Assessments of activities are also carried out at research programme level. The specific indicators monitored depend on the objectives of each research programme. As an example of project-level monitoring, in another research institute, the final reports of research projects should include a report on where the research results have been utilised. This was seen as a motivational factor to think about the utilisation of results already at the planning stage of the project. One state research institute considers both positive and negative impacts when planning projects. [...] on what geographical scale do we consider the effects, whether it is local or global, and then we consider that if we do good things in a small area, can the effects be negative in a larger area. Institutes also sought to monitor the impact of research on policy-making. However, systematic monitoring was considered difficult to put into practice (see also Chapter 4 on the challenges of assessment). One form of impact monitoring is stakeholder surveys. For example, customer satisfaction is monitored by research institutes by asking partners whether their cooperation with the institute has contributed to the company's business, responsibility, sustainability and ethics. An institute that provides open datasets has been studying how companies and other organisations use the datasets in their own activities. In addition to this, research institutes carry out general surveys on trust and reputation. Some institutes also prepare narratives that describe and seek to illustrate concretely how their activities have contributed to the use of research information in society. When asked about the key actors in assessing the impact of the institute's research, the supervising ministry was often considered an important party. In particular, the interviews highlighted the needs of the ministry and other elements of central government to make use of the research information produced by the institute. The other groups of parties that are seen as important in assessing the societal impact of research vary between institutes. For example, some institutes cooperate a lot with organisations, which made the organisations' views as users of research and development work relevant. In some institutes, other research partners were seen as key research stakeholders. Some institutes emphasise cooperation 49 with businesses, while others value cooperation with public authorities, in which case their views are considered important. Our cooperation with authorities is probably at the forefront, as is, in a way, the use of our research information by other authorities. We provide technical support to customs for border control purposes, for example, and we develop methods for them. [...] For example, we also aim to be able to support them and develop methods, new technologies and detection methods for monitoring radioactive and nuclear materials at the Finnish border. The UN's Sustainable Development Goals (SDGs) were said to be one way of linking research to wider societal challenges. State research institutes use the SDGs to monitor their activities in a variety of ways. At project level, the SDGs to which the project is related is considered. In addition to this, the institutes monitor project activities and working time allocation to see which UN sustainable development themes the institute is addressing. SDGs are also used in communication and sustainability reporting. Although the societal impact of research was seen as a key objective for research institutes in the interviews, some institutes focused their monitoring mainly on conventional indicators concerning the number of research publications and scientific impact. In addition to the number of scientific publications, the JUFO ratings or impact factors received by publishing forums, the number of citations of scientific publications, the amount of external competitive funding and the H-indices of staff were mentioned. The focus on scientific impact was justified by the difficulty of measuring societal impact and the lack of reliable indicators and data describing the phenomenon in a comprehensive way, in particular. In addition to this, scientific indicators were seen as the most natural indicators if the primary objective of the institute was to produce high quality research. In particular, the monitoring of scientific publications was considered to be useful because the data they provide is relatively reliable and comparable over time. 6.4 The Challenges of Assessing Societal Impact Although based on the interviews the societal impact of research and its documentation were considered very important by the research institutes, and although the interviews revealed that the topic was discussed a lot in the institutes, it was considered difficult to measure the impact in practice. In the interviews, there was extensive discussion about which indicators would be relevant to measure the societal impact of research by institutes and how the results of possible indicators should be interpreted. There must be a discussion and development of how to get such [indicators] that measure the right things and that are really validated and evidence-based. 50 Of course, impact is something [that] is talked about a lot, but whether it can be measured with a specific indicator is really difficult to say. The societal impact of research conducted by state research institutes was outlined as very broad and multifaceted, which calls for diversity to be taken into account in assessments. However, the lack of existing datasets and databases was seen as a challenge: it was not considered possible to use existing data to describe or assess the different aspects of the societal impact of research. [...] The challenge is that they [existing datasets and databases] do not really reflect the complex reality very well, as there are so many different activities and goals, so the numbers tell us very little on their own. Then you need the qualitative [assessment] and also self-assessment, but then it comes to how subjective or objective it is, that perhaps that kind of multi-methodological assessment is ultimately the best, however, and that there are different ways and different perspectives. If impact were to be assessed more systematically, some interviewees felt that there was a risk that the focus of operations would be diverted to activities or themes where indicators are more readily available. If you clearly begin to measure a particular thing, then the operations are often focused around it. Reporting and assessment of impact was therefore seen as inevitably incomplete. As it was seen as difficult to reliably and comprehensively measure the societal impact of research, some of the experts interviewed felt that the limited resources available should instead be directed towards raising awareness of research and improving communication. This would improve the usability of the research, rather than focusing on the design and implementation of the impact assessment. As mentioned above, due to the challenges of measurement and assessment, the monitoring focus of many institutes, according to interviewees, was on the scientific impact of research at both organisational and project level. 6.5 Development Needs in Assessment The interviewees addressed the need for improvement in the areas of societal interaction and monitoring and assessing impact. The more general development needs identified in the study are discussed separately in Chapter 8. The interviews showed that the research institutes were at different stages in their development. Some research institutes aimed to develop the basic foundation for assessment. Institutes with more experience in monitoring and assessment, on the other hand, sought not only to strengthen certain forms of cooperation and aspects of impact, such as business cooperation and economic impact, but also to develop and systematise the methods of assessment. The need for improvement was not only related to assessment, but 51 also to a broader reflection on the future impact of research and other activities of the institutes. A few institutions identified more systematic planning and assessment of impact at project level as areas for improvement. One interviewee felt that systematisation was important in order to gather useful information from assessments to support the planning of future projects. [...] almost all work, at least substantive work and research, is done in projects. In what way we could perhaps introduce a slightly more systematic way of thinking about the societal impact of both planning and assessment and, in a way, have a feedback loop in the planning of subsequent projects. This is a major area of development for us [...] One interviewee stressed the need for a stronger civil society perspective in the design, implementation and assessment of projects in order to ensure that the needs and ideas of different groups are more balanced in the research and development process. [...] projects should be considered even more broadly, so that better projects and better consideration of the perspectives of different parties could be achieved in a real and equal way, exactly from the perspective of [...] impact, so that it is not be so blinkered, and so that different perspectives are also taken into account on a broader scale. So that's what I think we have to do and is on the to-do list at the same time. Some interviewees wondered what the appropriate level of ambition for the societal impact objectives of research would be, especially in the context of performance-based management by ministries. Broad, general objectives were considered difficult from the perspective of governance and assessment, as individual organisations are not the only parties that influence the achievement of the objectives. The interviewees also highlighted situations where, for example, the impact targets set out in the institute's performance contract were too ambitious in relation to what could be achieved with the institute's own activities. This had led to the need to specify the level of ambition, to clarify the relationship between the institute's activities and impact, and to consider which societal mechanisms or parties could actually be affected by the research activities carried out by the organisation. One interviewee raised the need for more discussion on what kind of societal impact is expected from state research institutes; for example, whether global impact is expected and, if yes, what kind of global impact. Clarifying the objectives would make it easier to monitor the societal impact of institutional research and specify what should be monitored. Some interviewees pointed out that even the supervising ministries did not always seem to have a clear vision of what kind of societal impact is expected from research institutes and where they should focus their efforts. 52 And then, in my opinion, what has received relatively little attention in this discussion on impact is EU versus global level, where exactly people want to have an impact. [...] Especially now that funding is getting tighter and so on. If you look at what we need to do, is it something we need to focus on or do state research institutes have a role to play beyond Finland's borders. Interviewees considered it important that the indicators used to measure the societal impact of research institutes are diverse. The research activities of institutes are typically diverse within a single institute, involving many different stakeholders and impact pathways. In addition to this, there is wide variation between institutes depending on the nature of research and its objectives, funding and cooperation relationships. At project level, the assessment indicators depend on the projects’ objectives and the cooperation relationships. Some interviewees considered it to be useful for the development of assessment that state research institutes have a sufficiently uniform view of the societal impact of research. These interviewees considered a shared understanding of societal impact and the transparency of assessment criteria as important starting points for assessment. Joint development work between research institutes was carried out in 2024 under the coordination of Tulanet, a consortium of state research institutes. And in general, to make the terminology consistent and other things, I think it's really good that this discussion is now taking place at many levels. [...] then again the criteria or indicators or things like that, that they are objective, transparent and that everyone knows what we're talking about, not having different criteria and saying ‘oh, you use those criteria’. However, the majority of interviewees stressed that there should not be a single uniform and binding framework or set of indicators for state research institutes to assess their impact. The diversity of operations, research, objectives, stakeholders, resources and disciplinary practices and traditions of research institutes was felt to be so big that the diversity of assessment priorities, indicators and data should be taken into account and allowed. According to interviewees, instead of common indicators, the aim of joint development could be, for example, to create a framework for the societal impact of research at a general level. A framework of dimensions of societal impact of research, outlining what impact can include was mentioned as an example. This would allow research institutes to plan and reflect on their own operations in relation to the perspectives that are central to each institute. General information sharing, including case studies, tools, databases, indicators, benchmarks, and discussion of assessment challenges, was also considered useful. [...] you have to be aware that research institutes are different. They even have different functions. I'm not talking about the subject, but about what it should be like, so that there are indicators that 53 measure the different kinds of impact in a somewhat equal way. My guess is that it's not a very easy task. [...] some of these challenges that we have with smaller research institutes or, although we are a big organisation, our research is minor, and we don't have our own systems for all this [...] that perhaps this is why we also understand the challenges [...] that we are very different [...] that the emphases and resources and systems and background, history and other things, that everything has an effect. We do not come from the exactly same mould, but we are on the same line on these things. Yes, assessment is difficult because we have a terribly broad field of science in which we operate. And it has been noted that the assessment also differs between disciplines [...] take more medical research of this kind, for example, versus very behavioural-oriented research. Somehow those disciplines have different conventions. Some interviewees critically questioned the extent to which it is appropriate to use the limited resources of the institutes to reflect on impact indicators and carry out assessments in the current tight financial situation of the central government. In particular, the design and implementation of assessments covering the entire research activities of institutes was seen as laborious and costly in relation to the benefits they bring. Many interviewees stressed that resources should be allocated primarily to research itself, and only secondarily to improving communication and promoting the use of research results. There has been talk about whether we should do some kind of institutional assessment and so discussions of this kind have been had, but then the costs versus the potential benefits have been discussed, so nothing has been done, at least not yet. We are able to develop our activities ourselves through continuous development and refinement and other activities of this kind, so assessment probably does not necessarily add much value to this. But it may be that other research institutes are in a somewhat similar situation, that the financial resources are what they are, so investing in it may be a fairly distant goal. [...] then, of course, there is also the operational challenge, which is related to the fact that there are quite a few things we would like to monitor and assess, but the data is not available or would require quite substantial data collection, something like surveys or massive data collection. And then perhaps it no longer corresponds to the purpose, which is that we want to do the research and not just assess its impact. 54 7. ASSESSING THE SOCIETAL IMPACT OF RDI ACTIVITIES AT UNIVERSITIES OF APPLIED SCIENCES 7.1 Background There are 22 UASs operating under the administrative sector of the Ministry of Education and Culture at the time of the study, between 2024 and 2025. In addition to these, Åland University of Applied Sciences operates in Åland and the Police University College operates under the Ministry of the Interior. In the field of Finnish higher education, UASs are a new institution compared to universities. The emergence of UASs dates back to the 1990s, with a pilot phase in 1991–1996 and a subsequent consolidation and start-up phase in 1997–2003 (Rauhala et al. 2016). The young age of UASs compared to universities also means that some of their practices are less well established in the historical context of higher education. Together with universities, UASs form a so-called dual model of higher education (Lampinen 1994; Lampinen & Savola 1995), where the division of duties is based on the different tasks of the two institutions. According to the Universities of Applied Sciences Act (932/2014), in addition to their teaching activities, UASs are required to carry out applied research, development and innovation activities and artistic activities that serve UAS education, promote working life and regional development and support the reform of the economic structure of the region. At UASs, research and development operations are referred to as RDI, which covers research, development and innovation. The development of impact assessment at UASs can therefore be seen as part of a broader effort towards identifying and supporting the specific features of the societal impact of their RDI activities. The development of RDI in UASs has been described mainly as a "bottom-up" activity, which has created space for different strategic decisions and, consequently, shaped the RDI profiles of the organisations (Maassen et al. 2012). There are differences in the range of fields of study offered by the UASs, which has contributed to shaping the specific focus areas of RDI activities of each UAS. In addition to this, the geographical location of the UASs has influenced the type of partnerships they have formed with local businesses and public sector parties, especially in areas where UASs play a strong role in local innovation and regional development. Funding profiles also influence the extent to which UASs can participate in research and development projects at both national and international level. (Maassen et al. 2012.) 55 7.2 Societal Impact of RDI Activities Due to the applied and development-oriented nature of RDI activities, the societal impact of RDI was seen in the interviews as a natural and important goal of RDI activities. The impact of activities was described from the points of view of society, regions and stakeholders. At the level of society, the impact of RDI was linked to themes such as regional prosperity, vitality, employment and improved competence. These areas of impact were also seen as part of the basic mission of UASs. RDI activities at UASs are characterised by the fact that they aim to have an impact on the economy and employment in the region where the UAS is located. To achieve regional impact, projects are typically carried out in partnership with local organisations, such as businesses, public organisations, universities and associations. RDI activities in joint projects were seen as supporting regional development. RDI activities were seen as being built on the interests and cooperative relationships of each region: [...] the strategy of the university of applied sciences is actually based on the idea that we are regional developers in addition to teachers. In other words, the projects stem from the region’s needs, what the businesses and organisations in the region feel is important. We use them to produce projects and try to help the region develop. Cooperation with stakeholders was considered an important prerequisite for the societal impact of RDI activities. Interviewees described the creation of impact as a collaborative process, producing new knowledge and approaches for partners to adopt in their own work: Impact is created the same way, that is, if they [project ideas] come from partners and society, then you have to involve those parties. Because they then contribute to taking it forward and the impact comes through the partners when they introduce these new products, new ways of working or new ideas. According to the interviewees, the results of RDI projects were also integrated into the teaching content whenever possible. RDI projects were seen to be linked to regional labour needs and to create pathways for students to find employment in companies in their field. Some UASs also included students in RDI projects. At universities of applied sciences, RDI work is very strongly linked to what we teach, i.e. the fields of education and, in a way, the partnerships that go with that education. Because not only do we cooperate a lot with companies and, in particular, with the companies where our trainees will be employed, engineers and others, we also develop our own education through these projects. In addition to stakeholders and students, RDI activities were seen to have a positive impact on the development of the UAS’s own staff's skills. Interviewees stressed how participation in RDI projects offers not only RDI staff but also those in teaching positions an opportunity to 56 strengthen their own expertise. The development of competence was seen to have a further impact on the level of competence of the UAS as a whole and on the teaching received by students: [...] after all, the impact of RDI activities on us is an important element of impact and, in a way, societal impact. So [...] from the staff's point of view, the involvement of both teachers and RDI staff in RDI projects is, for them, the most important way to develop their own competence and thus the competence of [name of the university of applied sciences] and impact through students. RDI activities at UASs are largely carried out in individual RDI projects. However, the mechanisms of impact were not described in the interviews solely in terms of the impact of individual projects. On the other hand, long-term collaboration with the same stakeholders and on broader themes was considered essential for societal impact. Some interviewees also highlighted the long-term nature of RDI and the impact it can have as a strategic priority for their organisation. [...] we have been working with them [the same partners] on projects in certain areas for a long time. So that's why I think that long-term work also has an impact. Interviewees described how UASs are trying to develop their RDI activities in a more strategic and long-term direction. Continuity has been promoted, for example, by building coherent sets of projects that thematically support long-term development work. Continuity was seen as a prerequisite for long-term organisational commitment to certain broad themes: [...] now that our strategy is really strongly guided by what we do on the RDI side, we use terms like project family [...] there. That is to achieve some continuity, that our activities are not something like, well, we have achieved this and now we're pursuing that. In a way, we could systematically develop certain large thematic entities. And based on this, we hope at least that there will be effects and impact [...] We have recognised and acknowledged that, in most cases, the effects and impact are created over a longer period of time, so that's why we have started thinking of project families. 7.3 Reasons for Assessing Societal Impact Several reasons were given in the interviews for the importance of assessing the societal impact of RDI activities. The reasons are related, in particular, to four themes: the UAS’s own strategic work, the visibility of the role and importance of UASs in society, the satisfaction of stakeholders and the demands of funders. Assessment was seen to be important for the strategic management and monitoring of the organisation. In the planning phase of RDI projects, the project conceptualisation, the setting of objectives, the planning of partners and the outlining of the desired impact were described as taking place in relation to the strategy of each UAS. The projects were then examined from 63 8. VIEWS FOR NATIONAL DEVELOPMENT WORK In this chapter, we highlight the more general hopes, needs and challenges of impact assessment from the perspective of national development work. The assessment of the societal impact of research has different objectives. Examples from international case countries show that, in some countries, the societal impact of research is institutionalised as part of university funding allocation mechanisms. In Finnish universities, it is more typical that the objectives of impact assessment are related to learning and development. In future discussions on assessment, it will be important to distinguish between these two different objectives: on the one hand, impact indicators related to the allocation of funding and, on the other hand, formative assessment that is target-oriented and supports learning and development. Achieving these objectives requires a different approach both in the development of indicators and in the design of assessment practices. The Finnish universities, state research institutes and universities of applied sciences that participated in the study call for stronger national cooperation in developing the assessment of the societal impact of research. However, the hope is that cooperation will focus primarily on sharing good practices and learning together, rather than on building a binding assessment framework, for example. For universities, international examples such as the Dutch Strategy Evaluation Protocol (SEP) provide indications of how a national assessment framework can support a multifaceted and strategically oriented assessment of impact. The national development of impact assessment requires recognition of the autonomy of organisations and the diversity of their activities. Research and development activities differ not only between different types of organisations, but also within them, in terms of objectives, funding sources and disciplines. This diversity of contexts creates the need for an approach that allows for joint development without making the assessment too uniform. Common areas for development could include tools to support the assessment of the societal impact of research: sharing good practices and approaches, developing the knowledge base and measurement tools, and peer support between organisations. It also highlights the need to develop, in addition to quantitative indicators, qualitative ones, which allow for a more multidimensional view of impact and support a contextual understanding of it. The organisational perspective complements the discussion on the national development of impact assessment by showing that impact is not only generated by individual research projects, but is more broadly linked to how organisations support impact through their strategies, infrastructures, management practices and incentive and reward systems (cf. Sivertsen & Meijer 2025a). Taking these structures and practices into account is also essential when developing impact assessment at a national level. 64 The desire for national cooperation is also linked to the need to make the burden of assessment work lighter. It is hoped that the development of common practices will bring relief, particularly in terms of resources. The discussions highlight the need for assessment practices that are realistically feasible with the resources available to organisations. The need for national cooperation also relates to the development of databases and data collection. In some of the organisations studied, activities and outputs related to the societal impact of research are not currently collected systematically. Database coverage in this area is also limited at present. Many of the study participants are critical of the idea of large-scale data collection on different forms of societal impact, as it is seen as cumbersome, expensive and unsuitable in relation to the benefits it can bring. Such views underline the need for national cooperation in developing the knowledge base. Alongside the development of data collection, it is necessary to maintain the understanding that not all impact can be measured or easily demonstrated, and that the challenges of assessment cannot be solved solely by improving indicators. The interviews also raised the question of what happens if an assessment focuses only on what can be easily measured. Assessment is not "just assessment", but the things measured also serve as a sign of what kind of research and impact is considered desirable and valuable in society (cf. Himanen 2025). The scientific impact of research is conventionally assessed within the research community, for example, through expert panels and peer reviews of publications. However, assessing societal impact often also requires a perspective from outside the research community. Internationally, assessment practices have moved in the direction of including in assessment teams members with experience of using research in politics, central government, industry, organisations or other societal and practical contexts. The reform of assessment practices would benefit from a broader discussion on the possible roles of different groups of actors. 65 9. CONCLUSIONS AND NEXT STEPS The results of the report show that it is important to take into account the different core missions of each organisation when assessing the societal impact of research organisations. The main duties of universities are linked to research and teaching. At universities of applied sciences, teaching is a key task and it is linked to the demands of working life. The RDI activities of UASs are mainly applied and promote working life and regional development. State research institutes, on the other hand, have a wide range of research, expert and public authority tasks, which are often interlinked. The results of our report show that the practices for assessing the societal impact of research vary between universities, state research institutes and UASs, and are linked to the mission of each type of organisation and the nature of research activities. Assessment practices also vary within the same type of organisation. Although universities, for example, use relatively uniform practices for assessing impact, there are differences in the definitions of impact, the assessment criteria and the indicators used. Because of their different missions, universities, state research institutes and UASs have different research objectives, scopes, funding sources and priorities for the staff’s duties. The different tasks and natures of activities open up different perspectives on the societal impact of research and its assessment. This diversity also poses challenges for the national development of the assessment of the societal impact of research. The study highlighted the need to clarify the expectations of supervising ministries and research funders about the societal impact of research in state research institutes and UASs. It is difficult to assess impact if there is no common understanding of the kind of impact expected from the organisation in general, or if expectations are misaligned with the organisation's resources and the local and global context. The organisational autonomy of universities in Finland is relatively strong, and universities decide independently on their assessments and their objectives. However, in universities, the freedom of research, i.e. the freedom for researchers to choose their own research topics, can pose a challenge for the organisation's adoption of a strong strategic perspective on research. The national development of impact assessment also requires a clear understanding of the roles, tasks and objectives of the different actors in the research and innovation system. With regard to the assessment practices, it is essential that the complementary responsibilities and strengths of organisations are identified and considered. This creates a basis for cooperation in which different roles and operating logics are not a barrier but an asset in promoting impact. 66 Moving from the starting point of the assessment to its practical implementation, a key issue is the availability of data. The results of our report illustrate a situation in which the assessment of impact focuses primarily on those areas of activity and outputs for which systematic data can be obtained. In addition to this, the use of more informal impact case studies in assessments is already somewhat established in impact assessments. However, when developing the knowledge base for impact assessment, it is important to recognise that there is an inherent structural problem with impact assessment: not all forms of impact can be measured, tracked or linked to specific research activities. This fact is in a tense relationship with the principles of responsible assessment, which emphasise the need to recognise and value different forms of impact and to take into account the contextuality of research activities. In CoARA's development work, it has been recognised that the development of assessment practices requires sufficient financial and human resources and cooperation between actors. There is also a need for national discussion on the extent to which it is appropriate to leave the collection and maintenance of assessment data to individual organisations and, on the other hand, to what extent centralised solutions could support this work. The Publications Forum's assessment panels are an example of nationally coordinated, qualitative expert assessment of publication channels, which is more meaningful and cost-effective to carry out centrally than by each organisation individually. A similar model could also be considered a possible solution for the development of impact assessment. With regard to the practical implementation of assessments, it is also important to take into account how the organisations being assessed define impact. From the perspective of responsible assessment (e.g. Hicks et al. 2015; Wilsdon et al. 2015; Himanen 2025), the assessment should focus on things that are verifiable and within the control of the researcher, the organisation or the study (see also Muhonen et al. 2025). As in the Dutch SEP, the assessment can also focus on the strategy, i.e. the plan with which the organisation or unit aims to achieve its objectives for societal impact and interaction. Impact assessment could be improved by defining impact more clearly at different levels: interaction, concrete micro-level effects and wider macro-level impact. It is important to recognise that these can be assessed in different ways: interactions and micro-level effects can often be measured relatively reliably, unlike the wider impact. (Muhonen 2024.) In developing the practices of assessment, it is also necessary to consider to what extent it makes sense to target assessments at individual researchers or projects and where it makes sense to target assessments at the organisational or systemic level. Shifting the focus from the micro to the meso and macro levels can support a more holistic understanding of the nature of impact and how it is generated. Impact is typically a long-term process that involves a community, and it is not meaningful to assess impact solely through individual-level 67 performance. In addition to an organisational focus, it is important to develop approaches that take into account disciplineor theme-specific impact across organisational boundaries. National development should therefore consider that the assessments of organisations can highlight and form an overall picture of the societal impact of the higher education and research sector as a whole. The FIN-CAM assessment tool, planned for release in 2025, aims to support a multifaceted, transparent and systematic assessment of individual researchers' work (FIN-CAMarviointityökalun luonnos.pdf). FIN-CAM contains a wealth of examples of qualitative and quantitative indicators that can also be used to assess societal impact. However, FIN-CAM primarily provides individual assessment indicators, not a broader framework for assessing impact in the same way as the UK REF or the Dutch SEP systems presented in this report. As it is often challenging to assess impact at the level of a single researcher, it is more appropriate to look at it in broader terms. Nevertheless, FIN-CAM can also serve as a useful resource for the development of wider assessment practices, in particular by providing concrete examples of impact indicators. Assessments are carried out based on multiple parallel reasons, such as legal and financial obligations and the need to learn and improve. Our research also shows that making impact visible is a key and widely shared reason for conducting assessments in Finnish universities, universities of applied sciences and state research institutes. Assessments aim to demonstrate the societal relevance of research organisations and the benefits of public funding (cf. Donovan & Hanney 2011). In our age where research is expected to make an increasingly visible contribution to the many challenges faced by society, it is understandable that there is an emphasis on highlighting impact. At the same time, however, it should be recognised that the impact of research is never fully measurable or unequivocally demonstrable. This underlines the need to develop assessment practices that recognise constraints and respect the diversity of impact. The aim of this report is to provide stakeholders with information on the assessment of the societal interaction and impact of research from the perspective of organisations and case countries’ systems. The first report produced by the project for assessing the societal impact of research carried out by the Federation of Finnish Learned Societies, published earlier in spring 2025, included a survey of researchers’ views and experiences on the societal impact of research and its assessment. Together, these reports form an essential knowledge base for the national development of societal impact assessment, which the Federation of Finnish Learned Societies will continue to develop in line with its strategy for 2025–2030. The objectives of the Federation’s strategy include developing a research community-based assessment of societal impact, and reforming the assessment culture in collaboration with the whole science and research community. 68 REFERENCES Ahola, S. (1996). “Different but equal” – eli mikä ammattikorkeakouluissa on erilaista? In J.-P. Liljander (ed.): Erilaistuva korkeakoulutus. Collection of articles from the IV Symposium on Higher Education held in Jyväskylä, Finland, 15–16 August 1996. University of Jyväskylä. Finnish Institute for Educational Research, 37–49. Universities of Applied Sciences Act 932/2014. Finlex, up-to-date legislation. Bornmann L. (2013). What Is Societal Impact of Research and How Can It Be Assessed? A Literature Survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233. https://doi.org/10.1002/asi.22803 Commission for the Evaluation of Science (2021). Handbook for experts and for the entities being assessed with regard to description of the impact of scientific activities on the functioning of society and the economy. 19 p. Commission for the Evaluation of Science. Translated by DeepL. Donovan, C. & Hanney, S. (2011). The 'Payback framework' explained. Research Evaluation, 20(3), 181– 183. https://doi.org/10.3152/095820211X13118583635756 Hakala J., Kaukonen E., Nieminen M., Ylijoki O.-H. (2003). Yliopisto - tieteen kehdosta projektimyllyksi? Yliopistollisen tutkimuksen muutos 1990-luvulla. Helsinki: Gaudeamus. Hicks, D., Wouters, P., Waltman, L. et al. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature 520, 429–431. https://doi.org/10.1038/520429a Himanen, L. (2025). The promise of responsible research assessment. Dissertation. Tampere University. https://urn.fi/URN:ISBN:978-952-03-3920-3 Hölttä, S. (1995). Towards the self-regulating university. Publications in Social Sciences, No 23, University of Joensuu. FINEEC (2025). Korkeakoulutus. Visited 05 June 2025. https://www.karvi.fi/fi/arvioinnit/korkeakoulutus. Kulczycki, E. & Korytkowski, P. (2021). Introducing societal impact evaluation in Poland: the pilot study report. https://doi.org/10.6084/m9.figshare.16699408.v1 Act on the Geological Survey of Finland (167/2011). Finlex, up-to-date legislation. Food Authority Act (371/2018). Finlex, up-to-date legislation. Act on the Finnish Environment Institute (1069/2009). Finlex, up-to-date legislation. Lampinen, O. (1994). Suomalaisen ammattikorkeakoulun doktriini. Kasvatus, 25(4), 416–422. Lampinen, O. & Savola, M. (1995). Ammattikorkeakoulujen syntyvaiheet Suomessa. In O. Lampinen (ed.): Ammattikorkeakoulut - vaihtoehto yliopistolle. Gaudeamus, Helsinki, 26–80. 69 Late, E. & Puuska, H.-M. (2014). Tutkimusorientaatiot valtion tutkimuslaitoksissa ja yliopistoissa. Toimintaympäristöjen ja tutkimuskäytäntöjen vertailu sektoreiden välillä. In R. Muhonen & H.- M. Puuska (eds.), Tutkimuksen kansallinen tehtävä. Tampere: Vastapaino, 11–33. The Law on Higher Education and Science (2018). 261 p. Poland. Liuhanen, A.-M. (2008). How are university evaluations used? Academic dissertation, Tampere University, Tampere University Press. https://urn.fi/urn:isbn:978-951-44-7207-7 Maassen, P., Kallioinen, O., Keränen, P., Penttinen, M., Spaapen, J., Wiedenhofer, R., Kajaste, M. & Mattila, J. (2012). From the bottom up. Evaluation of RDI activities of Finnish Universities of Applied Sciences, Publications of the Finnish Higher Education Evaluation Council 7:2012, https://www.arene.fi/wpcontent/uploads/Raportit/2018/arene_uas_rdi_evaluation.pdf Minister of Education and Science (2022). Ministerial Decree on the Evaluation of the quality of scientific activity. 28 p. Poland. Translated by DeepL. Muhonen, R. (2021). Tutkimuksen yhteiskunnallisen vaikuttavuuden arvioinnin haasteet. Responsible Research, Federation of Finnish Learned Societies. https://vastuullinentiede.fi/fi/jatkokaytto/tutkimuksenyhteiskunnallisen-vaikuttavuuden-arvioinninhaasteet Muhonen, R. (2024). Makrosta mikroon, muutoksista myötävaikutuksiin – kohti vastuullista vaikuttavuuden arviointia. Open Up! blog. Open Science Accelerator. University of Turku. Muhonen, R. & Tellmann, S. (2022). Challenges of reporting societal impacts for research evaluation purposes – the case of sociology. In T.C.E. Engels & E. Kulczycki (eds.), Handbook on research assessment in social sciences. Edward Elgar Publishing. Muhonen, R., Himanen, L., & Pölönen, J. (2023). Evaluation of societal impact of the social sciences and humanities in Finland. In M. Oschner, & Z. H. Bulaitis (eds.), Accountability in academic life: European perspectives on societal impact evaluation, 95–112 https://doi.org/10.4337/9781800885738.0001 Muhonen, R., Pietilä, M., Pölönen, J. & Kärkkäinen, T. (2025). Tutkimuksen yhteiskunnallinen vaikuttavuus ja sen arviointi: Tuloksia tutkijakyselystä. Federation of Finnish Learned Societies, Online publications of the Federation of Finnish Learned Societies 18. Niiniluoto, I. (2015). Yliopistot ja ammattikorkeakoulut yhteiskunnallisina vaikuttajina: yhteenveto. Vastuullinen ja vaikuttava - Tulokulmia korkeakoulujen yhteiskunnalliseen vaikuttavuuteen. Ministry of Education and Culture, 2015:13, pp. 13–32. NWO (2025). Evaluations NWO institutes. Visited 5 June 2025. https://www.nwo.nl/en/evaluations-nwoinstitutes 70 Ministry of Education and Culture 2025. Tiede ja tutkimus. Tiedepolitiikalla tuetaan tutkimusta tekevien ja tukevien organisaatioiden toimintamahdollisuuksia. Visited 5 June 2025. https://okm.fi/tiede-jatutkimus Rauhala, P., Kantola, M., Friman, M. & Huttula, T. (2016). Ammattikorkeakoulutus tutkimuksen kohteena. Tiedepolitiikka, 41(3), 57–68. REF (2012). Panel criteria and working methods. 106 p. Research Excellence Framework, the United Kingdom. REF (2019a). Guidance on submissions. 121 p. Research Excellence Framework, the United Kingdom. REF (2019b). Panel criteria and working methods. 105 p. Research Excellence Framework, the United Kingdom. REF (2023). Research Excellence Framework 2028: Initial decisions and issues for further consultation. 34 p. Research Excellence Framework, the United Kingdom. Rekilä, E. (1996). Korkeakoulujen arvioinnista. In R. Laukkanen & K. Stenvall (eds.), Arviointi koulutusja tiedepolitiikassa, pp. 79–88. Tampere University Unit of Administrative Sciences A 9. Research Council of Norway (2017). Evaluation of the Humanities in Norway. 77 p. The Research Council of Norway. Research Council of Norway (2018). Evaluation of the Social Sciences in Norway. 51 p. The Research Council of Norway. Research Council of Norway (2021). Evaluation of Legal Research in Norway. 75 p. The Research Council of Norway. Research Council of Norway (2022a). Evaluation of life sciences in Norway 2022-2024 - LIVSEVAL protocol version 1.0. 16 p. The Research Council of Norway. Research Council of Norway (2022b). Evaluation of natural sciences in Norway 2022-2023 – Protocol version 1.0. 16 p. The Research Council of Norway. Research Council of Norway (2022c). Evaluation of mathematics, ICT and technology in Norway 2022-2024 – EVALMIT protocol version 1.0. 16 p. The Research Council of Norway. Research Council of Norway (2024a). Evaluations of specific subjects and thematic areas. Visited 5 June 2025. https://www.forskningsradet.no/en/analysis-numbers/evaluations/subject-theme/ Research Council of Norway (2024b). Evaluation of Biosciences in Norway 2022-2024. 62 p. The Research Council of Norway. 71 Research Council of Norway (2024c). Evaluation of Medicine and Health Sciences (EVALMEDHELSE). Visited 5 June 2025. https://www.forskningsradet.no/en/analysis-numbers/evaluations/subjecttheme/evaluation-medicine-health-sciences/ Research Council of Norway (n.d.). EVALBIOVIT - Self-assessment for research groups. 12 p. The Research Council of Norway. Sivertsen, G. & Meijer, I. (2025a). Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society? Research Evaluation, 29(1), 66–70, https://doi.org/10.1093/reseval/rvz032 Sivertsen, G. & Meijer, I. (2025b). Evaluating and improving the societal impact of research. In G. Sivertsen & L. Langfeldt (eds.), Challenges in research policy. SpringerBriefs in Political Science. Springer, Cham. https://doi.org/10.1007/978-3-031-69580-3_4 Smith, K. E., Bandola-Gill, J., Meer, N., Stewart, E., & Watermeyer, R. (2020). The impact agenda: Controversies, consequences and challenges. Bristol: Policy Press Research Council of Finland (n.d.) Yhteiskunnallinen vaikuttavuus. Visited 10 June 2025. https://www.aka.fi/suomen-akatemian-toiminta/mita-teemme/yhteiskunnallinen-vaikuttavuus/ Statistics Finland (2019). Valtion tutkimuslaitos. Visited 5 June 2025. https://stat.fi/meta/kas/valtutklait.htmlhttps://stat.fi/meta/kas/valtutklait.html UKRI (2023a). Explainer: dual support funding for research and innovation. Visited 5 June 2024. https://www.ukri.org/publications/explainer-dual-support-funding-for-uk-research-and-innovation/ UKRI (2023b). Explainer: QR funding and the REF. Visited 20 May 2024. https://www.ukri.org/publications/explainer-qr-research-funding-and-the-ref/ UKRI (2025). Governance. Visited 5 June 2025. https://2029.ref.ac.uk/about/governance/ State Treasury (2024). Kestävyysraportointi valtiolla. Visited 6 June 2025. https://www.valtiokonttori.fi/palvelut/valtion-konsernipalvelut/kestavyysraportointi-valtiolla/ Vipunen (2024). The Finnish National Agency for Education's statistics service vipunen.fi. Retrieved 15 November 2024. VSNU, KNAW & NWO (2003). Standard Evaluation Protocol 2003-2009. 41 p. The Netherlands. VSNU, KNAW & NWO (2009). Standard Evaluation Protocol 2009-2015. 23 p. The Netherlands. VSNU, KNAW & NWO (2014). Standard Evaluation Protocol 2015-2021. 31 p. The Netherlands. VSNU, KNAW & NWO (2020). Strategy Evaluation Protocol 2021-2027. 47 p. The Netherlands. Ministry of Finance (n.d.). Hallinnon ohjaus. Visited 5 June 2025. http://vm.fi/hallinnon-ohjaus 72 Välimaa, J. (1999). Managing a diverse system of higher education. In M. Henkel and B. Little (eds.), Changing relationships between higher education and the state. Jessica Kingsley Publishers. Wang, L., Vuolanto, P. & Muhonen, R. (2014). Bibliometrics in the research assessment exercise reports of Finnish universities and the relevant international perspectives. Tampere: University of Tampere. https://urn.fi/URN:ISBN:978-951-44-9660-8 Wilsdon, J. et al. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. DOI: 10.13140/RG.2.1.4929.1363 Wróblewska, M. N. (2025). One size fits all? A comparative review of policy-making in the area of research impact evaluation in the UK, Poland and Norway. Research Evaluation, 34, rvaf010, https://doi.org/10.1093/reseval/rvaf010 Universities Act (558/2009). Finlex, up-to-date legislation. 79 • What are the reasons, justifications and objectives of your organisation for assessing the societal impact of RDI activities? • How does your organisation support RDI activities (e.g. services and incentives for staff)? • What kind of indicators does your organisation use to assess RDI activities (if this has been considered in your organisation)? • What kind of data is collected to support the assessment of the societal impact of RDI activities (if this has been considered in your organisation)? • Does your organisation use the UN’s sustainable development themes in the assessment of the impact of RDI activities or in other contexts? How are the UN's sustainable development themes used? Assessment development • Has your organisation identified the need to improve the assessment of the societal impact of RDI activities? What are the problems or challenges in assessing the societal impact of RDI activities from your organisation's perspective? • What is your personal opinion on the extent to which national recommendations, methods and approaches would be useful for assessing the societal impact of RDI activities? What are your wishes for the national development of RDI assessment? What is your relationship with assessments carried out in universities?