Implementation proposal for language-aware assessments

Implementation proposal for language -aware assessments WORKING GROUP ON MULTILINGUALISM AND LANGUAGE BIASES IN RESEARCH ASSESSMENT Implementation proposal for language-aware assessments 1 Implementation proposal for language-aware assessments Table of Contents Introduction ............................................................................................................................... 5 Proposal 1: Make multilingualism a value in research assessment .............................. 6 Proposal 2: Assess and improve language diversity, competences and practices ... 6 Proposal 3: Create expectations and criteria recognizing multilingual work ............ 7 Proposal 4: Integrate multilingualism into assessment processes ............................... 8 Annexes to support implementation of language-aware assessments ....................... 9 Implementation proposal for language-aware assessments 2 Please cite as: CoARA WG on Multilingualism and Language Biases in Research Assessment (2025). CoARA Output. Implementation proposal for language-aware assessments. DOI: https://doi.org/10.5281/zenodo.17881907 Document Information Output Title Implementation proposal for language-aware assessments Working Group Multilingualism and Language Biases in Research Assessment Topic Draft Version 4.0 Publication Date 12 December 2025 Output Type Actionable policy resource Producer(s) WG on Multilingualism and Language Biases in Research Assessment Project Leader(s) Monica Dietl, Emanuelle Gardan, Martine Garnier-Rizet, GianMaria Greco, Emanuel Kulczycki, and Janne Pölönen Editor(s) Janne Pölönen Reviewer(s) WG members, TF3 Focus Group, Community consultation Other contributors André Brasil, Manuel Célio Conceição, Josep M. Cots, Lucimar Dantas, Bessie Dendrinos, Soraya García-Sánchez, Petteri Laihonen, Carlos Soler Montes, Eva Nueno, Michael Ochsner, Ginevra Peruginelli, Ismael Rafols, Mimi Urbanc Status Final Revision history Version Date Revised by Comments 1.0 09-05-2025 Janne Pölönen First draft 2.0 23-05-2025 Janne Pölönen WG members’ feedback 3.0 03-10-2025 Janne Pölönen TF3 Focus Group feedback 4.0 12-12-2025 Janne Pölönen Community consultation feedback Implementation proposal for language-aware assessments 3 List of Acronyms CERF Common European Framework of Reference for Languages CoARA Coalition for Advancing Research Assessment CRIS Current Research Information System CV Curriculum Vitae WG Working Group Implementation proposal for language-aware assessments 4 Summary This implementation proposal is aimed at leadership and management of higher education institutions, research and funding organisations to facilitate implementation of CoARA commitments. The proposal has been produced by the CoARA WG on Multilingualism and language biases in research assessment. The document presents four proposals for the possible implementation of language-aware assessments, including concrete actions and steps to guide the implementation. Implementation proposal for language-aware assessments 5 Introduction This implementation proposal is aimed at leadership and management of higher education, research and funding organisations to facilitate implementation of CoARA commitments by recognising multilingualism and mitigating language-based biases in research assessment. The proposal has been produced by the Coalition for Advancing Research Assessment (CoARA) Working Group (WG) on Multilingualism and language biases in research assessment. This document presents four proposals for the possible implementation of language-aware assessments, including concrete actions and steps to guide the implementation, and supporting annexes: • Proposal 1: Make multilingualism a value in research assessment • Proposal 2: Assess and improve language diversity, competences and practices • Proposal 3: Create expectations and criteria recognizing multilingual work • Proposal 4: Integrate multilingualism into assessment processes • Annexes to support implementation of language-aware assessments Language hierarchies in assessment contribute to global and socio-economic inequality in science, affecting researchers in all fields and citizens across all countries. The implementation proposal builds on balanced multilingualism as a research policy approach to recognise, without exclusion or priorities, that multiple languages are used and needed at individual and community level in the practice of science, in scholarly publications and in academic communications. It also addresses linguistic disadvantages to researchers and organisations in assessment criteria and processes and supports the use of researchers’ full linguistic repertoires to facilitate dissemination, understanding and co-construction of knowledge and meaning across languages, audiences, and contexts (see Annex I). Ideally, researchers can choose their language for publications, applications and communications. Because language use in research, communication and assessment is context-specific (see Annex II), this proposal is flexible and adaptable to institutional, disciplinary, cultural, and regional differences. Organisations implementing languageaware assessments need in practice to consider feasibility, legal frameworks, and available resources. Shared practices and resources can be developed in collaboration with other organisations facing similar implementation challenges, possibly within CoARA communities as well as networks, alliances and associations of higher education, research and funding organisations. Implementation proposal for language-aware assessments 6 Proposal 1: Make multilingualism a value in research assessment • Action 1: set up a committee, or assign the task to an existing body, for promoting language awareness, multilingualism and addressing language bias in research assessment – as outlined in this implementation proposal – in your organisation • Action 2: incorporate multilingualism into your organisation’s CoARA Action plan o Step A: review all assessment processes of your organization to identify language requirements and barriers (including legal), adverse incentives and language biases (see Annex III and V) o Step B: prioritise issues and create a roadmap with schedule for addressing and alleviating the identified language barriers, adverse incentives and language biases o Step C: implement the roadmap to change criteria, methods, data, guidance and support of assessment processes, monitor progress, collect feedback, and revise processes if needed • Action 3: create or update relevant strategies and policies of your organisation o Step A: review international and national legal frameworks, policies and initiatives your organisation has signed or adheres to (see Annex IV) o Step B: consult international and national expert bodies, committees or societies on language and multilingualism policies and practices o Step C: review and update (if needed), strategies and policies of your organisation in light of the international and national policies Proposal 2: Assess and improve language diversity, competences and practices • Action 1: create an overview of active and passive language skills, practices and needs in your organisation o Step A: create an overview of language skills of the research, teaching and administrative personnel (e.g. using the Common European Framework of Reference for Languages – CERF – as a standard for describing language abilities) o Step B: create an overview of communication practices, purposes and languages of the research, teaching and administrative personnel (e.g. by means of survey (see Annex VI) or collecting and analysing research information on publications o Step C: create an overview of the languages needed to most effectively communicate knowledge and contributions to relevant experts as well as diverse target audiences outside academia Implementation proposal for language-aware assessments 7 • Action 2: identify gaps and priorities for improvement o Step A: identify strengths, weaknesses, opportunities and threats in the language skills and competences, languages used and needed in your organisation o Step B: identify and prioritise areas of improvement and create a roadmap for better supporting the language skills and competences, improving language practices and meeting the language needs of target audiences • Action 3: develop capacity, services and infrastructure to support language skills and multilingualism o Step A: offer workshops, courses, or mentorship programmes to develop multilingual skills and competences o Step B: offer resources for human translation o Step C: point to suitable and secure resources (tools, machine translation, AI-based solutions, etc.) Proposal 3: Create expectations and criteria recognizing multilingual work • Action 1: implement criteria that recognise and reward contributions in any language as well as in many languages (e.g. through translations or adaptations) o Step A: explicitly state that research contributions are recognised irrespective of language and that multilingual communication strategies and translanguaging practices are positively valued o Step B: avoid criteria and indicators that explicitly use or implicate communication language as a proxy of quality, impact or internationality of contributions o Step C: recognise and reward citizen and community engagement, outreach activities and collaboration, public science communication, and media presence in the language(s) of the local/regional/national/minority communities • Action 2: Positively value and acknowledge as merit the creation, translation, or harmonisation/standardisation of field-specific terminology in any languages relevant to research and its users (e.g. bilingual or multilingual glossaries, vocabularies, and ontologies) • Action 3: recognise and reward promoting multilingualism in research process, outputs (e.g. data, publications, translations), teams and collaboration, conferences and events, teaching and education, community engagement, publication channels (e.g. journals, books), and assessments Implementation proposal for language-aware assessments 8 Proposal 4: Integrate multilingualism into assessment processes • Action 1: support multilingual communication throughout assessment process o Step A: introduce options for researchers to communicate (e.g. submit forms, narrative CVs, and other documentation), provide and receive feedback in a variety of languages determined according to the context o Step B: introduce options for expert evaluators to use a variety of languages determined according to the context, and provide them training to carry out language-aware assessment (see Annex III) o Step C: communicate progress, decisions and results of assessment in a variety of languages determined according to the context • Action 2: consider language diversity and competences relevant to the assessment context when selecting and instructing evaluators o Step A: recruit diverse pools of evaluators who are attentive of language diversity and challenges and can assess contributions in the relevant languages o Step B: make evaluators aware of language biases and disadvantages for second language users in written and oral communications, and explicitly instruct evaluators to focus on the scientific quality rather than the language of communication • Action 3: Make multilingualism integral part of quantitative and qualitative information supporting assessment o Step A: create an overview of information currently used to support assessments and how it represents contributions and facilitates evidence in different languages o Step B: create a plan for transitioning towards more open, linguistically inclusive and multilingual research information (e.g. local CRIS and repositories / global databases OpenAlex and OpenAIRE) o Step C: offer options, templates and guidance for providing qualitative information (e.g. in form of Narrative CV or self-assessment) in a variety of languages determined according to the context o Step D: develop multilingual user-interfaces, guidance and metadata for the research information