A Practical Guide to Implementing Responsible Research Assessment at Research Performing Organizations

This information may be freely used, copied and adapted for non-commercial purposes, provided that the source is acknowledged. MAY 2025 This publication is licensed under the Creative Commons Attribution-Noncommercial CC BY-NC. at Research Performing Organizations A Practical Guide to Implementing Responsible Research Assessment Table of Contents Foreword 4 Resources and references 61 Annex: Acknowledgments 65 1 INTRODUCTION 6 1.1 How to use the Guide 7 1.2 The goal and principles of Responsible 9 Research Assessment 2 KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY 13 2.1 Engaging the organization leadership 16 2.2 Working with the community at your organization 18 2.3 Mobilizing resources to develop and 20 implement an RRA strategy 2.4 Convening a working group/task force 22 2.5 Exploring the landscape for RRA 24 2.6 Reviewing current assessment practices 26 2.7 Developing a strategic vision for RRA 28 2.8 Developing effective engagement 30 and communication plans 2.9 Monitoring and updating your strategy 32 3 KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT 35 3.1 Recruitment, hiring, and promotion decisions 38 3.2 Institutional awards and internal grants 41 3.3 Evaluating internal research units 44 3.4 Developing corporate communications 50 4 GLOBAL INITIATIVES AND EXEMPLARS RELEVANT TO RRA 53 4.1 Leadership and guidance on aspects of open science 54 4.2 Driving research assessment reform 56 4.3 Encouraging a holistic and inclusive view of research 59 4.4 Enabling evidence-based policy and research integrity 60 A PRACTICAL GUIDE TO IMPLEMENTING RRA AT RPOs 4 FOREWORD When the San Francisco Declaration on Research Assessment (DORA) was announced in 2013, it aimed to raise awareness of the problematic metrics used to assess research and researchers and heralded a new era of research assessment - an era where researchers are assessed based on a broad range of contributions to knowledge, research, and society, with a focus on quality and impact. The Declaration itself provided 18 recommendations for the different actors in the research system to consider in undertaking research assessment processes and practices1. These recommendations aimed to enhance the transparency of metrics used to assess research across different contexts, and to expand beyond a focus on what can be easily counted. In practical terms over the past decade, the focus of DORA has been to raise awareness of the need for research assessment reform and to support research organizations in the development and implementation of policies and practices that act to foster a given institution’s mission-driven goals. The DORA team has created case studies and coordinates a searchable repository, Reformscape, to highlight examples of innovation and change in research assessment policies and practices occurring around the world2,3,4. From our experience working with many organizations over the past decade, we know that introducing new approaches and making changes to established research assessment practices is invariably challenging and requires input from many stakeholders working within what are complex research performing organizations. Over the past decade, we have learned that there are common factors that influence and impact research assessment reform across organizations. This Guide is intended to draw upon this learning and to provide practical guidance and tips for research performing organizations wanting to take steps to review or reform how they assess research and researchers. Now with over 26,000 signatures - from individuals and institutions across the world - the awareness of the need for responsible research assessment (RRA) reform is widespread and growing. A PRACTICAL GUIDE TO IMPLEMENTING RRA AT RPOs 5 Our target audience is the diversity of individuals working in research performing organizations who are seeking to make a positive change to research assessment practices, including but certainly not limited to those in research leadership roles (e.g. Provost, Vice President of Research, Dean), faculty, administrative managers and staff, as well as those responsible for shaping organizational policies and recruitment practices, research communication and outreach staff, and other teams who may be involved in shaping research culture-related initiatives. We know that one size does not fit all in the practice of research assessment. Each organization and jurisdiction faces its own set of complex challenges. The Guide is intended to be used flexibly and where possible, be discipline and geographically agnostic and we encourage organizations to use this Guide in ways that work best for them and their context; use it in its entirety or draw upon the pieces that resonate most strongly, always adapting and co-creating with your communities. This is the first version of the Guide, and one that we envision being modified and updated over time - and especially being enriched with real examples and through feedback from research performing organizations as they start to use the Guide. The Guide reflects the hard work of many contributors. We particularly want to recognize and thank the participants of a January 2025 working meeting in Washington, DC, USA who provided valuable reflections and input to help shape the Guide. We would like to thank the Arcadia Foundation who provided funding to support the development of the Guide. Finally, we would like to thank the DORA steering committee and executive board, and the DORA staff team, current and past, without whom the creation of the Guide would not have been possible. Ginny Barbour, Kelly Cobey, Rebecca Lawrence DORA Co-Chairs and Vice-chair May 2025 This Guide is intended to draw upon this learning and to provide practical guidance and tips for research performing organizations wanting to take steps to review or reform how they assess research and researchers. A PRACTICAL GUIDE TO IMPLEMENTING RRA AT RPOs 6 A PRACTICAL GUIDE TO IMPLEMENTING RRA AT RPOs Introduction 1 This Guide provides practical guidance, resources, and illustrative examples, for research performing organizations (RPOs) who are looking to shape and deliver responsible research assessment (RRA) practices. This includes organizations who are developing an RRA strategy, reforming existing policies and practices, or simply wanting to take steps towards more holistic and inclusive approaches to research assessment. We know that many RPOs are keen to reform their research assessment practices, but lack the time, space and/or expertise to reflect on how to do this, and either don’t know where to start or simply give up because the task of making changes can seem so daunting. RPOs also exist within an external research system, and there are often external actors and influences (including national assessment processes and funder mandates) that impact upon how organizations can act. The Guide presents a range of activities and prompts that we know, through working with the DORA community, can make a real difference in delivering changes to research assessment. Practical examples are provided to show how organizations have navigated the opportunities and challenges encountered along the way. 1.1 How to use the Guide 1.2 The goal and principles of RRA 6 7 1 The Guide is not intended to present a definitive approach to responsible research assessment; indeed, each organization is unique and what works at one organization may not work at another. The Guide is organized into Chapters; each Chapter stands alone, and we provide lots of options to jump across Chapters and to relevant sections. Readers are invited to draw upon the elements that resonate and have most relevance to their organizational context. Chapter 1 provides the context and presents some underpinning principles that typically inform responsible research assessment approaches. Chapter 2 presents nine activities that can help to catalyze research assessment reform, and we hope will provide ideas and inspiration to others. These activities are: 1. Engaging the organization leadership 2. Working with the community at your organization 3. Mobilizing resources to develop and implement an RRA strategy 4. Convening a working group/task force 5. Exploring the landscape for RRA 6. Reviewing current assessment practices 7. Developing a strategic vision for RRA 8. Developing effective engagement and communication plans 9. Monitoring and updating your strategy The Spotlight on the SPACE Rubric in this chapter provides guidance on how to use the rubric to support RRA. CHAPTER 1: INTRODUCTION How to use the Guide CHAPTER 1.1 8 Chapter 3 provides practical tips on how organizations can embed responsible research assessment practices into policies and practices that can directly impact upon researchers’ careers, specifically: 1. Recruitment, hiring and promotion decisions 2. Institutional awards and internal grants 3. Evaluation of internal research units 4. Developing corporate communications The Spotlight on Narrative CVs goes into detail on how this descriptive approach provides a more holistic view of a researcher's career. The Spotlight on Early Career Researchers highlights the desire among this population for research assessment approaches that value a diversity of outputs and contributions. Chapter 4 introduces a range of global initiatives and frameworks that align with RRA. We end the Guide with references and resources mentioned. We welcome feedback on the Guide - indeed, it is intended to be ‘owned’ by the RRA community. To this end we intend to evolve the Guide over time and add to the examples, use cases and advice included based upon real insights, feedback and learnings. If you create new materials, policies, practices, or tools related to RRA, please consider sharing these with DORA and the broader community through Reformscape, DORA’s online digital repository of RRA materials that collates criteria and standards that RPOs are using for hiring, review, promotion, and tenure. Please keep in touch and send questions and feedback to the DORA team at: [email protected] CHAPTER 1: INTRODUCTION CHAPTER 1.1 How to use the Guide 9 Before diving into the key activities in developing and implementing an RRA strategy, we first want to reiterate why RRA is important and its goals and underpinning principles. The term “Responsible research assessment” has emerged as an umbrella term for approaches to assessment that incentivize, reflect, and reward the plural characteristics of high-quality research, in support of diverse and inclusive research cultures5,6. It is ‘responsible’ in that, by adhering to a set of broad principles, and living those in practice, it encourages and incentivizes the types of research behaviors that deliver benefits in terms of outputs, outcomes, and impact to both RPOs and the research system more broadly. The goal of RRA is to support and encourage approaches to assessing research performance and impact, that focus on a holistic view of researchers, the research processes, outputs, outcomes, and impacts of research. RRA embraces a values-driven approach that aligns with an organization’s mission and values, promoting clear, transparent, open, and fair criteria in research assessment, ensuring non-discriminatory practices. CHAPTER 1: INTRODUCTION The goal and principles of Responsible Research Assessment CHAPTER 1.2 16 Possibly the activity that can have the most significant impact on organizational change, and in being able to develop and implement an RRA strategy, is to secure the ‘buy in’ of senior leadership. This could be engagement with leaders at the very head of the organization, so Vice Chancellors and/or Directors of Research, but faculty and department leaders can be equally influential in recognizing the need for change within a local context and can then support the case for more widespread change13,14. Leadership support for the RRA can help to mobilize the resources that are likely to be needed for the development and implementation of a strategy, including, but not limited to staff time; development of messaging and materials; and planning and delivery of engagement activities to support effective strategy implementation (e.g. town halls, webinars, workshops). Seek out organization leaders who might have previously spoken out about the need for research assessment reform or who may have engaged with global initiatives such as DORA, or are advocates of other relevant institutional changes such as open science16, interdisciplinarity, community engagement, societal impact17,18, research integrity and reproducibility, and may therefore be motivated to get involved with RRA efforts. CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Engaging the organization leadership ACTIVITY 1 Find out how the Department of Psychology at the University of Maryland overhauled their evaluation policies15. University of Maryland CASE STUDY 17 CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY ENGAGING THE LEADERSHIP Things to Think About How can you articulate the value proposition of RRA with your leadership? What opportunities can you propose for leadership to engage with and learn from other institutions within (inter)national networks? What steps can you take to understand current priorities and how can you integrate RRA principles within these rather than as a separate initiative? How can you encourage leadership to actively involve the broader research community in shaping RRA principles and practices? What role can you suggest leadership play in addressing potential resistance? While some organization leaders may be supportive of the need for research assessment reform, many leaders may have lived and worked through more traditional approaches to research assessment and be unfamiliar with the arguments or may not see the need for change. Making time to listen to the perspectives of leaders can help you to shape the arguments and build the evidence and use cases for reform and within your own organization context. DORA has created a factsheet that presents ‘five common myths’ about research assessment and evaluation, based on real examples that can be useful to draw on when starting conversations with organizational leaders. Framing RRA as ‘evolution rather than revolution’ can be helpful in making the case to leaders. An incremental approach can be especially helpful where there is skepticism about the potential benefits and/or impact of such change on the organization’s national or international competitiveness, or where there are limited resources to deliver any change. Discover common myths about research evaluation reform, and how they exist both inside and outside of academia19. Five Common Myths RESOURCE ACTIVITY 1 Engaging the organization leadership 18 There are a range of communities across RPOs whose participation and engagement in RRA initiatives can be hugely important in driving change. RPOs are home to diverse communities, working across research and professional roles, that can drive research assessment reform, including faculty researchers, early career researchers and students, librarians, administrators, and research managers. These communities are also working on trajectories such as open science, interdisciplinarity, societal impact to name a few, and joining forces may prove beneficial. Faculty members, working across disciplinary areas and at all career stages, can be important champions for RRA. Early Career Researchers (ECRs) can be particularly engaged in conversations and initiatives that can have a very real impact on their current career progression20. As such ECRs can bring fresh perspectives to organizational debates around academic incentives and their impact upon career progression (see Spotlight on Early Career Researchers on page 48). Librarians, research managers and administrators help to shape and implement institutional policies that can guide assessment and evaluation frameworks, including setting out the criteria and information that could be included. It can also be helpful to engage individuals or groups outside of your organization, including community groups that might partner with your organization and alumni of the organization who may be working in a variety of roles across sectors and industries. CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Working with the community at your organization ACTIVITY 2 WORKING WITH THE COMMUNITY Things to Think About Who are the different communities and their (in) formal leaders that have been encouraging discussions about RRA in your institution? How can you tailor your engagement strategies to effectively reach and involve the different communities interested in and affected by RRA, including those who left academia? How can you create opportunities for faculty researchers to become champions for RRA within their disciplinary and expertise areas? How will you ensure that ECRs and societal partners, who can bring fresh perspectives on academic incentives and impacts, have meaningful opportunities to influence RRA principles and practices? In what ways can you empower librarians and research managers and administrators to contribute with their expertise? Explore how the U.S. societal impact community is transforming universities to align research with societal needs21. Advancing Research Impacts RESOURCE 19 ACTIVITY 2 Working with the community at your organization Engaging the broader community in discussions around the value of research can help to ensure that the real and tangible benefits to society, innovation, and economy are considered in approaches to research assessment. As such, to ensure that the RRA framework is authentic, relevant and widely accepted, a comprehensive consultative process involving surveys, open forums, and feedback mechanisms is advisable, to gather input and distribute ownership from the diverse communities within the RPO. This aims to include their perspectives and secure their advocacy and involvement - perhaps as part of a working group or task force (activity 4) - which means they will be equipped for, invested in, and empowered to make the case for research assessment reform. As you reflect on your community engagement strategy (activity 8), consider if tailored communication, training and resources for faculty, ECRs, librarians, and research managers and administrators are needed to ensure they understand RRA principles and their respective roles in its implementation. Be aware of power imbalances arising from career stage, varying levels of central vs. local implementation autonomy, disciplinary resistances, and historical prioritization of academic roles in research assessment reform efforts. CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Learn how the University of Tokyo engaged with the broader research community at multiple levels across their organization22. University of Tokyo CASE STUDY 20 Ensuring that you have the resources - people, time, materials - to develop an RRA strategy is as important to its success as securing the initial buy-in and commitment to develop the strategy. Despite good intentions, many approaches or strategies lack the potency to have their intended impact because they have been poorly implemented. When thinking about developing a strategy that might require significant resources, consider starting small and pilot test an approach to help you to understand the potential resource and broader implications of a new approach or changes. Pilot testing can help you provide real evidence of the likely impacts and secure buy-in to a new approach while mitigating the risks and allow you to get things right before considering scaling. CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Mobilizing resources to develop and implement an RRA strategy ACTIVITY 3 Learn about how the University of Glasgow mobilized resources to support the implementation of its RRA strategy23. University of Glasgow CASE STUDY MOBILIZING RESOURCES Things to Think About What resources do I need, how much it will cost and over what timespan? What compelling arguments can you present to secure the necessary (financial) resources? Are there resources or networks that currently exist in your organization that you can repurpose or lever to support RRA efforts? What can you implement to support and value those engaged in reviewing criteria, developing tools, and participating in training initiatives? Looking beyond the initial implementation phase, how will you ensure sustained commitment of resources and leadership attention? 21CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Things that you may need resources for include: Seizing opportunities to talk about what you are doing and why Recruiting and rewarding members of a working group or task force Reviewing and aligning existing institutional policies and processes Creating training and ongoing support and guidance materials Adapting IT systems for RRA (e.g., diverse contributions, narrative CVs) Developing the working group’s remit and expected time commitment Drafting and writing the strategy and associated guidance Ongoing monitoring and evaluation of the overall RRA implementation Engaging with (inter)national networks for learning and collaboration ACTIVITY 3 Mobilizing resources to develop and implement an RRA strategy Chapter 2, Activity 1 Chapter 2, Activity 4 Chapter 2, Activity 4 Chapter 2, Activity 5 Chapter 2, Activity 6 Chapter 2, Activity 7 Chapter 3 Chapter 3 Chapter 3 Chapter 2, Activity 9 Chapter 2, Activity 8 Phasing or pilot testing of RRA approaches, and analysis of impact Developing a communications and roll out plan and actioning it 22 Bringing together a group who possess the expertise, experience and can provide broad representation across the organization to drive change can lead to increased awareness and commitment to the RRA strategy. Ideally the group should include individuals who are both motivated by RRA and have the remit (and time!) to get things done. If possible, try to secure an organization sponsor (e.g. an RRA advocate from the organization’s leadership) to support the group and help to provide visibility for their work among the organization leadership. Develop a ‘Terms of Reference’ for the working group that outlines the responsibilities of working group members, including things like attending meetings, providing expertise for specific activities (such as a communication plan, feedback mechanisms), and helping to develop and implement the strategy itself. CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Convening a working group/ task force Assigning individuals to specific roles within the working group can be useful, particularly if they have relevant professional skills, such as coordinating with senior management, managing finances, creating communication plans, or overseeing monitoring and evaluation efforts. Group members will also need access to resources to deliver the work required – and as noted in activity 1, this is where senior leadership ‘buy-in’ can be especially helpful if, for example, funds or administrative capacity is required to support the activities needed. CONVENING A TASK FORCE Things to Think About How can you ensure that diverse perspectives (researchers at all career stages, administrators, and your broader community) are meaningfully included? Are there any RRA experts that could join or advise your group/ task force? What existing committees or working groups can you connect with to ensure alignment and long-term sustainability/handover? What arguments and evidence can you present to leaders to ensure they actively champion the task force’s work? What mechanisms (e.g. consultations, town halls, surveys) will you use to maintain transparency in the process? How will you ensure that members contribute meaningfully without being overburdened, balancing formal leadership with distributed ownership? ACTIVITY 4 23 Consider carefully how you select or recruit individuals who might serve on a working group and try to adhere to principles that are aligned with RRA such as operating a transparent application process. Equally, consider how your organization’s mission and its community can be represented in the membership of the working group (e.g. early career researchers; patients and publics; indigenous involvement). Remember that engagement in RRA across the organization might not sit only with those typically involved in RRA; individuals involved in initiatives focused on research culture and diversity, research integrity, and open science and scholarship are likely to be active supporters of the principles of RRA and important allies! Finally, as the group forms, consider if training related to RRA might be helpful, such as debiasing and unconscious bias or Equity, Diversity and Inclusion (EDI) monitoring. CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY RETHINKING RESEARCH ASSESSMENT UNINTENDED BIASES COGNITIVE SYSTEM & Hatch, A and R. Schmidt. (2020) Rethinking Research Assessment: Unintended Cognitive and System Biases. DORA.This is Open Access content distributed in accordance with the Creative Commons Attribution 4.0 International License Judgment and decision-making biases that impact how we weigh options and make choices have been shown to result in inequitable review, promotion, and hiring practices. While recognizing these biases at a personal level is important, creating new structural and institutional conditions to reduce bias can be even more valuable. Matthew effect Campbell’s law Anchoring Availability Halo effect Confirmation bias Status quo bias We let positive impressions of individual attributes influence our overall opinions Example: A candidate from a prestigious institution is thought to have more potential than one from a lesser known university Why it’s problematic: Giving preferential treatment to people based on inherited attributes may reinforce inequitable norms, which can lock out otherwise worthy candidates and fails to consider individuals equally. Resources often flow to those who already have them Examples: Highly cited references may be more cited in part because researchers see that they’re highly cited. Researchers with a long track record of grants receive a disproportionate amount of new funding. Why it’s problematic: When people lack the time or motivation to vet results, this can make access to resources even less equitable. Once metrics are accepted as a way to gauge value, they start to lose meaning as objective measures Example: Reward systems that rely on easily measurable qualities — like citations and publishing in high-JIF publications — can lead people to “game” the system. Why it’s problematic: When quantitative measures have an outsize impact on how people are rewarded, it can increase the temptation to focus on a narrow set of activities and reduce investment in other meaningful, but less rewarded, achievements. Anecdotal, top-of-mind, or easily recalled data can inadvertently skew what we prioritize Example: Prioritizing individual or memorable anecdotes, both pro or con, like getting a well known grant. Why it’s problematic: Overweighting information that more readily comes to mind may result in failing to consider other important evidence, missing the bigger picture. The first piece of data we see or hear tends to set the bar against which we judge subsequent pieces of information Example: Negatively comparing post-COVID-19 research productivity to pre-COVID-19, or using one’s own personal life as a gauge to judge others’ experiences Why it’s problematic: Initial anchor data defines the “normal” against which all other data is compared, which can skew our reference points by emphasizing relative comparisons between options rather than their actual value. People tend to dismiss evidence that doesn’t fit their initial judgments or preconceptions Example: Cherry-picking information from a CV to confirm the view one already has, or dismissing potential warning signs because a candidate has already been accepted as a good fit. Why it’s problematic: Our initial conceptions are often based on subjective experiences and limited data. Failing to gather and consider counter-evidence makes us more likely to fall into old ways of thinking. We often take the path of least resistance unless there are strong reasons not to Example: Continuing to use citations from academic journals as a primary indicator of impact or quality, rather than considering alternate quantitative indicators of real-world value. Why it’s problematic: People often stick with recognizably flawed processes because the effort to fix them or adopt new ones is perceived as too much effort. “Objective” comparisons are not necessarily equitable Qualities that can be measured or ranked are tempting because they feel less subjective, but can feed a false sense of precision. What can institutions do? • Balance the use of quantitative metrics with qualitative inputs, like narrative CVs, that capture more intangible qualities • Select standards based on a wide set of inputs rather than a narrow or anecdotal set • Recognize where setting specific, quantifiable goals may be reinforcing some behaviors at the expense of others Incumbent processes and perceptions have the advantage Many institutions have deep legacy traditions that become normalized over time, but these organizational habits can also keep new ideas and people out. What can institutions do? • Make the benefits of new behaviors concrete, salient, and easy to grasp • Recognize where old assumptions may overly reward those who are more traditionally successful, at the expense of new or more diverse talent • Set, publicize, and adhere to measurable goals that look beyond traditional norms of success when reviewing potential candidates to broaden the pool of individuals under consideration We gauge value by association Highly rated or prominent institutions and journals (and those associated with them) often get the benefit of the doubt based on familiarity or reputation rather than reality. What can institutions do? • Use structured interview protocols to keep decision-makers focused on agreed-upon qualities, rather than on reputation • Explicitly articulate and consider long-term and qualitative values, as well as short-term or easily quantifiable needs • Have applicants highlight and articulate their most meaningful contributions to reduce reviewer reliance on journal names or quantifiable characteristics of productivity It’s hard to weigh all information equally, which can give initial or “shiny” data points and personal reference points an advantage. What can institutions do? • Assemble diverse teams — across gender, seniority, cultures, and under-represented minoritized populations — to bring a range of perspectives and experiences into decisions • Look outside your institution or discipline to broaden a sense of “normal” • Put reputation-based indicators like education at the end of applicant materials to reduce preconceived notions Individual data points can accidentally distract from the whole Tackling these infrastructural and institutional implications of common biases can help promote and support more equitable practices: RETHINKING RESEARCH ASSESSMENT COMMITTEE COMPOSITION AND DELIBERATIVE PROCESSES DEBIASING It is generally recognized that more diverse decision-making panels make better decisions: including more perspectives reduces bias, increases transparency, and exposes more individuals to how decisions are made. But old habits die hard, and increasing the diversity of committees demands behavioral change. Here are some strategies that can help. Schmidt, R. (2022) Rethinking Research Assessment: Debiasing Committee Composition and Processes. DORA.This is Open Access content distributed in accordance with the Creative Commons Attribution 4.0 International License This work was sponsored by Arcadia — a charitable fund of Lisbet Rausing and Peter Baldwin. Debiasing deliberative processes can also reduce “business as usual” decision-making tendencies Reducing leadership bias • Conduct and document “pre-briefs.” Spending time upfront to collectively craft the “rules of the road” for committee work can create alignment and serve as a shared touchpoint that everyone — no matter what their role of seniority — can point to if things go awry. • Make all votes count. Seeing how others are voting can sway where we put our own chips. Techniques like anonymous voting can help reduce tendencies to conform to others’ views or confirm safe choices rather than express true preferences. Reducing individual bias • Question what we think we know. Asking committee members to explicitly step through their thought processes and assumptions can surface and counteract “confirmation bias,” or the tendency to prioritize data that reinforces existing preconceptions 5 . • Even the playing field. Consider strategies to reduce advantages of circumstance; providing interview questions in advance can equalize candidates, and using relative measures — such as progress from a starting point rather than judging absolute accomplishments — can gauge applicant quality more fairly. Increasing systems thinking • Identify bias at a system level. Efforts to reduce personal bias can put the burden on individuals to change, and can ignore how systems themselves are often designed to reinforce “hidden in plain sight” biases. • Think downstream. Improving diversity through hiring will fall flat without equal investment in mentorship and retention. • Use structure to provide consistency. Structured approaches — like interview protocols and pre-determined criteria — can increase confidence in comparison without resorting to solely quantitative measures. 1 Chang, EH, D Chugh, M Akinola, and KL Milkman (2019) Diversity Thresholds: How Social Norms, Visibility, And Scrutiny Relate To Group Composition Academy of Management Journal, Vol. 62, No. 1, 144–171. https://doi. org/10.5465/amj.2017.0440 2 Chang, EH, EL Kirgios, A Rai, and KL Milkman (2020) The Isolated Choice Effect and Its Implications for Gender Diversity in Organizations. Management Science: 1-10 3 He, JC, SK Kangb and N Lacetera (2021) Opt-out choice framing attenuates gender differences in the decision to compete in the laboratory and in the field. PNAS 118 (42) e2108337118 | https://doi.org/10.1073/ pnas.2108337118 4 Crain, M, M Cherry, and W Poster (2016) Invisible Labor: Hidden Work in the Contemporary World. University of California Press 5 Hernandez, I and JL Preston. Disfluency disrupts the confirmation bias, Journal of Experimental Social Psychology (2012), http:// dx.doi.org/10.1016/j.jesp.2012.08.010 Transparency invites trust When decisions about who’s included (and who’s not) are decided upon behind closed doors, even well-meaning intent can seem mysterious. In contrast, transparent and consistently applied criteria create a baseline and build a foundation of credibility. Taking a portfolio view Keeping the bigger picture in mind can protect against the common tendency to make individual decisions, each reasonable in isolation — the socalled “isolated choice effect 2 ” — that collectively reinforces familiar norms or standards of decision-making. Fostering true diversity of opinion Non-traditional participants may fear judgment or feel a need to check themselves when making suggestions that run counter to established or commonly held views. More inclusive processes deliberately create space to consider all viewpoints, with shared goals in mind. Expanding a sense of what’s possible Traditions and historical norms are sticky in part due to status quo bias, but can also persist due to a perceived lack of other alternatives. Gaining exposure to new options by seeing what others have done can help overcome “the way things are done around here.” Question the norms about who is qualified to participate or contribute When traditional or overly narrow forms of inclusion and exclusion – like seniority or rank – are used as criteria too early, they may leave out individuals who can provide important alternative points of view. Relying on self-identification or selection by leadership can reinforce existing biases Research shows that making selection opt-out rather than opt-in can help boost inclusion of those who less comfortable with selfpromotion 3 , or those who may not seem like “obvious” choices. Diversify across characteristics to support a range of perspectives While increasing racial and gender representation is critically important, people from other less-represented groups – like first-generation or early career academics, or those with crossdisciplinary experience – can also invite new and valuable perspectives. Overcome “two-kenism 1 ” tendencies Research indicates that committees stop seeking diversity after selecting two underrepresented individuals, feeling like they’ve “checked the box.” Making diverse representation less like a quota to be filled can also reduce the perception that those individuals must represent entire segments rather than their personal expertise. Connect committee composition to outcomes through representation of those who will be affected Deliberately inviting perspectives from those who will be on the receiving end of policy or directly impacted by decisions ensures that issues which might otherwise go unseen have the chance to be addressed. Broadening who is exposed to processes can promote equity of opportunity The ability to see behind the curtain may be especially useful for first-generation researchers or those new to the field. But recognize when committees become a form of added burden in the form of “invisible labor 4 ” for those already expected to pull more than their fair share. Read on how institutional and infrastructural implications of personal biases influence hiring, promotion, and tenure decisions24. Unintended System Biases RESOURCE Get inspired by these strategies for including more perspectives and reducing biases in the evaluation processes25. Debiasing Committee Composition RESOURCE ACTIVITY 4 Convening a working group/task force 24 There are an increasing number of global initiatives focused on driving RRA and aligning your proposals with these movements can strengthen the case for change in research assessment. Exploring affiliations with organizations like DORA and those set out in Chapter 4, shows commitment and intent towards RRA and can spark discussions with the leadership and other influential groups within your organization, such as early career researchers. Organizations sit within a broader system and researchers are influenced by policies from their organization, as well as those from their funders, collaborators’ institutions, and national or regional requirements. In shaping your own approach to RRA, explore how other organizations manage research assessment. It may be expedient to align your organization’s RRA approaches with those of peers or other influential organizations. For example, check the policies of local and national funders and academies, and/or whether any policies are subject to updates that might impact your organization. CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Exploring the landscape for RRA ACTIVITY 5 Discover 11 dimensions set by the Global Research Council to articulate funders’ vision of RRA and their plan to embed RRA principles26. Dimensions of RRA RESOURCE EXPLORING THE LANDSCAPE Things to Think About What (inter)national initiatives are your community already connected to – such as university associations and academies – that can support your RRA efforts? Are your national or regional evaluation systems moving towards more RRA practices? Are the policies of other similar RPOs changing, or those of your biggest collaborating organizations? Do funding and assessment agencies encourage a diversity of research outputs, have open research mandates, or are using narrative CVs in their processes? How can you influence current initiatives with the experience and expertise of your approaches? 25 CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Chapter 4 of this Guide provides examples of some of the most well-known organizations and initiatives active in the research assessment and related spaces. Over the last decade DORA has built a rich resource of case studies, toolkits and guidance documents to help organizations share innovations and learn from each other. In addition, Reformscape provides examples of RRA practices that are being used particularly for hiring, promotion, and tenure decisions across research performing organizations. Explore how academic institutions worldwide are reimagining their hiring, review, promotion and tenure processes4. Reformscape RESOURCE ACTIVITY 5 Exploring the landscape for responsible research assessment Explore these innovative stories highlighting key elements of institutional change to improve academic career assessment2. DORA Case Studies RESOURCE 32 Organizations need data and information to help them to reflect and evaluate their strategy’s effectiveness, identify successes, and address areas for improvement. RRA emphasizes regular reflection and review. A solid RRA strategy should ideally include mechanisms for reflection, use evidence to guide decisions, and be transparent about its impact, building credibility in research assessment. Gathering information on the uptake and impact of a strategy can provide useful insights. This might range from tracking participation in the strategy’s launch, such as attendance at events, and collecting feedback from applicants. Monitoring key aspects of recruitment and promotion, like applicant numbers, demographics, or outcomes, can reveal areas of change. Comparing data from before and after implementation, as well as across departments or organizations, can also help you to identify the strategy’s key impacts and foster broader comparisons as RRA practices become more common. This can also help your RPO to identify any unintended consequences and take steps to mitigate these, helping to improve outcomes as well as to minimize resistance to change. Active communication and engagement about the impacts of implementing RRA practices – and with researchers from across career stages and across the organization - can help to build trust in the changes and build further buy-in for change across the organization (activity 8). CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Monitoring and updating your strategy ACTIVITY 9 MONITORING YOUR STRATEGY Things to Think About How will you use your communication and dissemination plans to keep an open iterative change process with your communities? What indicators can you use to track progress and your definition of success? Who will own addressing feedback and updating the strategy? How will you adapt to new (inter) national developments in RRA? What will you do to keep leadership, researchers, and staff engaged with the strategy? 33 CHAPTER 2: KEY ACTIVITIES IN DEVELOPING AN RRA STRATEGY Tools and approaches to capture relevant data and feedback to support monitoring and evaluation include embedding information in your organization research management system, creating simple feedback forms and surveys linked to specific practices designed to deliver RRA (e.g. recruitment practices, use of narrative CVs), and perhaps through the establishment of a dedicated website and email address or feedback box. There are several example frameworks that provide useful templates for RPOs seeking to shape reflexive approaches (see DORA SPACE Rubric11 and the SCOPE framework for research evaluation34 created by INORMS). ACTIVITY 9 Monitoring and updating your strategy Gauge the current state of infrastructural conditions to support reform initiatives and interventions using the SPACE rubric11. SPACE Rubric RESOURCE Developed for research assessment, the reflective approach of SCOPE can also help your reflection of the evaluation of your RRA strategy34. SCOPE Framework RESOURCE FROM FOUNDATION... Core definitions and shared clarity of purpose TO EXPANSION... Increased traction and capability development TO SCALING Accelerated uptake and continuous improvement CULTURE WITHIN INSTITUTIONS How are assessment practices perceived and adopted both within and outside of formal evaluation activities? ACCOUNTABILITY How are individuals and institutions held liable for executing on new assessment practices? EVALUATIVE AND ITERATIVE FEEDBACK How are intervention outcomes and progress toward institutional values captured and continually improved upon? STANDARDS FOR SCHOLARSHIP How are new definitions of “quality scholarship” formulated and applied? PROCESS MECHANICS AND POLICIES How are new practices incorporated into review structures, processes, and institutional policies? THIS MIGHT LOOK LIKE... More diverse types of individuals are involved in both defining and participating in career advancement processes, such as including early career researchers on RPT committees Representation of minoritized applicants meets or exceeds equity goals for both new hires and researcher retention Career growth and mentoring systems are intentionally designed to provide ongoing support for underreprsented hires THIS MIGHT LOOK LIKE... Adoption of new assessment mechanisms is supported and advocated for by departmental and institutional leaders All individuals actively contribute to building more equitable practices — not just minoritized ones New research assessment norms are increasingly adopted as a default by faculty, administrators, and applicants THIS MIGHT LOOK LIKE... “Positive friction,” or intentional pause points to reflect on assessment practices and slow down business-as-usual processes is incorporated into both formal and informal assessment practices All participants in assessment activities feel processes achieve a balance of effectiveness and efficiency THIS MIGHT LOOK LIKE... Goals and success criteria for individual academic assessment interventions are well-defined and shared Use of leading indicators (e.g. increased diversity of inquiries for open positions) supplements lagging indicators (e.g. increased diversity of hires) when gauging intervention efficacy Goals and success criteria are automatically reviewed whenever institutional strategy is updated THIS MIGHT LOOK LIKE... Quantitative and qualitative data from interventions are captured in a standardized way Mechanisms that capture both quantitative and qualitative feedback are explicitly designed and embedded into assessment processes from the outset Best practices and examples of measurement and/or gathering feedback are codified and shared across disciplines within the institution THIS MIGHT LOOK LIKE... Interventions that don’t achieve desired outcomes are considered learning opportunities, not failures Outcomes and data are collected and monitored to ensure high standards of evaluation quality and identify unintended consequences or adverse effects Feedback and other indicators are refined and/or examined in aggregate to identify and investigate patterns or opportunities for course-correction THIS MIGHT LOOK LIKE... Standards are explicitly designed and articulated to align with institutional mission and values, such as increasing equity and support for traditionally underrepresented, minoritized groups New standards for scholarship consider the balance across research, teaching, and service contributions including training, mentoring and good citizenship Specific definitions and standards of “quality” with regard to scholarship are articulated and shared across disciplines and review/promotion committees THIS MIGHT LOOK LIKE... Scholarship is assessed using diverse indicators (e.g. societal impact), units of assessment (e.g. full body of work v. individual articles), and forms of output (e.g. non-journal contributions) Indicators of quality recognize non-individualized activities and accomplishments like team science New definitions of “scholarship” are deployed across the full range of institutional disciplines THIS MIGHT LOOK LIKE... Faculty have the ability to customize success measures to reflect their research interests and goals New standards, definitions, and criteria for evaluating the quality and impact of scholarship are integrated into the language and processes of new assessment practices Hatch, A and R. Schmidt. (2021) Rethinking Research Assessment: SPACE to Evolve Academic Assessment. DORA.This is Open Access content distributed in accordance with the Creative Commons Attribution 4.0 International License Research and researcher assessment is a systems challenge, suggesting that institutions that prioritize developing infrastructures to support their efforts may be better positioned to achieve their goals than those focused only on individual solutions. S.P.A.C.E. TO EVOLVE ACADEMIC ASSESSMENT A RUBRIC FOR ANALYZING INSTITUTIONAL PROGRESS INDICATORS AND CONDITIONS FOR SUCCESS RETHINKING RESEARCH ASSESSMENT THIS MIGHT LOOK LIKE... The goals, principles, and practices of academic assessment and review, promotion, and tenure (RPT) activities are transparent and clearly articulated, and agreed upon by all participants Institutions have clearly defined expectations for adherence to academic assessment practices Examples of “what good looks like” are collected and shared to more concretely illustrate target outcomes and behaviors THIS MIGHT LOOK LIKE... Research evaluators self-monitor adherence to academic assessment principles and practices Senior leaders and committee members actively stipulate equitable assessment practices during both formal and informal career development contexts Institutions model ecosystem-level accountability, such as ensuring that system-level incentives align with and support agreed-upon principles and practices THIS MIGHT LOOK LIKE... Individuals actively contribute to the development and review of new practices and principles Departments proactively broaden and conduct outreach activities to include new or minoitized applicants Faculty serve as “ambassadors” for new academic assessment practices, such as when serving as external committee members THIS MIGHT LOOK LIKE... Meaningful and appropriately rigorous qualitative structures for academic assessment, such as narrative CVs, are given due weight Structures and processes are applied consistently across assessment activities, taking into consideration alternate paths and starting points Use of new assessment mechanics extend beyond traditional evaluative contexts into ensuring equitable opportunities, mentoring, and retention to increase research and researcher diversity THIS MIGHT LOOK LIKE... Training on the goals and procedures of assessment processes and practices are accessible and continually maintained Institutions design processes take into account the resource capacity of committee members to effectively adopt new assessment practices, such as additional burdens on time Institutions have designated senior functions or offices to ensure faculty capacity for new assessment practices and principles THIS MIGHT LOOK LIKE... Assessment mechanics can be flexibly applied and adapted to accommodate diverse disciplines Mechanisms to support practices are codified and written into institutional policies New processes and practices are seamlessly integrated and widely adopted TRANSPARENCY AND CLARITY OF GOALS ADHERENCE THROUGH COMMITMENT PROACTIVITY IN ENGAGEMENT DEBIASING DELIBERATIVE JUDGMENTS CAPACITY TO SUPPORT NEW ACTIVITIES INTEGRATION INTO EXISTING SYSTEMS INCLUSION AND ACCESS ADVOCACY AT INSTITUTIONAL LEVELS REFLEXIVITY THROUGH REFLECTION ARTICULATION OF DIVERSE INDICATORS SYSTEMATIZATION TO GAIN CONSISTENCY IMPROVEMENT USING FEEDBACK LOOPS ADOPTION OF NEW PRACTICESDIVERSIFICATION OF STANDARDS ALIGNMENT ON VALUES AND GOALS A one-page overview of the five-stage SCOPE Framework SC OP E START WITH WHAT YOU VALUE CONTEXT CONSIDERATIONS PROBE DEEPLY OPTIONS FOR EVALUATING EVALUATE YOUR EVALUATION The SCOPE Principles The five stages of SCOPE operate under three main principles: 1. Evaluate only where necessary. Evaluation is not always the right strategy. When it comes to incentivising behaviours, for example, it may be more fruitful to enable them than to evaluate them. 2. Evaluate with the evaluated. Any evaluation should be co-designed and cointerpreted by the communities being evaluated. 3. Draw on evaluation expertise. We should apply the same rigour to our evaluations that we apply to our academic research. START with what you value •  Clearly articulate what you value about the entity being evaluated •  Not with what others’ value (external drivers) •  Not with available data sources (the ‘ Streetlight Effect’) CONTEXT considerations •  Ensure your evaluation is context-specific •  WHO are you evaluating? (Entity size and discipline) • WHY are you evaluating? OPTIONS for evaluating •  Consider both quantitative and qualitative options •  Be careful when using quantities to indicate qualities PROBE deeply •  WHO might your evaluation approach discriminate against? •  HOW might your evaluation approach be gamed? •  WHAT might the unintended consequences be? •  CONSIDER the cost-benefit of the evaluation EVALUATE your evaluation •  Did your evaluation achieve its aims? •  Was it formative as well as summative? •  Use SCOPE to evaluate your evaluation. Design and publishing partner: Emerald publishing. inorms.net/research-evaluation-group 34 35 A PRACTICAL GUIDE TO IMPLEMENTING RRA AT RPOs Key moments in research and researcher assessment This Chapter provides tips on how organizations can embed RRA practices into policies and practices that impact upon researchers at crucial points in their careers. Specifically during recruitment and promotion decisions, when providing awards and allocating grants, when evaluating research units, and in corporate communications. 3.1 Recruitment, hiring, and promotion decisions 3.2 Institutional awards and internal grants 3.3 Evaluating internal research units 3.4 Developing corporate communications 35 3 36CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT Narrative CVs Narrative CVs are an alternative to traditional academic CVs that allow researchers to present their achievements, skills, and contributions through a descriptive, contextualized format. Since the introduction of narrative CVs as a tool to support RRA, several studies have assessed the difference that their use can make. While the evidence is variable thus far, there is a consensus emerging that narrative CVs can reduce bias in researcher assessment processes while shining a light on sub-optimal practices37. The adoption of narrative CVs aims to promote diversity, equity, and inclusion within the research community by recognizing varied experiences and reducing biases inherent in conventional assessment methods and allows researchers to ‘tell their story’. They allow candidates to describe their key contributions in research, and why they are important; moreover, researchers can provide evidence of their specific role to each research output, describe their competencies, their mobility experiences and their career paths, providing a richer, more qualitative understanding of a researcher's profile and impact. Organizations such as the UK funder UK Research and Innovation (UKRI) have implemented narrative CVs formats35 (see UKRI’s Resume for Research Information) to support a healthier research culture and to align with RRA principles. In addition, narrative CVs encourage applicants to reflect on their personal and professional growth, fostering a more holistic approach to career development36. IMPACT RATIONALE SPOTLIGHT ON 36 37 Instead of focusing solely on metrics like publications and grants, narrative CVs typically enable researchers to provide information across four areas: This approach and many other diverse approaches provide a more holistic view of a researcher’s career by emphasizing the quality and significance of their work rather than just quantitative outputs. Narrative CVs are increasingly used in research funding and hiring processes to promote fairness, diversity, and a more comprehensive assessment of academic contributions. Contributions to knowledge The generation of new ideas, tools, and methodologies Broader societal impact Contributions to broader research/ innovation users and audiences, and towards wider societal benefit Wider research impact Contributions to the wider research and innovation community Collaborations, teamwork and developing others Contributions to team working and the development of others CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT SPOTLIGHT ON NARRATIVE CVs Tip sheet on how to optimize, evaluate and iterate on the use of narrative CVs for funding decisions38. How to Use Narrative CVs RESOURCE 37 38 There are a range of interventions that institutions can implement to align with RRA throughout recruitment, promotion and/or tenure. Recognizing the diverse contributions and talents of researchers, prioritizing qualitative evaluation, moving away from inappropriate metric use, and ensuring a fair and transparent process in hiring, review, promotion and/ or tenure are the cornerstones of RRA. Objectives include to foster diverse career paths, ensuring transparency in assessment processes, and actively involving the academic community in these reforms, as discussed in Chapter 2. Table 1 presents possible approaches that organizations can take to ensure RRA at key moments in recruitment, hiring and promotion processes. CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT Recruitment, hiring, and promotion decisions SECTION 3.1 A guide for funders of research seeking to promote a more holistic approach to the evaluation of funding proposals40. A Practical Guide for Evaluators RESOURCE Find out how Open University created a new promotion track for researchers doing publicly engaged work39. Open University CASE STUDY The DORA team, in partnership with the Luxembourg National Research Fund (FNR), has created a video that presents a helpful overview for research evaluators involved in recruitment and promotion. 39 KEY MOMENT SUGGESTED INTERVENTION TO SUPPORT RRA CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT Creating a hiring or promotion committee Creating application material and criteria Creating and posting job adverts When creating applicant forms and associated instructions and/or notices: • Ensure they are transparent and easily discoverable by potential candidates. • Provide FAQs and/or a contact point so that individuals can pose questions that can help them optimize their application. • Ensure that the criteria is directly relevant to the role and/or tenureship and do not request information that is not aligned with RRA principles (and your institutional RRA strategy) or that would inadvertently censor out potentially qualified applicants. • Structure them to allow applicants to present a ‘balanced and broad’ spectrum of their competences and achievements. • Consider requiring a narrative CV and/or include questions about areas of impact that align with your institutional goals, such as public or patient engagement and knowledge mobilization (see Spotlight on Narrative CVs on page 36). When creating job advertisements or promotion application notices: • Ensure that the applicant eligibility requirements do not inadvertently censor out potentially qualified applicants. • Ensure job adverts clearly convey the organization’s mission and commitment to responsible research(er) assessment. • Emphasize the importance placed in recognizing a broad range of research outputs, career experiences and competences, including societal contributions, teamwork, and interdisciplinarity. • Publish your job advertisement across a range of platforms that enable you to reach a diversity of individuals. • Where possible, provide options for applicants to ask questions and receive support where appropriate. When putting together your hiring/promotion committee or panel, ensure that members are: • Equipped with the appropriate tools and information to enable them to practice RRA (e.g. aware of any institutional RRA strategy and associated assessment criteria). • Trained in making decisions that avoid bias and are RRA aligned (e.g. unconscious bias or debiasing training). • As representative and inclusive of the academic community as possible, to promote an inclusive approach to decision-making. TABLE 1 Assuring RRA in recruitment and promotion processes Approaches organizations can take to ensure RRA at key moments in recruitment, hiring and promotion processes. 40 KEY MOMENT SUGGESTED INTERVENTION TO SUPPORT RRA CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT Triaging applications Onboarding and researcher development in new positions Evaluating hiring and promotion process Interviewing candidates When interviewing candidates, interview committees can: • Use a standardized set of interview protocols and questions that are given to all applicants to avoid the potential for bias. • Offer candidates advanced sight of question areas to be covered, allowing time for reflective responses and reduce language bias. • In advance or at the start of the interview, inform candidates about the interview process and structure, and how they will be assessed. When welcoming candidates to new positions, consider: • Providing post-hire support, with individual development plans, regular meetings and discussion, and periodic evaluation to identify areas for support to ensure academic success. • Ensuring that new staff or promotion information is captured properly in research management and information systems, and that the information that is captured is aligned with RRA. • Providing training, especially for heads of departments and team leads, to share the organization’s approach to RRA. • Gathering regular staff feedback on working practices and research assessment practices on the ground. Try to build in ways for you to reflect on process from the candidate and recruiter perspective. For example: • Introduce a post-recruitment process feedback form, for candidates/ applicants and recruitment team members. • Aim to keep feedback forms: ·simple ·anonymous where possible – include contextual information that can be used in the aggregate to ensure you are aware of trends according to specific characteristics (e.g. gender, career age) ·applicable to different departments and faculty to enable comparisons and benchmarks When triaging applications: • Create a standard set of criteria upon which to assess applicants. • Train the staff performing the triage to follow the criteria and avoid any deviation that might introduce bias into the selection process (especially if triaging is not done by a member of the committee). • If you are capturing applicant data for monitoring purposes - such as tracking diversity or recruitment trends - ensure the data is anonymized or pseudonymized where possible. • Where possible, provide a route for applicants to receive feedback, including those not selected, especially after interview. TABLE 1 (continued) Assuring RRA in recruitment and promotion processes Approaches organizations can take to ensure RRA at key moments in recruitment, hiring and promotion processes. 41 Ensuring fairness and reducing bias in awarding grants, prizes, and funding is a key part of responsible research assessment. Internal awards and funding competitions may be a relatively easy place for institutions to consider piloting and experimenting with RRA changes. Awards recognizing broader impacts of research do not need to cost the institution money and internal funding calls that already exist can be reframed to align with RRA best practices. RPOs also receive funding from agencies who have developed their own standards and policies, for example, requiring open access publishing and full data sharing. To support their researchers, RPOs must stay up-to-date on these policies and work towards aligning their own policies with broader research system requirements. Many RPOs have also developed approaches to allow some internal flexibility to support researchers who may not be in receipt of dedicated external funding. For example, through the creation of organization funding reserves or pooled funding pots, monies can be shared across research teams to support specific activities and to align with an RRA strategy. These combined efforts help RPOs navigate funding constraints while fostering a research culture that values integrity, inclusivity, and academic freedom. Through its engagement with research funding agencies, the DORA team has produced a guide that can help organizations keep funding calls simple and effective41. Table 2 presents possible approaches that organizations can take to ensure RRA at key moments in making institutional and grant awards. Institutional awards and internal grants SECTION 3.2 CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT 48 CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT Early Career Researchers Graduate programs are often highly competitive with students competing for a limited number of places. Students are typically formally assessed by the institution as part of their degree progression, commonly through a faculty committee assessing their final thesis or dissertation. While of course individual thesis assessments do need to be nuanced, there are many things that institutions can do to make the assessment process align with the principles of RRA. There is an abundance of evidence that shows the desire among early career researchers (ECRs) for research assessment approaches that value a diversity of outputs and contributions42,43. However, if organizations do not adhere to progressive RRA practices, ECRs can find themselves sitting within a traditional paradigm that creates a ‘pressure to publish’44, which can be at the expense of broader research learnings and experience, and lead to mental health challenges45. There are many initiatives that are now working to fully involve ECRs in shaping their approach to research assessment to ensure that it best serves their interests46. ENSURING RRA AS PART OF THE GRADUATE STUDENT PROGRAM SPOTLIGHT ON 48 49 Consider whether your graduate programs include training on responsible research assessment. Useful topics for training that encourage RRA might include: Do you require students to publish papers to graduate? Avoid a focus on quantity over quality. Consider focusing on good scientific governance, for example by encouraging graduate students to make their research outputs and data discoverable and easily reusable, such as by publishing a preprint and/or using data repositories. Is your organization consistent in how it assesses theses? Do you apply standard criteria? CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT Learn more about how the University Medical Centre in Utrecht redesigned its evaluation of PhD candidates47. University Medical Centre CASE STUDY SPOTLIGHT ON EARLY CAREER RESEARCHERS Research culture How to be a constructive and fair peer reviewer Research integrity and ethics How to design research for robustness and transparency, and how to deal with negative results Responsible use of metrics How best to use quantitative and qualitative indicators in research assessment and some of the dangers of different indicators and approaches Sharing your research with impact Open science, research dissemination and publication practices Narrative CVs How to contextualize and evidence their contributions in a narrative format 49 EARLY CAREER RESEARCHERS: Things to think about 50 Effective communication of research quality and its broad impacts helps to ensure that RRA becomes part of an institution’s internal and external narrative and identity. It is important to ensure that colleagues who are responsible for communication across the RPO are aware and on board with the focus and aims of the RRA strategy and are positioned to share the stories and successes of researchers with a focus on a broader range of contributions. Table 4 presents possible approaches that organizations can take to ensure RRA practices when shaping corporate internal and external communications. Developing corporate communications SECTION 3.4 CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT 51 KEY MOMENT SUGGESTED INTERVENTION TO SUPPORT RRA CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT Developing institution policies and position statements Updates and internal newsletters from institutional leadership Announcing key appointments and new hires In regular communications to the community, organizations can: • Celebrate examples of the wider impact of the research and researchers, rather than just highlighting papers published in specific journals. • Focus on diverse pathways to impact including, but not limited to, societal engagement, commercialization, science for policy, science communication. • Recognize the different competencies and activities that are employed to achieve high quality and impactful research, including research robustness, reproducibility, integrity, ethics, and openness. When putting out communications around new appointments: • Focus on the broad skills, experience and future potential that a new appointment can bring and avoid focusing solely on narrow definitions of academic success (e.g. publication output). • Reinforce increased scope and flexibility of research quality and competence assessment in recruiting procedures. When communicating new policies or updates, organizations can: • Ensure existing and new policies and position statements (e.g. around EDI, hiring and promotion, the use of rankings, the approach to open science) align with the RRA strategy and are transparent to all interested parties. • Indicate values alignment with DORA, CoARA, and other institutional commitments, increasing community accountability. Possible approaches that organizations can take to ensure RRA practices when shaping corporate internal and external communications. TABLE 4 Assuring RRA practices in organization communication 52 KEY MOMENT SUGGESTED INTERVENTION TO SUPPORT RRA CHAPTER 3: KEY MOMENTS IN RESEARCH AND RESEARCHER ASSESSMENT Senior leadership engagement with community Commentary around national and/or institutional rankings Encourage that communication - both internal and external - around the institution’s position within national and international rankings: • Focuses on presenting a holistic view of an institution’s strengths and future development. • Avoids focusing on narrow definitions of research quality and impact, and only on successes. • Clarifies the data from which rankings are derived and what they represent. • Affirms that the criteria in these rankings do not trickle down to the assessment of teams or individual researchers. • Explains the limitations of the rankings. • Aligns with the More Than Our Rank initiative that encourages responsible use of institutional rankings (Chapter 4). Organizations can foster a culture of co-creation on RRA by: • Fostering dialogue around RRA initiatives as they are being developed and once an approach is agreed. • Organizing open town halls and seminars to facilitate internal discussions as well as other forms of wide intra-institutional information exchange. • Using various communication channels to engage all units, disciplines and career stages, including PhDs and professional staff. Possible approaches that organizations can take to ensure RRA practices when shaping corporate internal and external communications. TABLE 4 (continued) Assuring RRA practices in organization communication 53 A PRACTICAL GUIDE TO IMPLEMENTING RRA AT RPOs 53 Global initiatives and exemplars relevant to RRA There are a range of initiatives that offer frameworks and resources that can help inform responsible research assessment (RRA) practices and related topics. 4.1 Initiatives aiming to provide leadership and guidance on aspects of open science 4.3 Initiatives that aim to encourage a holistic and inclusive view of research 4.2 Initiatives that focus on driving research assessment reform 4.4 Initiatives that focus on enabling evidence-based policy and research integrity 4 Being knowledgeable about and integrating related topics such as open science, societal impact, research integrity, community engagement, interdisciplinarity, and reproducibility – along with others relevant to your context – could prove beneficial given there may be many ongoing efforts to support change in your organization. This Chapter includes wellknown examples; while the list of initiatives is non-exhaustive, we hope that it provides a helpful starting point. 54 The UNESCO Recommendation on Open Science16 was adopted by the 41st session of UNESCO General Conference in November 2021. The Recommendation provides an international framework for open science policy and practice that recognizes disciplinary and regional differences in open science perspectives. The UNESCO Recommendation is thought to help create the conditions for a holistic view of research and all its impact within and across countries. A set of principles has been designed to help countries to make open science a reality within their context. The principles closely align with those of RRA. • Transparency, scrutiny, critique and reproducibility - to reinforce the rigor of scientific results, enhance the positive impact of science on society and increase society’s ability to solve complex interconnected problems. • Equality of opportunities - to ensure that all scientists and those with an interest in science have equal opportunity to access, contribute to and benefit from science, regardless of origin or circumstance. • Responsibility, respect and accountability - to be responsible for and aware of public accountability, potential conflicts of interest, intellectual integrity and the possible social or ecological consequences of research activities. • Collaboration, participation and inclusion - to ensure that scientific collaborations transcend the boundaries of geography, language and resources, and include knowledge from marginalized communities to solve problems of great social importance. • Flexibility - to acknowledge that there is no one-size-fits-all way to practice open science and to encourage different pathways to practicing it while upholding the core values. • Sustainability - to be as efficient and impactful as possible by building on long-term practices, services, infrastructures and funding models to ensure participation of scientists from less-privileged countries or institutions. UNESCO Recommendation on Open Science CHAPTER 4: GLOBAL INITIATIVES AND EXEMPLARS RELEVANT TO RRA Initiatives aiming to provide leadership and guidance on aspects of open science SECTION 4.1 BACKGROUND CORE PRINCIPLES IMPACT 55 Founded in 2022, the Higher Education Leadership Initiative for Open Scholarship48 (HELIOS Open) was established to advance the adoption of open scholarship practices in higher education and is an initiative of the Open Research Funders Group49. Despite its transformative potential, the adoption of open practices can face barriers including cultural resistance, lack of funding for the needed infrastructures, and misaligned incentives. HELIOS Open was created to mobilize leadership within universities to champion the potential benefits of open scholarly and help to drive systemic change. Launched in 2023, the Barcelona Declaration on Open Research Information50 (Barcelona Declaration) aims to make ‘openness’ of information about the conduct and communication of research the norm. HELIOS Open has set up a network comprising US institution leaders and an Open Program Plan. Through this network and through a range of other discussion fora with the research community, HELIOS Open hopes to drive open scholarship. Signatories include a wide range of RPOs, funders and research-related organizations. The focus on open scholarship is closely aligned with RRA. HELIOS Open’s aim is to promote a more transparent, inclusive, equitable, and trustworthy research ecosystem. Higher Education Leadership Initiative for Open Scholarship (HELIOS OPEN) Barcelona Declaration on Open Research Information CHAPTER 4: GLOBAL INITIATIVES AND EXEMPLARS RELEVANT TO RRA SECTION 4.1 Initiatives aiming to provide leadership and guidance on aspects of open science CORE PRINCIPLES CORE PRINCIPLES BACKGROUND BACKGROUND IMPACT IMPACT Signatories to the declaration are required to sign up to four principles. • Transparency – Research information should be openly available and accessible, ensuring clarity in data collection, processing, and dissemination. • Equity – All researchers, institutions, and regions should have equal access to research information, preventing monopolization or restrictive access. • Sustainability – Open research information infrastructures should be built and maintained in ways that ensure long-term viability and community governance. • Openness – Research information should be freely shared and reusable to maximize its impact benefit for society. 56 Established in 2013, the San Francisco Declaration on Research Assessment1 (DORA) is a global initiative that advocates for improving research assessment practices, particularly by reducing reliance on journal impact factors and promoting qualitative, context-driven evaluation criteria. Founded in 2019, Foro Latinoamericano sobre Evaluación Científica51 (FOLEC-CLACSO) aims to promote activity and action to strengthen and develop research assessment processes in the Latin American region. Over 26,000 researchers and institutions have signed DORA, signaling a strong global commitment to responsible research assessment. More recently, DORA has focused on supporting active implementation of RRA by research institutions, funders and others. It has collated a significant volume of case studies3, tools frameworks, and guidance documents to support organizations looking to shift their practices towards RRA. FOLEC has been taking a lead across the Latin American region to build networks across national science and technology agencies and other RPOs to advance enhancements of academic evaluation systems and research culture. DORA provides 18 recommendations aimed at academic institutions, funders, and publishers, encouraging them to prioritize research quality and impact over quantitative metrics. DORA also emphasizes the need for transparent assessment criteria and broader recognition of research contributions. From an open, collaborative and public knowledge perspective, FOLEC seeks to strengthen democratizing and sustainable approaches and models of science, committed to societal problems. Declaration on Research Assessment (DORA) Latin American Forum on Research Assessment (FOLEC-CLACSO) CHAPTER 4: GLOBAL INITIATIVES AND EXEMPLARS RELEVANT TO RRA Initiatives that focus on driving research assessment reform SECTION 4.2 BACKGROUND BACKGROUND CORE PRINCIPLES CORE PRINCIPLES IMPACT IMPACT 57 The ‘More Than Our Rank’52 (MTOR) initiative was launched in 2022 by the International Network of Research Management Societies (INORMS) to address the over-emphasis placed on university rankings when evaluating higher education institutions. MTOR seeks to shift focus away from rankings (which are often derived from a narrow set of metrics) to the broader values and impacts that universities have on their communities, students, and global challenges. It advocates for a more holistic appreciation of what makes universities meaningful and effective. Make Data Count53 is an initiative that promotes the development of open data metrics to enable evaluation of data usage. Through the development of tools, strategic partnerships, and community engagement, Make Data Count advocates for the recognition of ‘data’ as an important output in research assessment. MTOR provides an opportunity for institutions to publicly highlight the unique strengths and societal contributions of their research that is not able to be recognized through international ranking alone. MTOR also emphasizes institutional work in research culture and how it aligns with local and global societal needs. MTOR signatories are not required to exit from traditional global research rankings but supplement those rankings with qualitative assessment related to their strengths and contributions. Make Data Count provides guidelines and tools to facilitate the integration of data metrics into data repositories and platforms. Several well-known data repositories, publishers, and research institutions are adopting Make Data Count metrics to track and showcase the impact of research data, including Dryad54, Figshare55, and Zenodo56. MTOR advocates for research institutions to be evaluated in broader and less biased ways, which recognize and encourage equity, diversity and fairness. MTOR is an initiative for all institutions that recognise the limitations of the indicators used by the rankings and for those who feel that the rankings do not reflect all of their strengths or institutional mission. More Than Our Rank (MTOR) Make Data Count CHAPTER 4: GLOBAL INITIATIVES AND EXEMPLARS RELEVANT TO RRA SECTION 4.2 Initiatives that focus on driving research assessment reform CORE PRINCIPLES CORE PRINCIPLES BACKGROUND BACKGROUND IMPACT IMPACT Make Data Count’s vision is that Research data are valued research outputs across scholarly activities, evaluation, and communications. 64A PRACTICAL GUIDE TO IMPLEMENTING RRA AT RPOs Resources and references 50. Barcelona Declaration on Open Research Information. https://barcelona-declaration.org/ 51. Latin American Forum on Research Assessment (FOLEC-CLACSO). https://www.clacso.org/en/folec/ 52. INORMS. (2022). More Than Our Rank. https://inorms.net/more-than-our-rank/ 53. Make Data Count. (2024). An Essential Infrastructure for Open Science. https://makedatacount.org/ 54. Dryad. Publish and preserve your data. https://datadryad.org/ 55. Figshare. Credit for all your research. https://figshare.com/ 56. Zenodo. https://zenodo.org/ 57. Coalition for Advancing Research Assessment (CoARA). https://coara.eu/ 58. CoARA. Sign the Agreement. https://coara.eu/sign/ 59. CoARA. The Commitments. https://coara.eu/agreement/the-commitments/ 60. CRediT. Contributor Role Taxonomy. https://credit.niso.org/ 61. EDI Caucus. EDICa. https://edicaucus.ac.uk/ 62. World Conferences on Research Integrity. (2025). Hong Kong Principles. https://www.wcrif.org/guidance/hong-kong-principles 63. World Conferences of Research Integrity. (2025). Form for individuals. https://www.wcrif.org/form-for-individuals 64. World Conferences of Research Integrity. (2025) Form for institutions. https://www.wcrif.org/form-for-institutions 65. Pew Charitable Trusts. The Transforming Evidence Funders Network. https://www.pewtrusts.org/en/research-and-analysis/fact-sheets/2022/04/the-transforming-evidencefunders-network All online sources were accessed in May 2025. 65A PRACTICAL GUIDE TO IMPLEMENTING RRA AT RPOs We acknowledge and thank the helpful contributions of participants at Project TARA workshop hosted by the University of Maryland on January 31, 2025, listed alphabetically below, as well as explicit feedback from DORA Steering Committee Members Sean Sapcariu and Bernd Pulverer. We are grateful to Janet Catterall for her help in compiling and organizing the references and supporting resources mentioned in the Guide. Liz Allen Jordana Armstrong Elyse Aurbach Stirling Bryan Kelly Cobey Erica Conte Katie Corker Alexandra D'Arcy Priya Doshi Michael Dougherty Abigael Eshete Kate Geddie Christian Gonzalez-Billault Helga Hallgrimsdottir Anna Hatch Stephen Jacobs Zubin Jelveh Patricio Korzeniewicz DORA University of Maryland Michigan State University Michael Smith Health Research British Columbia DORA, University of Ottawa Heart Institute Unity Health Toronto ASAPbio University of Victoria American University University of Maryland Association of Public & Land-Grant Universities Canadian Institute for Advanced Research Universidad de Chile University of Victoria HHMI Open @RIT University of Maryland University of Maryland Acknowledgments ANNEX NAME AFFILIATION 66 Rebecca Lawrence Giovanna Lima Andres De Los Reyes Lisa Maillard Flora Massah Caleb McKinney David Mellor Emily Miller Ben Miyamoto David Moher Kacy Redd Sarah Rovito Ruth Schmidt Michael Skovdal Rathleff Jordan S. Sly Brian Tarrallo Stephanie Warner Kent Weigel Baron Wolf DORA, F1000 Research Ltd DORA University of Maryland The Pew Charitable Trusts Illinois Institute of Technology Georgetown University Medical Center Center for Open Science Association of American Universities The Pew Charitable Trusts Ottawa Hospital Research Institute, Ottawa, Canada Association of Public & Land-Grant Universities National Academies Illinois Institute of Technology Aalborg University, Denmark University of Maryland Graphic Facilitator University of Calgary University of Wisconsin-Madison University of Kentucky NAME AFFILIATION ANNEX Acknowledgments A PRACTICAL GUIDE TO IMPLEMENTING RRA AT RPOs 67 A PRACTICAL GUIDE TO IMPLEMENTING RRA This Practical Guide is intended to offer practical guidance and tips for shaping and delivering responsible research assessment (RRA) practices at research performing organizations (RPOs). We are grateful to the Arcadia Foundation for its financial support for the Project Tools to Advance Research Assessment (TARA), which this Guide is part of. Arcadia helps people to record cultural heritage, to conserve and restore nature, and to promote open access to knowledge. Since 2002 Arcadia has awarded more than $1.2 billion to organizations around the world. This publication is licensed under the Creative Commons Attribution-Noncommercial CC BY-NC. Version 1.0 May 2025 Allen, L., Barbour, V., Cobey, K., Faulkes, Z., Hazlett, H., Lawrence, R., Lima, G., Massah, F., & Schmidt, R. (2025). A Practical Guide to Implementing Responsible Research Assessment at Research Performing Organizations. Declaration on Research Assessment (DORA). https://doi.org/10.5281/zenodo.15000683 Authors are ordered alphabetically by surname. GRANT ACKNOWLEDGMENT LICENSING CITE AS