Principles for Stakeholder Involvement in Evaluation in Latin America

1 Principles for Stakeholder Involvement in Evaluation in Latin America Julia Espinosa-Fajardo - University of Seville (Spain), [email protected] (corresponding autor) Pablo Rodríguez-Bilella - CONICET - National Research Council of Science and Technology (Argentina), [email protected] Esteban Tapella, National University of San Juan, San Juan (Argentina), [email protected] American Journal of Evaluation Over the last three decades, stakeholder involvement in evaluation has increased globally, as reflected in the number of theories and methodologies that prioritize participation in the evaluation process (Cardozo Brum & Rosas Huerta, 2021; Fetterman & Wandersman, 2018; Hay & Kumar-Range, 2014; Rodríguez-Campos, 2018; Zukoski & Bosserman, 2018). This interest in participation in evaluation is also evident in the Global Evaluation Agenda (2016-2020) 1 and the 2030 Agenda, in which the UN makes it clear that combating social inequalities means moving towards policies that “leave no one behind” (EvalPartners, 2015; United Nations Development Group, 2017). While there is evidence of an increase in these participant engagement practices in Latin America (Sanz et al., 2019; Tapella and Sanz, 2019), research on this practice specific to Latin America is limited. This article explores this evaluative practice in Latin America by examining a sample of 15 initiatives selected from within the framework of the EvalParticipativa project. 2 We also propose a set of principles that reflect current practice and offer a 1 The Global Evaluation Agenda (2016-2020), promoted by the EvalPartners global partnership, was approved at the II EvalPartners Global Evaluation Forum (Kathmandu, Nepal, 2015) and aims to promote and strengthen evaluation as a tool to help improve people's lives and social equity (EvalPartners, 2015). 2 EvalParticipativa is a joint initiative between the Research Program on Employment, Environment and Society (PETAS) at the National University of San Juan (Argentina), and the Focelac+ project (Strengthening a Culture of Evaluation and Learning in Latin America with a Global Outlook) run by the German Institute for Development Evaluation (DEval). The project has created a Community of Practice and Learning for Participatory Evaluation (PE) in Latin America, as a way of strengthening and 2 guide to stakeholder involvement in evaluation in Latin America. In order to link our regional discussion regarding evaluation approaches to stakeholder involvement to that of the Global North, this article addresses the following questions: What characterizes evaluation practices that integrate stakeholder participation in Latin America? What principles can be drawn from this practice to define stakeholder involvement approaches in the region? The article begins by summarizing existing literature on evaluation and participation. We then outline our methodology before presenting the results alongside theoretical reflections on the practice of stakeholder involvement in evaluation. The final section discusses our findings and links them to stakeholder involvement theory and practice in Latin America. Existing Research on Participation in Evaluation The relationship between evaluation and participation arose in the 1960s along with the emergence of qualitative evaluation and Participatory Action Research as social science researchers began to question positivist approaches (Plottu & Plottu, 2009). The classical evaluation models focused on measuring results from an external perspective, adopted a top-down approach, and applied quantitative techniques. Stakeholder involvement in evaluation, on the other hand, highlighted reflexivity as part of the evaluation process, proposed to include stakeholders’ voices to assess programs, and began to develop new qualitative techniques (Cousins et al., 2013; Everitt & Hardiker, 1996; Guba & Lincoln, 1989; House, 1980; Scriven, 1973). contributing to the inclusive involvement of civil society in evaluation processes in the region. https://evalparticipativa.net/en/ 3 However, it was not until the mid-1990s—and within an international framework aimed at promoting human development—that the field began to witness a flourishing of reflection and debate on participation and evaluation (Cartland et al., 2008; Chouinard & Milley, 2018; Cornwall, 2008; Cullen et al., 2011; Daigneault & Jacob, 2009; King et al., 2007, Ryan et al., 1998). During these three decades, various versions of stakeholder involvement in evaluation arose and evolved. These included empowerment evaluation, transformative evaluation, democratic deliberative evaluation and collaborative evaluation. All of these approaches focused on both the dynamic and continuous involvement of stakeholders in evaluation processes and the recognition of stakeholders’ distinct expertise (Chouinard & Milley, 2018; Fetterman et al., 2018; Nugroho et al., 2018; Mbava, 2017; Chouinard & Cousins, 2013; Cartland et al., 2008; Cousins & Whitmore, 1998; Monnier & Conan, 1995). In Latin America, “participatory evaluation” has been used as an umbrella term. More broadly, the field has identified three main evaluation approaches to stakeholder involvement: collaborative evaluation, participatory evaluation and empowerment evaluation. 3 In order to distinguish among these evaluation approaches, developed in the Global North, Fetterman et al. (2018, pp.2-3) consider the role of the evaluator or the evaluation team in each type. Collaborative evaluators are clearly in charge of the evaluation and are interested in creating ongoing engagement between the stakeholders and themselves. This contributes to stronger evaluation designs, enhanced data collection and analysis, and results that stakeholders can easily understand and use (Rodríguez-Campos, 2018). Participatory evaluators share control of the evaluation in a position of equality with the stakeholders, encouraging them to become involved in defining the evaluation, 3 This is expressed in the long history of work by the Collaborative, Participatory, and Empowerment Topical Interest Group of the American Evaluation Association (AEA), which is constituted by around 20 percent of the AEA membership (Fetterman, Rodríguez-Campos and Zukoski, 2018, p.9). 4 developing the instruments, collecting and analyzing the data, and reporting and disseminating the results (Zukoski & Bosserman, 2018). Empowerment evaluators understand the various stakeholders (program staff members, program participants, and community members) to be those in control of the evaluation, while the evaluator serves as a “critical friend” or coach to ensure a high-quality process (Fetterman & Wandersman, 2018). Figure 1 illustrates how these three main approaches vary. Figure 1. Stakeholder Involvement Approaches Source: Prepared by the authors. This article presents an integrated definition of the principles of stakeholder involvement in evaluation in Latin America, making connections across these three evaluation types. Taking a “stakeholder involvement in evaluation approach” requires involving varied stakeholders—including program managers, technical staff, end users, and other relevant social actors—throughout the evaluation process. Specifically, it 5 gives them a voice and establishes moments for consultation regarding issues that include what is evaluated; who participates; when it happens; what data collection and analysis methods are used; what the results, conclusions and recommendations are; and how the results are communicated (Coupal, 2000; Shulha, 2010, cited in Fetterman et al., 2014). Stakeholder involvement in evaluation, therefore, has a normative component, focused on democratizing the inquiry process, co-creating knowledge and responding better to the needs and particularities of each specific context (Chouinard & Milley, 2018; Plottu & Plottu, 2009). Today, stakeholder participation constitutes a dynamic process that cannot be detached from the policy and program type, the community and its participation capacity and history, the cultural context, and the purpose and conditions under which the evaluation is conducted (Trickett & Ryerson-Espino, 2004). In stakeholder involvement approaches, trained evaluators work in partnership with the stakeholders of a social intervention to produce evaluative knowledge concerning the intervention, generate learning, and make decisions in a concerted manner (Cousins et al., 2013; Cousins & Whitmore, 1998). Within the broader framework of stakeholder involvement in evaluation, the following two specific developments focus on promoting greater collaboration in evaluative processes (Cousins et al., 2013; Cousins & Whitmore, 1998; Patton, 2017). Practical evaluation —developed mainly in the developed Anglo-Saxon world and most applicable to the collaborative and participatory approaches to evaluation— emphasizes the involvement of stakeholders from a pragmatic problem-solving approach that seeks to improve the use and results of evaluation (Cousins et al., 2013). Transformative evaluation —attributed to the empowerment evaluation approach—, on the other hand, 6 has emerged in Latin America, India and Africa and seeks emancipation, empowerment, and social change. With regard to these two developments, there is an interesting debate around whether stakeholder participation should be instrumentalist or empowering (Arnstein, 1969; Hart, 1992; Mark & Shotland, 1985). From an instrumentalist point of view, participation is a top-down endeavor that implies involving stakeholders as informants. From this perspective, evaluation may lose its potential to democratize or transform (Cornwall, 2008). In contrast, an empowerment-style evaluation places stakeholders at the center of the evaluation. They co-design the methodology, carry out the field work, analyze the data, and present the results and recommendations— all with the support of a facilitating evaluation team (IDS, 1998). In short, these two conceptualizations of participation in evaluation are in tension with each other as one emphasizes the methodological demands of evidence-based practices, while the other focuses on social justice (Chouinard, 2013, cited in Chouinard & Milley, 2018; Eybem et al., 2015). These two perspectives are not only marked by evaluative culture and politics, but also by the power relations that exist between those who fund the policy and its evaluation, and those to whom the policy is addressed. These two perspectives have implications for who participates, how they participate, and to what degree. In the Latin American context, the transformative approach to stakeholder involvement in evaluation has been driven and promoted by a rich tradition in the field of social action that has placed an explicit and notable emphasis on the participatory dimension inscribed in a liberating perspective of reality. Specifically, stakeholder involvement in evaluation has been nourished by three traditions of inquiry-action, widely developed in the region: Popular Education, Participatory Action Research, and the Systematization of Experiences. Despite their distinct nuances, these traditions all 7 recognize the relevance of dialogue and participation, as well as the facilitating role of those who promote inquiry-action processes. Likewise, they all focus on social change and empowerment 4 (Sanz et al., 2019; Tapella & Sanz, 2019). Since the 1970s, knowledge of local stakeholders’ particular conditions has become integrated into the notion and practice of participation in evaluation. In particular, stakeholders’ well-being and vulnerability have been central to Latin American considerations of participation in evaluation. This includes native peoples (Kushner & Rotondo, 2012), peasants (Romero et al., 2017), older adults (Huenchuan & Paredes, 2007), at‐risk communities (Letichevsky & Penna Firme, 2012), and citizens and organized social groups in social policy management and evaluation (Cardozo Brum, 2008). In sum, this paper explores stakeholder involvement in evaluation in Latin America, where the literature on this subject is still scarce. In order to do so, we first present the methodology we have used in the research and then, link stakeholder involvement in evaluation in this region to current debates in the wider field of evaluation, before offering a set of principles. We specifically explore how the Latin American practice and principles connect to the collaborative, participative and empowerment approaches. Methodology 4 Popular Education (which emerged in the 1960s from the work of Brazilian pedagogue Paulo Freire) conceives education as a horizontal dialogue between educator and learner that becomes the key to awareness and change in the face of oppression and social exclusion. Participatory Action Research, (also developed in the region since the 1960s and based on the work of Colombian, Orlando Fals Borda) focuses on the joint creation of knowledge for social transformation and empowerment. Similarly, the Systematization of Experiences (which was birthed in the region in the 1970s) focuses on the recovery and critical analysis of cases to assess the changes generated by social and educational projects (Tapella & Rodríguez-Bilella, 2014; Tapella et al., 2021). In this regard, empowerment evaluation and Freirean pedagogy share a common emancipatory tradition, aligned in principle and practice. See Fetterman (2017, pp. 111-112). 8 This study aims to understand the complexity of stakeholder involvement in evaluation practice within the specific context of Latin America. To this end, we opted for “moderate generalization” 5 (Payne & Williams, 2005), which is based on rich data collection, does not postulate grand theories, and expresses its findings hypothetically. In this vein, the research was carried out using a qualitative and collaborative approach in two connected phases (see figure 2), based on concrete initiatives and joint work with stakeholders. Figure 2. Phases, Activities and Dimensions of Analysis Source: Prepared by the authors. 5 According to Payne and Williams, a study that aims to achieve moderate generalizations should take into account the following three criteria: generalizations of this type are valid in studies where there are complex processes, different situations and high heterogeneity; generalizations are plausible when made with caution, moderating the range of generalization in the conclusions; and a coherent relationship between evidence and conclusions is important, which should preferably be based on a limited period of time and space (2005). These three criteria have been followed in this research. First phase Virtual Forum: conditioning and facilitating factors of stakeholder involvement in evaluation. Selection of 15 experiences in Latin America. Second phase First Gathering of Participatory Evaluation Experiences in Latin America. Exchange of experiences: difficulties or limitations, advantages, specific features, key lessons and challenges from the development of stakeholder involvement in evaluation. The ‘what’ of stakeholder involvement in evaluation: principles of stakeholder involvement in evaluation. 9 In a first phase, the EvalParticipativa team 6 set up a Virtual Forum 7 as a collective learning space, aimed at generating an extended conversation regarding the field of stakeholder involvement in evaluation. In order to generate new lessons from real and concrete initiatives, the team issued an open invitation (through its webpage, social networks, evaluators’ networks) to identify examples of initiatives that integrate stakeholder involvement approaches to evaluation in Latin America. As a result of this process, and following a non-probability sampling procedure oriented by six criteria 8 (Patton, 2002), the team selected 15 initiatives that reflected the diversity of existing initiatives in the region. Some of these initiatives (see Table 1) worked at local level while others addressed regional issues. Most focused on social and community development and were often interested in transforming social conditions. They were multi-method evaluations that generated tools and instruments for the evaluation and generally made use of qualitative approaches. Table 1. Sample of Evaluations for This Project Evaluation Location(s) Stakeholders involved Monitoring and evaluation of the Experience Capitalization Project. Argentina, Brazil and Colombia in Latin America. Also, East Africa and Asia Technical Centre for Agricultural and Rural Cooperation (CTA) and International Fund for Agricultural Development (IFAD) 6 The EvalParticipativa team was coordinated by two professors and experts in stakeholder involvement in evaluation from the National University of San Juan. They were in charge of designing the project and carrying out its first phase. In the second phase, the coordinators and four evaluation experts participated, as facilitators, in the First Gathering of Participatory Evaluation Experiences in Latin America. 7 The EvalParticipativa Virtual Forum can be found at https://evalparticipativa.net/en/virtual-forum/. 8 The six criteria used were: (1) Multiple actors involved in the decisions during the evaluation process; (2) Scope/coverage of the initiatives and conditions for replicability; (3) Importance of the “evaluative” component over others (systematization, follow-up, etc.); (4) Initiatives completed versus initiatives in progress; (5) Innovative tools or instruments; (6) Initiatives that have documentary material to be communicated and disseminated. 16 diversity of stakeholders engaged and generates social spaces for meeting, dialogue and joint knowledge creation (Tapella et al, 2021). 13 Stakeholder Involvement: A Tool for Improving Evaluations According to the experiences we have examined from the region, the active involvement of affected individuals and entities allows the evaluation to view the object more closely than in conventional external evaluations. This favors a better understanding of the local context. In accordance with the Latin American literature, stakeholder involvement enhances methodological designs that are flexible and adaptable to varied contexts and evaluative purposes (Cavanna, 2019; Fundación Paraguaya, 2019). Likewise, it favors a greater understanding of the processes and, therefore, of the results, both expected and unexpected. In the initiatives reviewed, the varied actors involved in the evaluation felt that participatory evaluation approaches would be beneficial in future work. One way of encouraging institutional spaces to take on participatory approaches could be to share with them the advantages experienced in similar contexts (Sanz et al., 2019). The Challenge of Effectively Involving Excluded Voices From a social transformation perspective, stakeholder involvement initiatives emphasize the inclusion of all relevant voices, especially groups that are often excluded. They also insist on treating stakeholders as subjects of rights rather than mere 13 These characteristics -that emphasize how people can learn to take greater control of their own lives and the resources around themare central to the transformative branch of empowerment evaluation (Fetterman, 2015) with its focus on liberating individuals from pre-determined, conventional roles and organizational structures or “ways of doing things.” 17 informants. However, in the evaluations analyzed, the degree of effective involvement varies among the different evaluations. Firstly, most stakeholder involvement in evaluation processes is encouraged by the actors closest to the management of the intervention: technical teams, mid-level program officials, donors interested in enhancing participatory initiatives, etc. Only in some long-term initiatives with strong stakeholder involvement have these actors pushed and advocated for a greater influence in the evaluation dynamics. Secondly, on many occasions —and especially in governmental projects— the identification of relevant stakeholders is based on an a priori and often standardized list, adopting the aforementioned top-down approach. In the initiatives we have examined, a broad and complete definition of what constitutes a relevant stakeholder in each specific evaluation is considered essential. This ensures that historically excluded stakeholders take part in the evaluation process. Thirdly, not all stakeholders want to get involved and those who do get involved may not be clear about their roles or may not perceive themselves as evaluators. 14 This was the case in half of the initiatives analyzed. We also detected a challenge in maintaining consistency in participation over long periods of time. What Principles Emerge as Defining? During the First Gathering of Participatory Evaluation Experiences in Latin America, participants engaged in a collaborative process of defining the principles of stakeholder involvement in evaluation (see Table 2). These principles were drawn from the practical experiences of the participants and are presented below. These principles 14 This situation usually characterizes the initial phases of a collaborative evaluation approach (Rodriguez-Campos, 2018; Shulha et al., 2016). 18 —at times, in line with previous theoretical reflections— serve as postulates to guide this type of evaluation as it takes root in the region. They do not, however, constitute a closed set of principles. They remain open to dialogue with other present and future initiatives, as well as conceptual frameworks. Table 2. Principles of Stakeholder Involvement in Evaluation in Latin America 1 Actors are actively and consciously incorporated into the evaluation process as subjects of rights. 2 Local knowledge is recognized as valid and essential knowledge for evaluation. 3 Institutional or organizational representatives work in partnership with local stakeholders. 4 External evaluation teams act as facilitators of the evaluation process. 5 The use of didactic techniques and materials facilitates dialogue and exchange. 6 Stakeholders take ownership of both the processes and the results of the evaluation. Source: Prepared by the authors based on the discussions of the First Gathering of Participatory Evaluation Experiences in Latin America. Actors are Actively and Consciously Incorporated Into the Evaluation Process as Subjects of Rights. In the case of Latin America, the participants in the First Gathering of Participatory Evaluation Experiences in Latin America consider stakeholders to be subjects of rights. They do not view them as objects of the intervention, individuals in need of protection, or mere beneficiaries, as often happens in conventional evaluations. Coming from a transformative perspective, they recognize stakeholders as beings that are empowered for speech and critical thinking, decision making, and autonomy. They also recognize these stakeholders to be bearers of their own interests, expectations and priorities. As pointed out by various authors, these proponents of stakeholder involvement in evaluation also recognize that participation relates to the democratic function of public policy evaluation (Cornwall, 2008; Monnier & Conan, 1995; Plottu & Plottu, 2009). A commitment to the active involvement of the different actors has implications at every stage of the evaluation process, from its design through to the dissemination of 19 results (Sanz et al., 2019). Such active involvement entails that those who make the decision to carry out an evaluation with stakeholder involvement must assume that this implies sharing decision making at each stage of the evaluative process. This is nothing short of a ceding of power (Chouinard & Milley, 2018; Fetterman & Wandersman, 2018). Likewise, the understanding that participants are subjects of rights implies also recognizing the responsibilities determined by this condition. These characteristics align this first principle with the empowerment evaluation approach. Local Knowledge is Recognized as Valid and Essential Knowledge for Evaluation. A fundamental facet of the incorporation of stakeholders in evaluation requires that governmental bodies and international development agencies recognize and value the knowledge and know-how of the local actors. This is also asserted in the literature (Chouinard & Milley, 2018; Gadotti, 2017; Kushner & Rotondo, 2012; Mbava, 2017; Nugroho et al., 2018). The legitimization of their knowledge (at times, the fruit of the actors’ reading of their own reality) forms the basis for advancing in transformative processes (Zukoski & Bosserman, 2018). In the evaluation experiences that were shared, the way this knowledge unfolds is significant and is linked both directly and indirectly to the evaluation’s specific focus. In fact, emerging elements of local knowledge, that may seem insignificant to the evaluative purpose at first glance, can become productively integrated into the evaluation when facilitators are sensitive to it. These characteristics align this principle more closely with stakeholder involvement approaches to evaluation that acknowledge the role of people in producing evaluative knowledge. 20 Institutional or Organizational Representatives Work in Partnership With Local Stakeholders. Stakeholder involvement in evaluation is a process, facilitated by professional evaluators, in which different stakeholders work in partnership to generate evaluative knowledge. These stakeholders include program managers as well as local stakeholders. In accordance with the framework of stakeholder involvement approaches to evaluation, the stakeholders are the ones who should define what is evaluated, who participates, when it happens, what data collection and analysis methods are used, and how the results are communicated. Collaboration is required at every stage of the evaluation process. However, the delimitation of the focus of the evaluation and the negotiations that take place among the different actors can set the tone for the rest of the activities. Aubel (2000) notes that this first stage aims to generate moments of participation that are effective both for clearly delimiting the objective sought by participation and for structuring and channeling it during the evaluation process. This principle emphasizes the key role of the stakeholders during the whole evaluation process, thus it reflects an empowerment evaluation approach. External Evaluation Teams Act as Facilitators of the Evaluation Process. External evaluation teams can use participation in evaluation to contribute to empowerment and learning as well as improve the democratic culture of organizations and social groups. However, evaluation teams can also use it in a purely instrumentalist manner, reproducing existing power asymmetries and diminishing its transformative capacity (Cornwall, 2008; Hart, 1992; Mark & Shotland, 1985). 21 In this regard, the role of the evaluation facilitator gains relevance, as they are key to the success of any stakeholder involvement in evaluation approach. According to the Latin American experiences examined, and in line with other initiatives (Espinosa- Fajardo, 2019), it is important that the evaluation team is respectful and capable of managing groups as they collectively define what is important in the evaluation, decide how it should be carried out, and determine the results, conclusions and recommendations of the evaluation. The Use of Didactic Techniques and Materials Facilitates Dialogue and Exchange. In diverse and culturally complex contexts, inequalities in terms of power, voice and the capacities of the different actors present a challenge to the effective development of participation. Stakeholder involvement in evaluation has confronted this challenge by creating innovative didactic materials and participatory tools that help communicate and recover the perspectives of all involved in the evaluation process (Chouinard & Milley, 2018). In line with the experiences we have analyzed from Latin American practice, we have noted that the techniques activate both the mind and emotions, contributing to a greater and more real ownership of the reflections and findings. In addition, the tools enhance the emergence and development of critical thinking in the people involved in the process. The techniques, workshops and games seek to generate exchanges that deepen individual and collective knowledge while placing the stakeholders firmly at the center of the process. While all evaluation approaches that value and make space for stakeholders pay attention to the use of didactic techniques and materials, this principle most closely aligns with the framework and spirit of the collaborative evaluation approach. 22 Stakeholders Take Ownership of Both the Processes and the Results of the Evaluation. Stakeholder involvement in evaluation requires local actors to capture contextual specificities and ensure that their perspectives are included. This translates, from a transformative perspective, into a horizontal distribution of power in decision making and deeper democratization. Active participation enhances the understanding of other points of view and facilitates access to a common interpretation of the program under evaluation. Furthermore, according to the experiences examined from the region, stakeholder involvement in evaluation generates a process through which local actors increase their opportunities to take ownership of the evaluation processes and, thus, assume more ownership of the results. This principle aligns perfectly with empowerment evaluation. Discussion Research on participation in evaluation is constantly evolving and is now a key issue in the 2030 Agenda and the “Global Evaluation Agenda” (2016-2020). In this regard, the launch of the Sustainable Development Goals constitutes a turning point in the theory and practice of evaluation given the prominence awarded to civil society (EvalPartners, 2015). The Latin American experiences shared here reveal the characteristics of the practice in this region. Some of these characteristics are common to other territories and reveal some shared advances and challenges related to stakeholder involvement in evaluation practice at global level. Our research has led us to propose a set of principles which, based on experiences taken from Latin America, we think are relevant to stakeholder involvement approaches in evaluation in other 23 regions. We will now reflect on the key conditions required to advance stakeholder participation in evaluation in Latin America. These conditions are open to future discussion in this and other regions. The Latin American experience underscores that stakeholder involvement in evaluation is based on the active and conscious participation of all individuals as subjects of rights and on the recognition of their local knowledge. It also relies on the collaborative and horizontal work of all stakeholders, promoted by the facilitation of evaluation teams and the use of didactic techniques and materials to ensure that stakeholders take on a sense of ownership of the evaluation. Facilitation can act like a lubricant for the different gears that make up stakeholder involvement in the evaluation process. However, facilitation is sometimes a complex task that requires specific training 15 . Techniques, games or workshops can contribute to the expression of multiple perspectives from several social actors. This implies facilitating debates and trust, putting one’s whole self into the dynamics of learning (not only thinking) as well as generating exchanges that allow one’s own and mutual knowledge to deepen. In this sense, it requires detaching stakeholder involvement in evaluation from merely technocratic approaches, in which techniques and instruments are used for purely entertainment purposes and do not really help integrate the perspectives of local actors. Progress in stakeholder engagement also requires changes to the evaluation culture and policy, evaluation approaches and purposes, as well as the degree and mode of fostering horizontal relationships between the different stakeholders. At present, the influence of governmental bodies and international development agencies (who fund the vast majority of the evaluations) define how evaluation is understood and implemented. 15 The EvalParticipativa project, with its specific training on facilitation and the use of tools, and its specialized manual on the subject, provides opportune ways to advance in this direction. 24 While civil society organizations that have a focus on social justice tend to promote participation in evaluation, governmental bodies and agencies usually prefer more classical approaches and adopt a style of participation that is closer to an instrumentalist perspective than an empowering one. Therefore, proponents of stakeholder involvement in evaluation should prioritize working with public administrations and agencies to show them the benefits of integrating participation into their evaluations. In the same vein, to ensure that evaluations do not reduce participation to mere consultation under the symbolic pretense of participation, the experiences we examined reveal the need to highlight social and power inequalities, manage conflicts of interest that emanate from them, and promote dialogue and exchange throughout the evaluation process. This would open up evaluative culture and policy to non-classical approaches that are committed not only to accountability, but also to learning and improvement. Likewise, evaluators need to define concrete mechanisms and apply tools to promote the effective involvement and participation of a diverse set of stakeholders that evaluations have historically left out in the region. Furthermore, after examining the experiences we have mentioned, we believe that evaluation in the region has two pending tasks if varied stakeholders are to have an effective voice in initiatives. Firstly, it needs to build shared language on stakeholder involvement in evaluation which breaks away from the flexible and multiple uses of the concept of “participation”. Secondly, it needs to find ways to encourage active and empowering involvement among all stakeholders. This implies looking at two interconnected sides of the same coin: reflecting on the opportunity to participate and the capacity to participate. The first, linked to the evaluative culture and policy of the context, is determined by the institutional and political will of those who design and conduct an evaluation to create real spaces for participation. The second, linked to the 25 evaluation process itself, is mainly determined by the attitudes and skills that the different stakeholders have developed through previous participatory experiences. Regarding the opportunity to participate, the analyzed initiatives show that not all institutions or organizations that request an evaluation are really willing to facilitate and accompany processes of stakeholder involvement in evaluation. Challenges include whether or not the commissioning body is actually willing to go through with this kind of evaluation or question its legitimacy, and the availability of spaces and mechanisms required for dialogue between the different social actors. Furthermore, Latin American institutions often reject or question the very democratic values that are required to bring about these changes. In line with the lessons drawn from the initiatives analyzed, it is particularly important to involve those who are in decision-making positions and who promote the institutionalization of participation in the entities that commission the evaluations. Likewise, time and economic resources are required to develop these types of processes which, given the above, are usually very limited and insufficient. In relation to the capacity to participate, although in the Latin American context there is an increasing number of stakeholder involvement in evaluation initiatives, there are important challenges to be tackled. The capacity for participation is enhanced when institutions offer opportunities for participation and encourage stakeholders to become decisively involved in projects. However, as we have indicated, in Latin America there is a predominance of institutions where citizen participation is only promoted to a limited extent. These participation capacities can also be promoted during the evaluation process itself through active facilitation. 16 16 This type of training has been promoted since 2019 by the EvalParticipativa project through the development of facilitation training and the production of training materials. 32 Romero Sarduy, M. I., Moreno Moreno, I., & Hernández Chávez, C. N. (2017) Evaluación Participativa por Protagonistas: una experiencia de innovación social. Revista Estudios del Desarrollo Social: Cuba y América Latina, 5 (2). 81-91. Ryan, K., Greene, J., Lincoln, Y., Mathison, S., & Mertens, D. M. (1998). Advantages and challenges of using inclusive evaluation approaches in evaluation practice. American Journal of Evaluation, 19(1), 101–122. Sanz, J. C., Tapella, E. & Meneses, A. (2019). Participatory Evaluation of Cancer Prevention and Care Services. A Case Study from Valle de la Estrella, Costa Rica. In J. B. Cousins (ed.), Collaborative Approaches to Evaluation: Principles in Use (pp. 47-53). Thousand Oaks, USA: Sage. Scriven, M. (1973). Goal free evaluation. In E. R. House (ed.), School Evaluation: The Politics and Process (pp. 319-328). Berkeley, USA: McCutchan. Shulha, L. M. (2010, November). Participatory evaluation essentials: Highlighting the essential features of participatory evaluation. Paper presented at the American Evaluation Association Conference, San Antonio, Texas. Shulha, L. M., Whitmore, E., Cousins, J. B., Gilbert, N., & Al Hudib, H. (2016). Introducing Evidence-based Principles to Guide Collaborative Approaches to Evaluation: Results of an Empirical Process. American Journal of Evaluation, 37(2), 193-215. Tapella, E. & Rodríguez-Bilella, P. (2014). Shared learning and participatory evaluation: The sistematización approach to assess development interventions. Evaluation, 20(1), 115–133. Tapella, E., Rodríguez Bilella, P., Sanz, J. C., Chavez-Tafur, J., & Espinosa-Fajardo, J. (2021). Sowing & harvesting. Participatory Evaluation Handbook. Bonn: German Institute for Development Evaluation. 33 Tapella, E. & Sanz, J. C. (2019). La otra mirada. Evaluación participativa y mejora de los Servicios de Prevención y Atención del Cáncer en Valle de la Estrella, Costa Rica. Revista Gestión y Análisis de Políticas Públicas, 22, 102-119. Taut, S. (2008). What have we learned about stakeholder involvement in program evaluation? Studies in Educational Evaluation, 34(4), 224-230. Trickett, E. J., & Espino, S. L. R. (2004). Collaboration and social inquiry: Multiple meanings of a construct and its role in creating useful and valid knowledge. American Journal of Community Psychology, 34(1-2), 1-69. United Nations Development Group (2017). Fulfilling promises: A practical guide for UN advocacy to promote implementation of the 2030 Agenda. United Nations. Zukoski, A. P., & Bosserman, K. (2018). Essentials of participatory evaluation. In D. M. Fetterman, L. Rodríguez-Campos, & A. P. Zukoski (Eds.), Collaborative, Participatory and Empowerment Evaluation. Stakeholder Involvement Approaches. Guilford Press.