Evaluation as a Tool for Leading Change

183 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change CHAPTER 10 Evaluation as a Tool for Leading Change PASI-HEIKKI RANNISTO & ANNA SALORANTA Takeaways for Leading Change In this chapter, we acquaint the reader with evaluation, touching shortly on its functions and some of the theories informing it, as well as the role it plays in the changing public sector. We introduce practical ways of using evaluations as tools for managing change. Finally, we argue these are tools we already often have at our disposal but do not utilise fully. Leaders may not recognise the opportunities and tools an evaluation provides them with in terms of leading change. We hope to show how embracing a wider culture of evaluation will build a foundation of systematic and shared information helping leaders create and discuss change across sectors. 184 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change E ffecting change in complex and challenging contexts requires tools in addition to leadership skills. Evaluation is one such tool; it offers a functional point of view of leadership and the practical management of change by describing complex environments, relationships, and problems from the perspectives of all stakeholders. Evaluation does not only show whether the progress achieved is in line with project objectives. It can also be used to educate the organisation or network about an ongoing process of change. Such a process provides an opportunity to define and communicate leadership visions and goals as they pertain to individuals, processes, and organisational structures. Change, evaluation, leadership, and management share many of the same functions and operation models, most notably the setting of objectives and the defining of practical outcomes and measurements. Several studies have shown that leading change efficiently requires evaluation. In practice, few organisations do this. Consequently, they never find out why the desired change has manifested as something undesirable or why their outcomes do not match their designs (Doyle, Claydon, & Buchanan, 2000; Hayes, 2002; Kirkpatrick, 1985; Owen & Rogers, 1999; Skinner, 2004). Both change projects and evaluations are central themes in results-based management. Their key factors are achieving development targets, clarifying the roles and responsibilities of actors, increasing transparency and accountability in functions, and using good-quality data to improve decision-making (Rist, Boily, & Martin, 2011). Our approach to evaluation and its use in leading change is broad, requiring us to examine our subject from many different angles. First, evaluation can be seen as a means of measuring performance. The objective can vary from the success of an individual development or change project, operational work, organisational restructuring, and implementation of organisational strategy, to the success of a policy on a national or local level (Figure 1). Second, evaluation is needed when leading the process of change itself. Evaluation can be done for a number of different purposes. In education, for instance, evaluating students is a key process for teachers. Based on the information gained, a teacher might decide to teach in a different way or give extra support and encouragement in certain areas. In social work 185 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change or healthcare, the effects of interventions and treatments in the lives and behaviours of individuals is under constant scrutiny. In this context, the process shows the connections between the chosen intervention, the individual’s behaviour, and the outcome, all in relation to the original objective. The change, development, and healing of an individual is followed closely during care, as are its effects on his or her life. In these instances, evaluation is a natural part of a service process, like a visit to your healthcare provider or an exam as part of a learning process. Evaluation can also be directed towards the outcome of a larger government reform or policy. This can be done from a number of different perspectives, such as that of the citizen, politics, or government (Airaksinen, Haveri, & Vallo, 2005). The different roles of citizens can be the object of interest. They are voters, decision makers, service users, and taxpayers. The political viewpoint would entail examining the political administrative systems as well as the division of labour between different actors. From a governmental point of view, one would focus on the administrative culture and the work of civil servants as well as on the government as a whole. Another point of interest might be the objectives stated in legislation and how these have been implemented by the government (Temmes, 1994). Reforms are often evaluated in different ways throughout the stages of their implementation. From the very beginning, the focus would be on the design, objectives, and available resources. At the point of completion and even past that, the focus would be on how the original objectives were met and whether the change had the desired effects. A reform can also be POLICY STRATEGY CHANGE OPERATIONAL INDIVIDUAL Management Leadership FIGURE 1. Different foci of evaluation 186 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change evaluated during its implementation, the object then being the process of change itself and the actions taken to achieve it. It has been suggested that all government reforms should undergo evaluations from their planning stage onwards (Skinner, 2004; Temmes, 1994). A Short History of Evaluation in the Public Sector Three distinct stages can be identified in the development of evaluation in the public sector, all of them linked to a change in welfare state policy making (Wollmann, 2003). From the 1960s onwards, there has been increasing interest in public policy-making and its implementation. In both the academic and the public spheres, it has become clear that formulating policies and funding programmes are simply not enough to effect societal change. There has to be accountability and some idea of how programmes work. Research and evaluations focused on how policies are taken from plan to reality. Many factors have been identified as having an impact on the process of implementation, not least of which was the effect of individual actors themselves (for an introduction to policy implementation, see, for instance, Pressman & Wildavsky [1984] or Sabatier [1991]). However, research on implementation alone did not sufficiently inform policy-makers of the efficacy of their policies. Examining outcomes as well as implementation was necessary for sufficient feedback for policy reformulation. The goal was to improve not only societal discrepancies but also the policies addressing them. This was the beginning of the second stage of development in the 1970s and 1980s (Rist et al., 2011; Wollmann, 2003). Simultaneously, New Public Management (NPM) strongly affected the execution of public policies. They were to be brought under the same kind of efficiency and output requirements as the private sector and industry. NPM strove to bring about a more efficient and economic thinking to public sector operations and management (Pollitt, 1995). This meant that both the policy outcomes and formulation were examined, changing the way evaluations were conducted. Merely measuring output does not satisfy the 187 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change question “Is this policy efficient?” In the 2000s, evaluation has developed alongside the metamorphosis of NPM into New Public Governance (NPG), bringing about the third stage of development. During this period, policy-makers have sought to adapt to growing changes in increasingly complex environments. Instead of policies or projects being implemented from the top down, the emphasis has been on shared responsibility and shared decision-making between partners. Controlled public-private partnerships are evolving into networks. Top-down implementation has given way to co-creation. Partnerships are no longer seen as being formed solely between organisations but now also include citizens, either as active individuals and customers or as non-profit organisations and interest groups (Christensen & Laegreid, 2017). Not only has the public sector changed in the Western welfare state, especially in the Nordic countries, but democracy itself is also finding new ways of expression. The rate of change is challenging and it changes the way we have to think about and conduct evaluations. To produce the information necessary for effective leadership in this complex context, evaluations need to factor in a wider range of stakeholders, operational cultures and networks. The range of different identifiable mechanisms possibly effecting the outcomes becomes much wider. The outcomes of varying combinations of contexts and mechanisms offer evaluation researchers the building blocks for testing different theories. There is no one true evaluation theory but several coexisting ones. Evaluators are responsible for clarifying their theoretical approach and the methods they have chosen in their own work. One of the theories commonly used for complex evaluations is, the realistic evaluation framework. It is popular today because it accommodates a variety of methods, thus making it easier to measure more complicated changes. The “fathers” of realistic evaluation are Pawson and Tilley (1997). They developed the approach around a basic theory-driven concept of Mechanism + Context = Outcome. This means that the building of the evaluation framework starts with a theory of causal explanation. The regularities of patterns in social activities form the mechanism, and all human activities happen in a specific context. Top-down implementation has given way to co-creation. 188 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change The realistic evaluation cycle is built around testing for variations in variables in an attempt to expand the underlying theory. The testing itself, according to Pawson and Tilley (1997), is done with the method best suited for answering the research question. Realistic evaluations are done in cycles, each one adjusted by the knowledge acquired in the previous cycle. When evaluating a social intervention, it senses to focus first on statistical analysis to get the big picture. Then, in consecutive cycles, focus more deeply on more interesting aspects by using qualitative methods (e.g., interviews) in a specific community/part of the population. The specific value of realistic evaluations is that the design is adaptable to the course and needs of the project. There are various opportunities to educate and engage the actors and stakeholders during the different cycles of the evaluation. Ultimately, it is the responsibility of evaluators to ask and answer questions that will most benefit the project at that particular time. These could be questions about what the preconditions for a successful implementation are, questions about how a project is being implemented and where it is likely to lead, and questions about what kinds of outcomes a certain project has produced and whether these are in line with the objectives. Evaluation as a Tool for Creating and Leading Change The theories of change management and realistic evaluation share some commonalities, not least of which is the fact that each evaluation process or change is unique. Generalisations only work when operating with a broad analytical frame of reference that is comprehensible to all involved. Without a well-thought-out design, neither change management nor evaluation will produce sufficient information to determine success or enable learning (Kuipers & Richardson, 1999; Skinner, 2004). Evaluations and processes of change are affected by the context and conditions. Both attempt to create a systematic method of implementation 189 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change and approach to the objective. In the former, the attempt is to conduct the evaluation process itself. In the latter, it is to effect change (Preskill & Torres, 1999; Russ-Eft & Preskill, 2009). Context is an important factor when evaluating different changes and interventions or when trying to generalise and learn about causal pathways (Blamey & Mackenzie, 2007). Both change and evaluation approaches share a disinclination to use qualitative methods as the only data collecting strategy for the evaluation of complex interventions or changes. One reason for this could be that even though the participant’s experience and view, as measured by qualitative methods, produce an estimate of the effects of conditions particular to a project, the results will not be relatable to any other project. The results are relative. For this reason, using qualitative methods might leave the question of what works in the greater context of things unanswered (Blamey & Mackenzie, 2007). This is not to propose that qualitative methods are obsolete, but rather that being aware of the possibilities and restrictions of the methods one intends to apply is central. The material or data for evaluation can be obtained from many sources. A secure approach is to collect data from numerous sources, both quantitative and qualitative. This enables data triangulation in order to ensure the reliability of the results. In the management of change, the final result is the focus of all attention, whereas this is only part of what falls under an evaluation, where the processes, different conditions, steering mechanisms, and management are all of interest and examined with the eyes of an outsider. The objectives of a project or reform direct people’s actions as well as the evaluation of the progress. Operations and processes are measured and can be realigned if they are not in line with the set objective. Furthermore, evaluation will examine the change as a whole during the entire process, while management of change looks forward to the end result. In the management of change, decisions are made for an assumed future reality, when evaluation can operate more or less in reality (Blamey & Mackenzie, 2007; Skinner, 2004). By striving for economy, effectiveness, and practical solutions, processes of change and evaluation aim at finding new ways to function by looking for unsuitable ways of action and changing them. Measures central to reaching the set objectives can be found naturally in these processes (Blamey & Mackenzie, 2007; Pawson & Tilley, 1997). Pawson 190 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change and Tilley (2001) emphasise the importance of developing a culture of evaluation and the perseverance required to do so. They add that the traits of the evaluation factors also work well in managing organisational change. When problematic themes arise during an evaluation, their origins may, after further inspection, already be visible in earlier evaluations. Being able to rely on several consecutive evaluations or evaluation cycles helps to identify the root causes of a problem, thus enabling leadership and management to tackle the disease instead of the symptom. This plural nature is also true for the management of change. Each process of change already bears within it the seed for the following one. Some things will change in the intended direction, but something else will fail to change. Development through evaluation is cyclical, not linear (Bouckaert & Halligan, 2007). During the evaluation, small questions are worth asking, even to test big theories. In evaluation, as in change, it is worthwhile to proceed with small steps even if the overall picture is big. It is important to use as many different methods as possible. This way, a clearer, more versatile picture can be obtained from the actual outcome. It is important to notice which mechanisms are relevant to producing optimum outcomes by context. Policies affect several practical mechanisms, changing and directing them. It is possible for different environments and conditions to produce similar results. Meta-analytic inquiries may help to make common policy mechanisms more visible. Asking the “why” questions and trying to answer how the conditions have affected the outcomes is important (Pawson & Tilley, 2001). As leaders and evaluators, we need to be careful never to assume to know what works. Instead, we must persistently try to prove it. People act differently, conditions change, and new points of view arise. These can all have an effect on what works in a given situation. In both evaluation and managing change, the focus should be on finding out what genuinely works and why (Pawson & Tilley, 2001). Leadership makes up its own level of evaluative inquiry. Today, many national policies have goals such as the change of organisational culture. This means changing the way different professions or departments communicate, form networks, and acting around shared processes 191 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change for the benefit of shared goals and/or clients. These are truly complex environments. From a leadership point of view, creating a shared culture of operation in a multiorganisational context poses new demands, as the actors involved are not under shared management. Instead, it has a number of leaders, all representing their own professions. Evaluations can provide a tool for shared objective setting and the defining of goals, as well as a natural platform for discussion and education. Different studies have pointed out the necessity of feedback in gaining acceptance of, and commitment to, change objectives. Evaluation done from the beginning of the change process can provide staff with the opportunity to tell the management and leadership about its own experience; it also offers development proposals for the implementation of change (Carnall, 1990; Kirkpatrick, 1985; Preskill & Torres, 1999). Evaluation in Practice The reason that evaluations have become so common may lie in the direction in which the public sector has evolved, requiring projects to quickly and efficiently source information and gauge their starting point. The choice of what to measure to determine the success of a project is not trivial or simple. Between deciding what to measure (quality, customer value, output, or input) and what point of view to choose (customer, employee, service provider, or service commissioner), it can be hard to show indisputably the effects of any process. Evaluation is always connected to some purpose or objective. Without this, it is often difficult to see the difference between a success and a failure in complex situations and environments. Successes and failures can be difficult to relay without the information provided by an evaluation process (Patton, 1987). At best, evaluation will provide new information, points of view, and solutions that can be used in the next phase of a reform or organisational change (Chelimsky & Shadish, 1997). At its core, it is a question of making value judgements and assessments to reach a conclusion in terms of success. This is done through a regulated process wherein a phenomenon is examined through a set of evaluation criteria (Vedung, 1997). 198 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change Bouckaert, G., & Halligan, J. (2007). Managing performance: International comparisons. London, England: Routledge. https://doi.org/10.4324/9780203935958 Brodkin, E. Z. (2008). Accountability in street-level organizations. International Journal of Public Administration, 31(3), 317–336. https://doi.org/10.1080/01900690701590587 Carnall, C. A. (1990). Managing change in organizations. New York, NY: Prentice Hall. Chelimsky, E., & Shadish, W. R. (1997). Evaluation for the 21st century: A handbook. Thousand Oaks, CA: Sage. Chen, H. (1990). Theory-driven evaluations. Newbury Park, CA: Sage. Christensen, T., & Laegreid, P. (2017). Transcending new public management: The transformation of public sector reforms. Aldershot, England: Ashgate. https://doi.org/10.4324/9781315235790 Doyle, M., Claydon, T., & Buchanan, D. (2000). Mixed results, lousy process: The management experience of organizational change. British Journal of Management, 11(Supplement 1), S59–S80. https://doi.org/10.1111/1467-8551.11.s1.6 Gambrill, E. (2001). Social work: An authority-based profession. Research on Social Work Practice, 11(2), 166–175. https://doi.org/10.1177/104973150101100203 Hayes, J. (2002). The theory and practice of change management. New York, NY: Palgrave. Kirkpatrick, D. L. (1985). How to manage change effectively. San Francisco, CA: Jossey-Bass. Kuipers, H., & Richardson, R. (1999). Active qualitative evaluation: Core elements and procedures. Evaluation, 5(1), 61–79. https://doi.org/10.1177/13563899922208823 Mänttäri-van der Kuip, M. (2013). Julkinen sosiaalityö markkinoistumisen armoilla? [Public social work at the mercy of markets] Yhteiskuntapolitiikka, 78. Retrieved from http://urn.fi/URN:NBN:fi-fe201306103881 Owen, J. M., & Rogers, P. J. (1999). Program evaluation: Forms and approaches. London, England: Sage. 199 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change Paasio, P. (2003). Vaikuttavuuden arvioinnin rakenne ja mahdollisuus sosiaalialalla [The structures and possibilities of evaluating efficiency in the social sector]. Helsinki, Finland: Stakes. Palola, E., & Karjalainen, V. (2011). Sosiaalipolitiikka: Hukassa vai uuden jäljillä?[Social politics: is it lost or on to something new?]. Helsinki, Finland: Terveyden ja hyvinvoinnin laitos. Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage. Pawson, R., & Tilley, N. (1997). Realistic evaluation. London, England: Sage. Pawson, R., & Tilley, N. (2001). Realistic evaluation bloodlines. American Journal of Evaluation, 22(3), 317–324. https://doi.org/10.1016/S1098-2140(01)00141-2 Pollitt, C. (1995). Justification by works or by faith? Evaluating the new public management. Evaluation, 1(2), 133–154. https://doi.org/10.1177/135638909500100202 Preskill, H. S., & Torres, R. T. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage. Pressman, J. L., & Wildavsky, A. (1984). Implementation: How great expectations in Washington are dashed in Oakland; or, why it’s amazing that federal programs work at all, this being a saga of the economic development administration as told by two sympathetic observers who seek to build morals on a foundation of ruined hopes. Berkeley, CA: University of California Press. Rajavaara, M. (2007). Vaikuttavuusyhteiskunta: Sosiaalisten olojen arvostelusta vaikutusten todentamiseen [The outcome-oriented society. Styles of reasoning about the outcomes in social policy]. Helsinki, Finland: Kela. Rist, R. C., Boily, M., & Martin, F. (2011). Influencing change: Building evaluation capacity to strengthen governance. Herndon, VA: The World Bank. https://doi.org/10.1596/978-0-8213-8403-9 Russ-Eft, D. F., & Preskill, H. (2009). Evaluation in organizations: A systematic approach to enhancing learning, performance, and change. New York, NY: Basic Books. 200 Leading Change in a Complex World: Transdisciplinary Perspectives Evaluation as a Tool for Leading Change Sabatier, P. A. (1991). Toward better theories of the policy process. PS: Political Science and Politics, 24(2), 147–156. https://doi.org/10.2307/419923 Scriven, M. (1991). Evaluation thesaurus. Newbury Park, CA: Sage. Skinner, D. (2004). Evaluation and change management: Rhetoric and reality. Human Resource Management Journal, 14(3), 5–19. https://doi.org/10.1111/j.1748-8583.2004.tb00123.x Temmes, M. (1994). Hallinto puntarissa: Hallintouudistusten arvioinnin mahdollisuudet ja edellytykset [Weighing up the government: the possibilities and prerequisites of government reform]. Helsinki, Finland: Ministry of Finance. Vedung, E. (1997). Public policy and program evaluation. New Brunswick, NJ: Transaction. Wollmann, H. (2003). Evaluation in public-sector reform: Concepts and practice in international perspective. Cheltenham, England: Edward Elgar. Leading Change in a Complex World: Transdisciplinary Perspectives Copyright © 2019 Tampere University Press and the authors Edited by Anni Kangas, Johanna Kujala, Anna Heikkinen, Antti Lönnqvist, Harri Laihonen and Julia Bethwaite Tampere: Tampere University Press