An improved evaluation framework to support local level planning

1 An improved evaluation framework to support local level planning Authors R. Giné Garriga 1 , A. Jiménez Fdez. de Palencia 2 , O. Flores Baquero 3 and A. Pérez Foguet 4 . Abstract Today, a vast proportion of people still lack a simple pit latrine and a source of safe drinking water. To help end this appalling state of affairs, there is a pressing need to provide policymakers with evidences in base-effective planning, targeting and prioritisation. Amongst others, two major challenges often hinder this process: i) lack of reliable data to identify which areas are most in need; and ii) inadequate instruments for decision-making support. In tackling previous shortcomings, this paper proposes an evaluation framework to compile, analyse and disseminate water, sanitation and hygiene (WASH) information. In an era of decentralisation, where decision-making moves to local governments, we apply such framework at the local level. The ultimate goal is to develop appropriate tools for decentralised planning support. To this end, the study first implements an innovative methodology for primary data collection, which combines the household and the water point as information sources. In so doing, we provide a complete picture of the context in which WASH services are delivered. Second, the collected data are analysed to underline the emerging development challenges. The use of simple planning indicators serves as the basis to 1. Reveal which areas require policy attention, and to 2. Identify the neediest. Various mechanisms are then proposed to translate previously identified development potentials into development initiatives, in which base the formulation of strategies to steer progress. Three different case studies from East and Southern African countries (Kenya, Tanzania and Mozambique) are presented. Results indicate that accurate and comprehensive data, if adequately exploited through simple instruments, may be the basis of effective targeting and prioritisation, which are central to sector planning. The application of the proposed framework in the real world, however, is to a certain extent elusive. We point out to conclude two specific challenges that remain unaddressed; 1 Research Group on Cooperation and Human Development, University Research Institute for Sustainability Science and Technology, Universitat Politècnica de Catalunya, Spain. Email: [email protected] 2 Research Group on Cooperation and Human Development, University Research Institute for Sustainability Science and Technology, Universitat Politècnica de Catalunya, Spain. E-mail: alejandro.jimenez.fdez.de.[email protected] 3 Research Group on Cooperation and Human Development, University Research Institute for Sustainability Science and Technology, Universitat Politècnica de Catalunya, Spain. E-mail: [email protected] 4 Dept. of Applied Mathematics III, Research Group on Cooperation and Human Development, University Research Institute for Sustainability Science and Technology, Civil Engineering School, Universitat Politècnica de Catalunya. Email: [email protected] 2 namely the effective and continued use of these instruments in sector decision-making processes and the design of data updating mechanisms. Keywords Household survey, local decision-making, planning indices, Sub-Saharan Africa, waterpoint mapping Introduction Diseases related to insufficient and unsafe drinking water, unimproved sanitation and poor hygiene education are common causes of illness and death (Cairncross et al., 2010, Esrey et al., 1991). In addition, the benefits of improved services provision are central to the cycle of disease and poverty, but they are rarely enjoyed by the most vulnerable (Cortinovis et al., 1993). Up to date, progress in ensuring access to these basic services has remained elusive in much of the developing world, where recent estimates show that a vast proportion of people still lack a simple pit latrine and a source of safe drinking water (Joint Monitoring Programme, 2012). At the same time, the sector has been experiencing a decentralisation of responsibilities where decision-making moves to local administrative units and decentralised bodies assume some political autonomy. For decentralisation to work effectively, however, there is a need of self-governments that are accountable for the performance of service delivery. This requires, amongst others, innovative management tools for bringing about a more equitable allocation of resources (Jiménez and Pérez-Foguet, 2010a). In turn, mechanisms for prioritisation and targeting depend on the availability of reliable information which is essential to assist decision-makers in i) identifying those sector areas and population groups most in need, ii) improving transparency in budget allocation procedures, and iii) measuring progress. Such information is often missing in many countries, but even when it is available, there is no guarantee that it is adequately exploited for planning and monitoring purposes. Political will and management-related capacities are further requirements that hinder informed decision-making. In an effort to address the first shortcoming cited above, i.e. lack of reliable data and inadequate governance tools, and ultimately improve sector planning, the aim of this study is to outline an evaluation framework to compile, analyse and disseminate water, sanitation and hygiene (WASH) information. As regards to data collection, it takes the Water Point Mapping (WPM) as starting point to comprehensively record all improved water sources at the area of intervention. This information is combined with data provided from a household-based survey in which a representative sample of households is selected to assess sanitation and hygiene habits. The data is analysed to highlight the emerging development challenges and provide evidences that help determine what gets done and where. To do this, a set of simple planning indicators serve as the basis to rank population groups and reveal which areas may be most in need of further investment. Different dissemination mechanisms are finally in place to translate previous development potentials into beneficial development initiatives, in 3 which base the formulation of strategies to steer development and progress in the region. This paper documents three different case studies in East and Southern Africa, namely the district of Kibondo (Tanzania, 2010), the district of Homa Bay (Kenya, 2011) and the municipality of Manhiça (Mozambique, 2012). Methodology In terms of method, study’s implementation is two-fold. A comprehensive assessment of WASH issues at local level is carried out through an innovative methodology for field data collection, which combines the household and the water point as information sources (Giné Garriga et al., 2012, Under review). On the basis of the analysis of collected data, a set of easy-to-use planning tools are developed to improve decisionmaking, specifically for prioritisation and targeting support. However, the uptake for such instruments by policymakers is, at best, challenging, and they commonly do without them (WaterAid, 2010). Limited capacities of recipient institutional bodies, inadequate sector-related institutional framework, lack of data updating mechanisms or poor interaction between academics and practitioners are common reasons that hamper an adequate appropriation and continued use of the developed tools. This study considers the local authority as the principal stakeholder and specifically engages in various stages of the process with those government bodies with competences in WASH. Moreover, all planning instruments are applied at the administrative scale in which decisions are based. Finally, the proposed planning tools are not only user-friendly (easy to assess, easy to understand) but presented in a way that provides clear messages and communicates a picture to decision-makers and potential beneficiaries quickly and accurately. As further discussed below, these measures are necessary and proved helpful but probably become insufficient to effectively address by themselves the challenges cited before. Assessment of water, sanitation and hygiene issues The approach adopted for data collection combines a water point mapping with a household survey; both of them conducted in the three case studies by a consultancy firm working in close collaboration with government’s technicians. In brief, the mapping methodology can be described as an “exercise whereby the geographical positions of all improved water points1 in an area are gathered in addition to management and technical data” (WaterAid and ODI, 2005). WPM involves the presentation of this information in a spatial context which enables a rapid visualisation of the distribution and status of water supplies. By linking these point data with demographic information, WPM objectively demonstrates who is and is not served; thus becoming a valuable analysis and planning tool for decentralised governments. Besides the mapping, a survey is conducted to assess sanitation and domestic hygiene in which the household (HH) is taken as the basic sampling unit. The design and selection of the sample draw on the Multiple Indicator Cluster Survey (MICS), i.e a methodology developed by UNICEF to collect social data (United Nations Children’s Fund, 2006). 4 Much like the MICS, the study population is stratified into a number of small mutually exclusive and exhaustive groups (strata). However, since mapping of water points is obliged to cover the whole area of intervention, the main difference when sampling is that a sample of households is selected from each stratum (stratified sampling) rather than selecting a reduced number of strata, from which a subsample of households is identified (cluster sampling). In so doing, the risk of homogeneity within selected houses remains relatively low, thus reducing the overall sample size required to obtain reasonable statistical precision of final estimates (Giné Garriga et al., 2012, Under review). Ideally, for household sampling, a defined number of houses would be identified in a statistically random manner from the population census. More often than not, however, accurate and updated censuses are lacking and literature suggests different sampling techniques to achieve a near-random selection (Lemeshow and Stroh, 1988, Bennett et al., 1991). In these cases, two aspects should be considered: i) design a clear method which does not give the enumerator the opportunity to make personal choices, and ii) define a purposive distribution of field workers to cover the whole study area. In each visited dwelling, the service level is captured through a structured questionnaire and direct observation of sanitation status and hygienic practices. In all, key features of the methodology include: i) an exhaustive identification of enumeration areas (administrative sub-units as locations, villages, barrios, etc.); ii) an audit in each enumeration area of all improved waterpoints accessed for domestic purposes; and iii) a random selection of a sample of households that is representative at the local administrative level (e.g. district, municipality, etc.) and below (Giné Garriga et al., 2012, Under review). The proposed framework, thus, makes use of two widely accepted methods, i.e. the water point mapping and the household survey, to collect WASH data in a cost-effective manner. The need for joining officers belonging to the local government is central at this stage of the process. First, they ensure a link between field workers and the local structures at community level. Second, and being the principal end-user of the outcomes produced, their involvement promotes sense of ownership over the process, as prerequisite for incorporating the data into decision-making. However, as important as promoting collaborative methods in data collection is, to foresee the viability of future data update activities. Accessibility and reliability of information should, therefore, be two core criteria when preparing the questionnaires for data collection. Development of planning tools for targeting and prioritisation To effectively improve decision-making on the basis of a reliable and sector-specific dataset, two elements are necessary (Grosh, 1997): the data must be analyzsd to produce outcomes that are relevant to the policy question and the analysis must be disseminated and transmitted to policymakers. In terms of poverty reduction, successful planning also relies on selecting beneficiaries based on real hardship. The 5 ultimate goal of local level planning is thus to target the neediest and promote equityoriented prioritisation mechanisms. With this in mind, this study first analyses baseline data as the starting point for planning. The analysis should provide a complete picture of how well the sector is faring, while enabling comprehensive understanding of key sector-related constraints to development. For this purpose, the evaluation framework needs to look beyond data on service coverage to integrate a broader view of service delivery (Jiménez and PérezFoguet, 2012, Joint Monitoring Programme, 2011). Amongst others, information about institutional, financial, management and environmental issues should be adequately addressed. However, exhaustiveness needs to be balanced with simplicity, and statistics are useful at this stage to identify a reduced but sufficient number of non-redundant indicators2. We, then, define planning criteria on the basis of such indicators, and this is done in the form of simple indices (Jiménez and Pérez-Foguet, 2010a). For each index, one ranking is produced and transposed into one league table to denote priorities. A different threshold limit is set per list for this purpose; and whenever two locations score same index value in one ranking, the most populated one is first positioned to maximise number of beneficiaries. To show at a glance both index values and priorities, different maps are developed which enable a quick identification of key focus areas. Finally, each priority list is related with specific remedial actions to be accomplished by the local government, ultimately translating development challenges into beneficial development activities. A proposed list of indices is summarised in Table 1. Again, to promote appropriation and continued use of developed planning instruments by policymakers, a consultative approach has been adopted for indices definition which imposes, amongst others, the criterion of simplicity. On the other hand, the analysis of the data often goes beyond the means and capacities of the local technicians and special effort has to be devoted to ensure that the underlying messages of the data are fully understood. 6 Table 1 Indices used for planning Index Definition Formula Threshold for prioritization Action INDICES RELATED TO WATER SERVICE COVERAGE Coverage index % of covered population by improved water points(IWP) in a location, according to the standards of service level (e.g. 1 water point / 250 people) 250* Population IWP ofNumber 25% / 50% Construction of New water points INDICES RELATED TO THE MANAGEMENT OF THE SERVICE Functionality Index % of functional improved water points (FIWP), compared to the total number of IWP 100* IWP Total IWPFunct ofNumber 50% / 75% Rehabilitation of existing water points Management Index % of FIWP with declared income and expenditure in the year before the survey 100* FIWP Total FIWPMan ofNumber 50% / 75% Management supporting activities, particularly those related to creation / establishment of water entities or to financial issues (tariff collection systems) Maintenance Index % of FIWP with good / acceptable access to technical skills and spare parts 100* FIWP Total FIWP Maintained of No. 50% / 75% Management supporting activities, particularly those related to technical issues. Improve spare parts accessibility 7 INDICES RELATED TO THE QUALITY OF THE SERVICE Seasonality Index % of FIWP that are year-round 100* FIWP Total FIWP Round-Year of No. 50% / 75% Actions to increase reliability of the source (catchment protection actions, regulation of different uses) and/or finding of additional sources Water Quality Index %of FIWP with acceptable bacteriological quality 100* FIWP Total FIWP Safe of No. 50% / 75% Actions to improve quality of water: catchment protection, protection of WP, water treatment, etc. If salinity is high and becomes dangerous, check other alternative sources WP INDICES RELATED TO SANITATION SERVICE Coverage Index % of covered households by improved sanitation facilities HH Total ISF with HH of No. 25% / 50% Construction of new facilities Open Defecation Index % of households that practice open defecation HH Total OD practicing HH of No. 50% / 25% Community-led Total Sanitation INDICES RELATED TO HYGIENE Latrine Sanitary Conditions Index % of latrines that are maintained in adequate sanitary conditions. Risky conditions might prevent an adequate use Latrines Total LatrinesSanitary of No. 25% / 50% Hygiene promotion campaigns Hand-washing index % of adults with appropriate handwashing knowledge Adults Total HW with Adults of No. 50% / 75% Hygiene promotion campaigns, particularly focused on hand-washing 8 Results and discussion This section highlights the relevance of the indices listed above from the viewpoint of policymaking. Each index may represent an emerging challenge and is thus linked to mitigation strategies that steer regional development. As regards to data exploitation and dissemination, indices are categorised based on their nature, i.e. i) water supply, and ii) sanitation and hygiene, as each category is assessed at different administrative scales. Water-related indices are computed on water point data, which offer advantages over household data in terms of statistical precision and data update routines. WPM data is exhaustive and can be meaningfully analysed at all scales; the location (Kenya), village (Tanzania) and barrio (Mozambique) have been opted for this study since they embody the last level of the institutional ladder in which planning decisions are made. In contrast, HH data is only statistically represented at division (Kenya), ward (Tanzania) and barrio (Mozambique) levels, and the analysis of sanitation and hygienerelated indices has thus been performed at this administrative scale. Adopting other territorial framework for data analysis would have implied large sample sizes, resulting in hindering the replicability of the methodology elsewhere. Water supply planning Access to water is determined primarily by distance to the source since quantity that will be collected will probably not reach a minimum requirement for domestic purposes where fetching takes more than 30 minutes (Cairncross and Feachem, 1993). Other aspects which may hinder accessibility are seasonality, quality and affordability (Howard and Bartram, 2003, Jiménez and Pérez-Foguet, 2012). Therefore, water coverage can be categorised in terms of service level, by considering a combination of aforementioned requirements. However, where optimal access is provided but the supply is not functional, other unimproved sources might become a temporary solution (Hunter et al., 2009). This draws attention to the issue of service management. Access to water The common method to estimate coverage is based on standard assumption on the number of users per water source, i.e. the source:man ratio. First index depicts the number and geographic distribution of water points in terms of the population living in the area, and thus identifies those administrative subunits most in need of new water points’ construction (Figure 1). To tackle water shortages, two different approaches can be adopted when defining list of priorities. In terms of regional equity, the goal would be to reach a minimum coverage threshold in every location. But based on an efficiency criterion, those locations with highest number of potential beneficiaries would be first targeted, regardless of coverage. From Table 2, it can be seen that one different ranking is produced depending on each of abovementioned criteria, showing both ranks poor correlation (Figure 2). The equity criterion has been opted for in this planning exercise. It emphasises those underserved locations with lowest source:man ratios, and 9 vulnerability is considered higher in total absence of improved supplies (Jiménez and Pérez-Foguet, 2010a). Figure 1: Coverage Index (Kibondo District). Figure 2: Coverage Ranks (equity versus efficiency). Functionality of water points The second group of indices aims to analyse those key aspects that enable a water scheme to remain operational over a long period of time. Lack of continuity may oblige households to search for alternative sources, often of inferior availability and poorer quality. Thus service continuity is essential in benefiting health. Functionality is defined herein as the percentage of improved sources that are functional at the time of spot-check. In those locations with lowest index values, the strategy should consider the rehabilitation of non-operational water points as an alternative to the construction of new infrastructure. In parallel, and to reduce recidivism, management and operation capacity gaps should be properly identified to promote long-term sustainability. Soft-based support initiatives to water user entities emerge as cost-effective solutions, such as promotion of their legal registration, financial and technical support to build up capacities of managers and technicians, etc. 0 10 20 30 40 010 20 30 40 Rank (efficiency) Rank (equity) 16 informed decision-making. To effectively improve decentralised planning, however, other specific challenges remain elusive, namely: 1. The continued use of these instruments in decision-making. 2. The development of appropriate updating mechanisms.  The effective appropriation of planning instruments by decision-makers is challenging in different ways (WaterAid, 2010). The approach adopted in this study - engages end-users to throughout the process, and - develops user-friendly instruments to facilitate an adequate understanding of their potential in decisionmaking, i.e. ranks, league tables and maps. Nevertheless, many other issues need to be addressed and continued support to local authorities emerges as crucial. In the short term, multi-stakeholder alliances between governments, NGOs, academics and consultants may help in the process of turning mapping into monitoring and monitoring into decision-making. In the medium term, however, political will and commitment at all levels, i.e. from central government to local authorities, are imperative to ensure that improved use of collected data results in effective propoor planning.  Ideally from the viewpoint of sustainability, the evaluation framework needs to be rethought so that it could be updated autonomously by local stakeholders or replicated elsewhere. In this regard, a major shortcoming is the trade-off between the scope and quality of the data required for decision-making support and the complexity of updating mechanisms (WaterAid, 2010). Despite successful initiatives of simple data updating based exclusively on local capacities, as one case study reported in Tanzania (Jiménez and Pérez-Foguet, 2010b), the limited capacities of local stakeholders is a principal concern. In data collection, communities can contribute to the achievement of sustainable updating mechanisms, though this should not draw attention away from the responsibilities of local authorities (WaterAid, 2011). As regards to data analysis, rankings and league tables can be easily computed through pre-programmed spread sheets, but GIS-related skills may not be easily found at local level. To ignore the need for external support may be counterproductive in the short run. From the government side, one alternative may be the establishment of regional units that provide support with data collection and data analysis. The role of NGOs, in contrast, may focus on political lobbying for behavioural change. These two challenges suggest the way forward. Notes 1 The types of water points considered as improved are consistent with those accepted internationally by the WHO/UNICEF Joint Monitoring Programme (WHO/UNICEF, 2006), where definition of improved is technology-based. 17 2 Statistical analysis has employed tools such as the Pearson's chi-square test and the Principal Component Analysis (PCA), using in both cases a standard statistical package (SPSS 15.0, 2006). 3 Sanitation technologies are considered as providing adequate access to sanitation as long as they are private (but not shared / public) and hygienically separate human faeces from human contact (improved). Based on these two requirements, sanitation coverage is presented as a four-step ladder that distinguishes between: i) open defecation; ii) unimproved sanitation; iii) shared improved sanitation; and iv) improved sanitation. Only last step is considered as “coverage” (Joint Monitoring Programme, 2008). 4 Assessment of hand-washing behaviour requires specific evaluation techniques, which were out of the scope of this study. 18 References Bennett, S., Woods, T., Liyanage, W. M. & Smith, D. L. (1991) A simplified general method for cluster-sample surveys of health in developing countries. World Health Stat Q, 44, 98-106. Breslin, E. D. (2010) Rethinking Hydro‐Philanthropy: Smart Money for Transformative Impact. Denver, Water for People. Cairncross, S. & Feachem, R. G. (1993) Environmental Health Engineering in the Tropics: An Introductory Text, Chichester, John Wiley & Sons. Cairncross, S., Hunt, C., Boisson, S., Bostoen, K., Curtis, V., Fung, I. C. & Schmidt, W. P. (2010) Water, sanitation and hygiene for the prevention of diarrhoea. Int J Epidemiol, 39 Suppl 1, i193-205. Cortinovis, I., Vella, V. & Ndiku, J. (1993) Construction of a socio-economic index to facilitate analysis of health data in developing countries. Social Science & Medicine, 36, 1087-1097. Curtis, V. & Cairncross, S. (2003) Effect of washing hands with soap on diarrhoea risk in the community: a systematic review. The Lancet Infectious Diseases, 3, 275-281. Esrey, S. A., Potash, J. B., Roberts, L. & Shiff, C. (1991) Effects of improved water supply and sanitation on ascariasis, diarrhoea, dracunculiasis, hookworm infection, schistosomiasis, and trachoma. Bull World Health Organ, 69, 609-21. Giné Garriga, R., Jiménez Fernández De Palencia, A. & Pérez Foguet, A. (2012, Under review) Water - Sanitation - Hygiene Mapping: an integrated approach for data collection. Science of the Total Environment. Giné Garriga, R. & Pérez Foguet, A. (2012, Under review) Water, sanitation, hygiene and rural poverty: issues of sector planning and the role of aggregated indicators. Water Policy. Grosh, M. E. (1997) The policymaking uses of multitopic household survey data: A primer. World Bank Research Observer, 12, 137-160. Howard, G. (2002) Water quality surveillance: a practical guide. Leicestershire, Water, Engineering and Development Centre. Loughborough University. Howard, G. & Bartram, J. (2003) Domestic Water Quantity, Service Level and Health. Geneva, World Health Organization. Hunter, P. R., Zmirou-Navier, D. & Hartemann, P. (2009) Estimating the impact on health of poor reliability of drinking water interventions in developing countries. Science of the Total Environment, 407, 2621-2624. 19 Jiménez, A. & Perez-Foguet, A. (2011) The relationship between technology and functionality of rural water points: evidence from Tanzania. Water Science and Technology, 63, 948-955. Jiménez, A. & Pérez-Foguet, A. (2010a) Building the role of local government authorities towards the achievement of the human right to water in rural Tanzania. Natural Resources Forum, 34, 93-105. Jiménez, A. & Pérez-Foguet, A. (2010b) Challenges for Water Governance in Rural Water Supply: Lessons Learned from Tanzania. International Journal of Water Resources Development, 26, 235 - 248. Jiménez, A. & Pérez-Foguet, A. (2012) Quality and year-round availability of water delivered by improved water points in rural Tanzania: effects on coverage. Water Policy, 14, 509-523. Joint Monitoring Programme (2008) Progress on Drinking Water and Sanitation: Special Focus on Sanitation. Joint Monitoring Programme for Water Supply and Sanitation. Geneva / New York, WHO / UNICEF. Joint Monitoring Programme (2011) Report of the First Consultation on Post-2015 Monitoring of Drinking-Water and Sanitation. Geneva / New York, WHO / UNICEF Joint Monitoring Programme for Water Supply and Sanitation (JMP). Joint Monitoring Programme (2012) Progress on Drinking Water and Sanitation: 2012 Update. Joint Monitoring Programme for Water Supply and Sanitation. Geneva / New York, WHO / UNICEF. Lemeshow, S. & Stroh, G. (1988) Sampling Techniques for Evaluating Health Parameters in Developing Countries. Washington, D.C., National Research Council. Scott, R., Cotton, A. P. & Govindan, B. (2003) Sanitation and the Poor. Loughborough, London & Delft WELL Resource Centre (WEDC, LSHTM & IRC). United Nations Children’s Fund (2006) Multiple Indicator Cluster Survey Manual 2005. New York, UNICEF, Division of Policy and Planning. WaterAid (2010) Water point mapping in East Africa. Based on a strategic review of Ethiopia, Tanzania, Kenya and Uganda. London, WaterAid. WaterAid (2011) Sustainability framework. London, WaterAid. WaterAid & ODI (2005) Learning for Advocacy and Good Practice – WaterAid Water Point Mapping. London, Overseas Development Institute. WHO/UNICEF (2006) Core questions on drinking-water and sanitation for household surveys. Geneva WHO / UNICEF.