Methodological guidance to assess the value for money of premium and capital support towards climate and disaster risk finance and insurance

Ward, John; Weingärtner, Lena; Panwar, Vikrant Research Report Methodological guidance to assess the value for money of premium and capital support towards climate and disaster risk finance and insurance Advisory report Provided in Cooperation with: ODI Global, London Suggested Citation: Ward, John; Weingärtner, Lena; Panwar, Vikrant (2022) : Methodological guidance to assess the value for money of premium and capital support towards climate and disaster risk finance and insurance, Advisory report, Overseas Development Institute (ODI), London This Version is available at: https://hdl.handle.net/10419/280299 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by-nc-nd/4.0/ Advisory report Methodological guidance to assess the value for money of premium and capital support towards climate and disaster risk finance and insurance John Ward, Lena Weingärtner and Vikrant Panwar December 2022 Disclaimer: This advisory report has received financial support from Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH through the InsuResilience Global Partnership (IGP) secretariat. The views expressed do not necessarily reflect the official policies of GIZ or IGP. Readers are encouraged to reproduce material for their own publications, as long as they are not being sold commercially. ODI requests due acknowledgement and a copy of the publication. For online use, we ask readers to link to the original resource on the ODI website. The views presented in this paper are those of the authors and do not necessarily represent the views of ODI or our partners. This work is licensed under CC BY-NC-ND 4.0. How to cite: Ward, J., Weingärtner, L. and Panwar, V. (2022) Methodological guidance to assess the value for money of premium and capital support towards climate and disaster risk finance and insurance. Advisory report. ODI and InsuResilience Global Partnership. London: ODI (www.odi.org) Acknowledgements The authors would like to thank members of the advisory working group, Annette Detken, Daniel Clarke, Nicola Ranger, Olivier Mahul and colleagues at the IGP secretariat particularly Daniel Stadtmüller, Janek Töpper and Kay Tuschen for their insights, inputs and guidance. We are grateful to the interviewees who participated in the key informant interviews (KIIs) on the political economy of premium subsidies and provided critical inputs for the development this advisory report. At ODI, we are grateful to Silvia Harvey. This study has received financial support from Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH on behalf of the InsuResilience Global Partnership (IGP) secretariat. The views and findings presented here are of the authors’ and do not represent the views of GIZ or IGP. About this publication This advisory report is an output of the Global Risks and Resilience Programme (GRR) at ODI. GRR provides rigorous analysis of multiple interconnected risks, interrogates narratives and risk perceptions, and uses this evidence to recommend tailored solutions for the management of systemic risks in development, humanitarian, climate adaptation and disaster risk management policies and actions. About the authors John Ward An economic and strategy expert working at the intersection between climate, disaster risk and development policy. He has more than 20 years’ experience in economic strategy and analysis. His work involves both supporting companies, governments, international financial institutions, philanthropies and donors in developing their strategic approaches to climate action and disaster risk management and finance. Lena Weingärtner Research Associate with ODI’s Global Risks and Resilience programme, working primarily on disaster risk management and financing. She has experience in researching and informing financial instruments and delivery mechanisms at micro, meso and sovereign levels. This includes approaches – such as parametric insurance, adaptive social protection, or anticipatory action. Vikrant Panwar Senior Researcher with ODI’s Global Risks and Resilience programme. An economist with specialisation of macro-fiscal disaster and climate risks and impacts. He conducts research around disaster and climate risk financing, macro-fiscal climate and disaster impacts at the sovereign and sub-sovereign levels. Email: v.panw[email protected].uk ORCID: https://orcid.org/0000-0003-1259-9789 Contents Acronyms / iii Excecutive Summary / 1 Background / 3 When to use this guidance document / 4 Who should use this guidance document / 5 How to use this guidance document / 6 Conventional approaches for assessing VfM / 7 What we know from previous studies about analysing the VfM of PCS towards CDRFI / 8 SMART PCS approach to assessing VfM / 12 Numerator: Weighted combination of outcomes from PCS towards CDRFI (value) / 12 Denominator: Funding provided (costs) / 20 Interpreting results for informed funding decisions / 21 References / 22 Annex: Potential criteria for use in Multi-criteria analysis (MCA) / 24 Acronyms ADB Asian Development Bank ADF African Development Fund, AfDB ADRiFi African Disaster Risk Financing Programme AfDB African Development Bank ARC African Risk Capacity CATDDO Catastrophe Deferred Drawdown Option CCRIF Caribbean Catastrophe Risk Insurance Facility CDRFI Climate and Disaster Risk Finance and Insurance CEO Chief Executive Officer DRF disaster risk finance FCDO Foreign, Commonwealth and Development Office, UK IDA International Development Association, World Bank IGP InsuResilience Global Partnership KII key informant interview MoF Ministry of Finance M&E monitoring and evaluation PCRIC Pacific Catastrophe Risk Insurance Company PCS premium and capital support PEA political economy analysis PIC Pacific Island country TWG technical working group UK United Kingdom 1ODI Advisory report Excecutive Summary In 2021, the InsuResilience Global Partnership developed a set of SMART Principles for the purposes of guiding the design and implementation of appropriate premium and capital support (PCS) that could help scale up climate and disaster risk finance and insurance (CDRFI). One of the five principles, ‘Value for Money’ (VfM) describes the impact each dollar of premium and capital support has on the resilience of poor and vulnerable countries and people (Töpper and Stadtmüller, 2022). This guidance note contributes to the practical implementation of the VfM principle. Aiming to inform allocation decisions, it provides a framework and methodology for the ex-ante assessment of the VfM of PCS options. This includes allowing decision-makers to compare premium versus capital support towards CDRFI, synthesising the effects of the different support options within one country, or of the same option across different countries. The SMART PCS approach to VfM proposed here presents a middle way between the two conventional cost-effectiveness and cost–benefit analysis approaches, as it measures the cost of delivering a synthetic multi-dimensional set of outcomes: VfMi = value i (measured as weighted combination of additional CDRFI outcomes resulting from subsidy) (moneyi (measured as cost of funding provided)) This metric is similar to a cost-effectiveness metric in the sense that the outcomes are expressed in non-monetary terms: for instance, number of people covered. At the same time, it is similar to a cost–benefit analysis metric, as it recognises that an expansion of a CDRFI scheme that is supported by PCS delivers multiple outputs and outcomes of value and that these need to be aggregated in some way. To quantify the ‘value’ component of the equation, this guidance note proposes a five-step process: 1. Pre-screen CDRFI scheme 5. Aggregate scores and weights 2. Determine criteria 3. Design scoring methodology 4. Weight criteria The ‘money’ part of the equation represents the grant equivalent of donor funding towards PCS. This means that the resulting assessment is not an assessment of overall benefits and costs to society, but rather of the benefits derived from each euro or dollar of donor spending. The approach to assessing VfM proposed in this guidance note requires a relatively large amount of judgement. Therefore, it is important that the analysis is conducted by an impartial party so that it can be truly transparent and comparable, and that outputs from the analysis are peerreviewed by suitably qualified people with relevant experience, expertise and local knowledge. 2ODI Advisory report The results from applying this approach to assessing the value for money of PCS are synthetic and do not have a ‘real-world’ interpretation. This means that using the approach for funding decisions would need to involve setting thresholds to give meaning to the scored outcomes – i.e. final scores above and below given thresholds need to be associated with clear next steps as to which PCS option should proceed or not proceed, or should otherwise indicate that further assessment and discussion is required. The specific cut-off points for these decisions will need to be determined in a next step of developing and rolling out the approach presented in this guidance note. This should involve testing and calibration – e.g. by applying the approach to a sample of past PCS appraisals (where information is available) or by piloting it on upcoming appraisals, alongside the existing criteria the funding entity has been using. Such testing should include projects which were approved, as well as some that were rejected, on the basis of the funding entity’s criteria at the time. Background In 2021, the InsuResilience Global Partnership (IGP) developed the SMART PCS Principles to guide the design and implementation of Premium and Capital Support (PCS) to support the scaleup of climate and disaster risk finance insurance (CDRFI) solutions (Töpper and Stadtmüller, 2022). One of the five principles, ‘Value for Money’, requires that each dollar of PCS should ‘support needs-based CDRFI products that add value … and requires the development of … a clear assessment framework that makes improvements in resilience verifiable and comparable’ (ibid.: 8). Value for Money, according to the principles, is defined as ‘the expected impact on poor and vulnerable countries’ and people’s resilience for each dollar of premium or capital support’. The principles also highlight that the value proposition of PCS should include crowdingin, rather than undermining, private capital, ‘recognizing the key role that effective private insurance markets can play in resilience-building of developing economies’ (ibid.). This guidance note contributes to the practical implementation of the SMART Principles Value for Money approach. It does so by proposing a framework for the ex ante assessment and comparison of different PCS options, aiming to inform and support decision-makers. The guidance note is based on, and aligned with, the SMART Principles, the IGP’s monitoring and evaluation framework (IGP, 2021), and IGP pro-poor principles (IGP, 2019). 10 ODI Advisory report • further progress is made in using additional indicators to complement or verify weatherbased indices so that the degree to which countries can rely on ARC in extreme years is increased; • ARC acts as catastrophe insurance for the government’s contingent liability, and other instruments are used for regular, smaller losses; and • the facility pays out less frequently and retains more risk. (ibid.: 3) This ex-ante analysis was followed by an updated CBA after several years of implementation in 2020 (Kramer et al., 2020), as well as an additional CBA and a further VfM analysis carried out as part of a larger ARC impact evaluation in 2022 (OPM, ongoing and unpublished). Over time, these assessments continuously refined methodologies, criteria and assumptions on the basis of ex-post observed benefits and costs of the scheme. The most recent published analysis (Kramer et al., 2020) puts the ex-ante CBA findings into perspective; still estimating a positive ratio, but one that is below the $1.90 potential outlined by Clarke and Hill (2013). This is mainly because the premium rates assumed in the ex-ante analysis were lower than they turned out to be in practice. Furthermore, countries mainly used ARC payouts for food aid, rather than channelling them through existing state-contingent welfare schemes. As a result, the speed, cost and targeting gains have not been as large as initially assumed (Kramer et al., 2020). This experience highlights the challenges of establishing criteria and assumptions in an exante scenario, where the details of the CDRFI instrument itself are still being worked out. This is especially the case in a CBA setting when a number of the inputs needed to undertake the calculations are very difficult to know or observe; for example, the extent of targeting of payouts to households of different incomes, or the marginal utility of income for households with different incomes. This raises the possibility that if this technique is used to help make decisions regarding the allocation of PCS between different schemes, as well as being labour-intensive, the resulting prioritisation may be driven as much by analysts making different assumptions about key methodological inputs as it is by intrinsic differences between schemes. This suggests that this sort of analysis may be better suited to the assessment of an individual scheme in which stakeholders want to understand whether it will offer (or has offered) value for money and to calibrate the design in order to maximise that value for money over time. In this case, close engagement with stakeholders, alongside the use of independent experts, can help ensure the analysis delivers useful insights. A further analysis, looking explicitly at the difference between premium subsidy and capital support, was undertaken by the UK’s Government Actuaries Department. It used a cost-effectiveness approach to compare the effects of a £1 premium subsidy versus a £1 capital injection on the expected cumulative discounted premium that members of a risk pool would have to pay. Under the specified assumptions (summarised in Box 1 in Vivid Economics et al., 2016), a premium subsidy would result in an expected cumulative discounted premium reduction that is 69% higher than what it would be for an additional capital injection of the same amount. However, the authors also caution that the assumptions made in the analysis – e.g. on the discount rate, the multiple for re-insurance, or the risk pool capital base – are generally realistic but generic, and would need to be adapted to programme specifications to inform actual donor 11 ODI Advisory report decisions between capital injections and premium subsidies in practice (Vivid Economics et al., 2016, referencing Government Actuary’s Department, 2016). While this approach sheds light on the relative cost of different PCS options, the focus is on comparing the effectiveness of capital support versus premium subsidy in the context of a specific scheme. However, it does not provide a means of assessing the overall value of that scheme, or how the value of support for one scheme might be higher or lower than the value of support for a different scheme. 12 ODI Advisory report SMART PCS approach to assessing VfM On the basis of the review and discussion of advantages and limitations of different methodologies, the approach to assessing VfM of PCS towards CDRFI proposed in this guidance note presents a middle way between the two conventional cost-effectiveness and cost–benefit analysis approaches, as it measures the cost of delivering a synthetic multi-dimensional set of outcomes: This metric is similar to a cost-effectiveness metric, as the outcomes are expressed in non-monetary terms; for instance, number of people covered. At the same time, it is also similar to a cost–benefit analysis metric, as it recognises that (a PCSsupported expansion of) CDRFI delivers multiple different outputs and outcomes of value and that these need to be aggregated in some way. This hybrid approach has some similarities to health literature, where interventions are measured in terms of disability-adjusted life years (DALYs), which requires users to aggregate and weight two different outcomes: the number of life years that the medical intervention provides AND the quality of those additional life years. Numerator: Weighted combination of outcomes from PCS towards CDRFI (value) The criteria to be included in the numerator of the above equation can be defined through multicriteria analysis (see Box 1), following five steps: 1. Pre-screen CDRFI scheme 5. Aggregate scores and weights 2. Determine criteria 3. Design scoring methodology 4. Weight criteria While this guidance note provides a common framework and approach for assessing VfM of PCS towards CDRFI, this five-step process entails some flexibility to customise and weight criteria. This is important to ensure that the analysis is appropriately based on context, and that it can be fit for the specific purpose of the VfM analysis – e.g. whether the aim is to compare potential PCS allocations across countries, or select between different PCS options within a country. 13 ODI Advisory report Box 1 Multi-criteria analysis (MCA) Multi-criteria analysis is frequently used in appraisals when it is not considered possible or appropriate to place monetary values on the outcomes delivered. It involves scoring an intervention against a range of criteria that capture dimensions of value (expected outcomes) and then weighting those scores to allow comparison across interventions. This means that multi-criteria analysis is very suitable to the objectives and limitations of VfM analysis under the SMART PCS principles, where the assessment is conducted ex ante with limited information and time, where the main outcomes of interest (‘improvements in resilience’ (Töpper and Stadtmüller, 2022)) are difficult to express in monetary values, and where some flexibility is required to account for differences in context. The multi-criteria analysis approach has a number of advantages and disadvantages, which are summarised in Table 2. Often, cost/cost-effectiveness is simply used as one of the criteria in the assessment, but it is also possible, as the SMART PCS principles propose, to assess interventions against ‘positive’ dimensions of value and then divide by costs. Table 2 Advantages and disadvantages of using multi-criteria analysis for PCS of CDRFI Advantages Disadvantages • Provides a way of prioritising interventions • Allows for trade-offs: weak performance on one criterion can be offset by strong performance on another • Flexibility in design means method can be tailored to context while remaining transparent • Opportunities for participation to support assessment • Provides a way of incorporating evidence that may be difficult to quantify • Interventions must be ‘sufficiently’ comparable so that they can be scored (implies that the approach is better for intra-CDRFI comparison than comparing CDRFI with other interventions) • Only provides a relative assessment, not an ‘absolute’ assessment of whether any of the projects should proceed • Subjectivity of scoring and weighting can be high, leading to difficulty in generating consistent scores STEP 1: Pre-screen CDRFI scheme The VfM assessment necessarily focuses on the incremental value resulting from the provision of subsidy and compares this against the incremental costs of providing subsidy. However, there are a number of design considerations related to CDRFI schemes that will affect the overall value that the scheme is able to provide, but which are unlikely to be influenced by the provision of PCS. To deal with this challenge, it is recommended that a series of screening criteria are used to help exclude poorly designed schemes from benefiting from PCS. This can help to ensure that the incremental value created by the provision of PCS is realised in 14 ODI Advisory report the context of CDRFI schemes that are robustly designed. The key criteria used for this prescreening should include:7 • evidence that the scheme is likely to result in benefits for the poorest and most climate-vulnerable • evidence that the scheme will finance timely response • evidence that the scheme has been designed in a way that takes account of the risk context –and aligns with the bigger picture of how risks are managed and how resilience is strengthened in the country – such that it focuses on the most important risks and complements other risk management and risk finance measures • evidence that those targeted by the scheme and other key stakeholders have been consulted in the design of the scheme, and that the scheme creates power for people facing risk • evidence that, where parametric or other triggers are used, the extent of possible basis risk has been assessed and efforts taken to minimise this risk, so that the scheme provides reliable protection • evidence that the system is set up to learn and improve • evidence that the scheme itself offers good value and, in particular, is not reducing emphasis on investments in risk reduction where these are cost effective. STEP 2: Determine criteria The following five factors are critical to consider when determining which criteria to include in the numerator for analysis of the VfM of PCS, i.e. the indicators that constitute ‘value’: 7 These criteria are aligned with the IGP’s pro-poor principles (IGP, 2019) and follow the 7 keys of highly effective disaster risk finance that have been proposed by the Centre for Disaster Protection (Scott and Hill, 2020). 1. Completeness: Criteria should capture all outcomes that are considered to be of value when deciding upon supporting a CDRFI intervention through PCS. 2. Avoid redundancy: Exclude criteria that are not considered important or where it is likely that all possible PCS interventions will achieve the same score. 3. Operational: Criteria must be capable of being assessed; multi-criteria analysis can accommodate both quantitative and qualitative criteria, but the operational factor may make the assessment of indirect or secondary benefits challenging. 4. Preference independence: Only include outcomes that are valued intrinsically and not because they are a means to supporting other outcomes (e.g. is leveraging private capital an outcome that is valued for itself, or is it only important because it will allow greater penetration or help achieve other outcomes?). 5. Number of criteria: Criteria must be manageable and easy to communicate. Further considerations in determining which criteria should make up the numerator of the VfM analysis include whether the benefits of using set and standardised criteria are more important than the flexibility of being able to add or alter criteria to context in the assessment. The former approach may be preferred in a situation where the aim is to understand what the relative VfM of an insurance premium subsidy to country A would be, compared to allocating the same amount towards premium subsidies in country B and country C. More flexibility to adapt criteria to context, on the other hand, could be preferred when assessing whether a premium subsidy to country A provides more or less VfM than 15 ODI Advisory report allocating the same amount towards other types of PCS. These trade-offs between comparability and context-specificity should be discussed between stakeholders, and the approach determined accordingly, in the early stages of the VfM analysis. As far as possible, the criteria should capture the intended development outcomes from expanding CDRFI products, as it is these outcomes that are ultimately of value. This consideration suggests that criteria linked to interim outcomes that are only important because they enable intended development outcomes, but do not have intrinsic value – such as (for example) affordability – may not be appropriate. Assuming increased penetration is included, affordability would also be unlikely to satisfy the requirement for preferenceindependence. The criteria considered to be ‘important’ and ‘of value’ can be highly subjective. The SMART PCS policy note, along with the IGP M&E framework and the IGP pro-poor principles, can guide these considerations through: 1. consideration of the five factors identified above 2. assessing consistency with the IGP M&E framework and pro-poor principles, 3. identification of the outcomes from CDRFI solutions frequently cited in the literature, and 4. considering those outcomes that can be plausibly influenced by the provision of different types of PCS. Some of the criteria that are most likely to be relevant are: 8 Experience of payout has been found to increase likelihood of purchasing insurance in the future. For further discussion of this relationship and evidence from a macro CDRFI scheme (ARC), see Scott et al. (forthcoming and OPM (forthcoming). • the projected increase in the number of beneficiaries • the projected contribution to reduction of protection gap • the extent to which subsidy design contributes to sustainability of the insurance product, incorporating considerations of payout frequency, which is a strong predictor of future purchase, possibility of crowding out private capital, and other measures of sustainability which have a robust evidence base.8 These criteria reflect some of the primary motivations that different stakeholders have when providing PCS (criterion 1 above) and are likely to be relatively easy to assess in a wide range of different contexts (criterion 3). They are also largely preference independent (criterion 4). They are also criteria that can be applied both to cases where the support is being provided as a premium subsidy, as is relatively clear, but also when the support takes the form of a capital injection (Box 2). However, ultimately, stakeholders should choose criteria that align well with the decision that they are seeking to make at a particular point in time. In this regard, they may wish to refer to the Table in the Annex which provides a longer list of potential criteria (or sub-criteria) derived from: (1) criteria proposed in the initial SMART PCS policy note; (2) criteria typically included in other assessments of VfM of PCS/CDRFI in the literature; and (3) criteria identified through conversations with different stakeholders (including CDRFI-implementing countries, CDRFI operators and donors) in the form of key informant interviews and advisory group meetings conducted as part of developing 16 ODI Advisory report this guidance document. This Table also assesses the performance of these criteria against the five factors identified above – although, as per point (iv) above, further scrutiny of the criteria in the Table in the Annex would be required on a case-bycase basis to ensure that they could inform intraCDRFI decisions. Box 2 Relevance of potential criteria to the provision of capital support The provision of additional capital to a CDRFI scheme can have a number of different objectives, including: (i) allowing the scheme to cover more risks/write more policies; (ii) allowing a sustained reduction in premia; (iii) allowing the scheme to make larger payouts without the risk of insolvency. Each € of capital support provided could only be used for one of these purposes, but a large enough capital injection could be used to support a combination of these objectives. Depending on the way the capital was used, any one or all three of the criteria identified above might be affected. Capital provided to support scheme expansion could allow an increase in the number of beneficiaries and/or a reduction in the protection gap, e.g. if a greater number of perils were covered. However, the credibility of any projections would need to be assessed carefully. Moreover, using capital in this way may raise questions regarding sustainability, if there was a possibility that the donor-provided capital could crowd out private capital. Using a capital injection to sustain premium reductions for a macro CDRFI product would not lead to an increase in the number of beneficiaries or to the protection gap being closed. However, it could promote sustainability if the premia reduction meant that the recipient was more likely to (continue to) purchase the CDRFI instrument into the medium term. Capital to support scheme solvency could enhance the sustainability of the product, although there would need to be confidence that the capital would adequately address any underlying challenges that had led to the solvency concerns in the first instance. In all of these cases, the mechanisms through which the capital injection would lead to these and/ or other impacts would need to be assessed carefully, by a credible, independent party, taking into account the current strength of the evidence base, as discussed in the section headed ‘Who should use this guidance document?’ STEP 3: Design scoring methodology As part of this third step, a scoring methodology is designed that will facilitate the assignment of scores against different quantitative and/or qualitative criteria that have been selected in the previous step. To ensure good decision-making, it is essential that the scoring focuses on the differences between a CDRFI scheme with and 17 ODI Advisory report without the provision of PCS. This ensures that the scoring only captures the additional value that the PCS provides. In order for final assessment to be meaningful, each criterion needs to be scored on a standard metric. Often, in multi-criteria analysis, scores are done on a 0–5 range, but a wider range (e.g. 0–10 or 0–100) can provide practitioners with more flexibility and add greater nuance to the scoring. This is the case especially as the absolute difference between scores is meaningful; i.e. on a given criterion, moving from a score of 2 to a score of 4 should be only half as valuable as moving from a score of 2 to a score of 6. The illustrative example in Figure 1 uses a range of 0–100, which has the optical appeal that the numerator will likely be larger than the denominator, meaning that the resulting ratio will usually exceed 1 (although, as stressed below, the ratio has no intrinsic meaning). In this example, if the global maximum number of additional beneficiaries per intervention is 100 million, then a project that supports an additional 4 million would receive a score of 4; a project supporting an additional 12 million people would receive a score of 12; and a project supporting an additional 30 million people would receive a score of 30. Figure 1 Illustrative example of scoring for a criterion on number of additional poor and vulnerable beneficiaries covered by allocating PCS towards CDRFI Note: While this analysis assumes a linear relationship between number of beneficiaries and score, it would be possible to assume a non-linear relationship between performance and score, where this reflects underlying values/ preferences. In the case of this guidance note, a range of 0–10 or 0–100 is proposed to ensure sufficient flexibility for the potential range of scoring values of the different criteria considered above (see STEP 2). If most indicators included are of quantitative nature, a scale of 0–100 is preferable 80 60 40 20 020 million 40 million 60 million 80 million 100 million 100 18 ODI Advisory report to allow for greater nuance, whereas a scale of 0–10 is more appropriate if most indicators are qualitative, as scorecards are easier to develop and apply for a 0–10 range, rather than 0–100. Once the range has been determined, practitioners should determine what corresponds to best score (i.e. 10 or 100, depending on the scale) and worst score (0). There are two options that can be used at this stage: • Local perspective: consider the best and worst performance on each criterion among the interventions currently under appraisal • Global perspective: consider the best and worst performance, on each criterion, that is ever likely to be achieved. For instance, for the number of additional poor and vulnerable beneficiaries set a score of 10 or 100 for 100 million (assuming no intervention will achieve more than 20% of IGP’s target) and 0 for no additional beneficiaries (see Figure 1). It is recommended here that the global perspective is used for assessing the VfM of PCS towards CDRFI under the SMART PCS framework, as this will allow comparison of projects over time and across countries and thus aligns best with SMART PCS implementation objectives. Finally, once maximum and minimum values have been determined, then scores can be identified for each criterion. For quantitative criteria, the score can reflect how far the expected quantity is from pre-specified high and low points. For qualitative criteria, judgement will be required. Developing scorecards for what justifies a particular score for 9 For examples, and discussion of the use of scorecards in VfM assessments, see (for instance) Tables 1 and 2 in King (2018). These examples use traffic light systems, a 1–4 point scale, or a 1–5 point scale scoring against different criteria. More refinement and nuance would be possible – and calculation of value against cost facilitated – if similar scorecards were developed on a 1–10 scale, as suggested in this guidance document. each criterion will help increase transparency in the scoring. Furthermore, participatory processes (e.g. consulting stakeholders through surveys, key informant interviews or focus group discussions) can support the scoring process. The assessment and scoring should reflect the expected impact of the provision of PCS towards CDRFI over the lifetime of that support (and, potentially, beyond). Suggested ranges and scorecards should be developed, ideally on the basis of a participatory approach.9 It is important to note that these ranges and scorecards are initially only indicative. In a next phase – not included in the current project – their application and the scoring would need to be tested, and the scoring methodology refined, before they are recommended for use in VfM of PCS assessments that inform intra-CDRFI comparisons and decision-making over PCS allocations. As stressed in the section ‘Who should use this guidance document?’, it is essential that the scoring is undertaken by an impartial third party and subject to peer review. STEP 4: Weight criteria Weights are important in the SMART PCS VfM approach, because they can help factor priorities and principles into the VfM assessment. For instance, small island developing states (SIDS) may be particularly vulnerable to disasters and eligible for PCS, but using a criterion relating to the projected number of additional beneficiaries 19 ODI Advisory report covered by PCS in the analysis may result in a relatively low estimated VfM for SIDS, due to their small population size. In such cases, weights could be used to ensure that SIDS are not disadvantaged in VfM comparisons (Töpper and Stadtmüller, 2022). To determine weights, practitioners need to ask: ‘How much do we value a swing of 0–100 on criterion ‘x’ compared to criterion ‘y’?’ This ensures that, if two criteria are given the same weight, the same incremental change in the score on each criterion has the same impact on the overall outcome of the assessment. For this reason, it is important to only set the weights after: • the minimum and maximum scores are determined (if using a global scoring approach), or • scoring has been undertaken (if using a local scoring approach). Typically, weights will be set so that they sum to 100%, but other approaches are valid. As for the scoring approach, it is possible to use participatory 10 Examples of some of the challenges in using a linear additive model are provided in Tofallis (2014). approaches in the process of determining weights. IGP could consider identifying indicative weights but providing flexibility for local users to change weights according to local contexts. Potential trade-offs between comparability and flexibility will need ato be considered in this decision. STEP 5: Aggregate scores and weights There are generally two main models available to aggregate scores and weights: the linear additive model and the weighted product model (summarised in Table 3). Although less common, we recommend that the weighted product model approach is taken. This is because the linear additive model is very sensitive to the approach taken to normalise scores which are measured on different scales. The linear additive model also raises the possibility that an ‘extreme’ score on one criterion could allow a particular PCS to be preferred over a PCS that scores well on three different criteria of interest.10 The use of the weighted product model overcomes some of these problems. Table 3 Available models for aggregating scores and weights Linear additive model Weighted product model • Each score is multiplied by weight, weighted scores added together and then divided by cost: (s1 * w1) + (s2 * w2) + (s3 * w3) cost, €m • This approach is the most typically used and probably easiest to understand • However, these calculations can be very sensitive to the weights • Weights are reflected as powers and the weighed criteria are then multiplied: (s1 w1) * (s2 w2) * (s3 w3) cost, €m • This approach is less sensitive to the weights selected 26 ODI Advisory report Proposed value criteria Source Completeness Avoid redundancy Operational Preference independence Increased affordability of CDRI products by reducing the cost of the insurance premium Panda et al. (2021) Aligned with IGP M&E framework (low cost of providing coverage indicator), SMART PCS principles and pro-poor framework Potential redundancy with expected cumulative discounted premium reduction indicator Yes; e.g. as per methodology proposed in GAD (2016), but sensitive to assumptions No, if reduced premium cost or outcomes about increased coverage are included Developed new markets to boost initial demand for insurance to reduce disaster vulnerability Panda et al. (2021) Not clearly aligned with IGP principles Potential redundancy with performance in attracting private capital indicator Difficult to project due to dependence on other factors/ assumptions No: valued because it contributes to other objectives (e.g. reducing protection gap, sustainability) Promoted higher insurance penetration coverage Panda et al. (2021) Aligned with IGP targets and critical for funding decisions Potential redundancy with protection gap and coverage criteria Yes: can draw on IGP M&E framework methodology Yes, IGP target Reduced (implicit) contingent liability of the government Panda et al. (2021) Not explicitly aligned with IGP principles No redundancy Yes, based on product parameters (coverage, return period…) and risk profile, country CDRFI strategy and/or risk register if available Yes, independent from other criteria considered Expected cumulative discounted premium reduction UK Government Actuary‘s Department (2016) Aligned with IGP M&E framework (low cost of providing coverage indicator), SMART PCS principles and pro-poor framework Potential redundancy with increased affordability criterion Yes; e.g. as per methodology as proposed in GAD (2016), but sensitive to assumptions No, if reduced premium cost or outcomes about increased coverage are included 27 ODI Advisory report Proposed value criteria Source Completeness Avoid redundancy Operational Preference independence Confidence that designed scheme will provide support when needed (basis risk) Advisory group consultations Aligned with IGP pro-poor quality principle No redundancy Perceived confidence could be based on stakeholder consultation; objective accuracy would require basis risk/quality assessment, so likely only feasible where this is already available, or data is available to assess Yes Transparency Advisory group consultations Aligned with SMART PCS principles No redundancy Could involve qualitative judgement, based on stakeholder consultation Yes, PCS principle Long-term sustainability Advisory group consultations Aligned with SMART PCS principles No redundancy Maybe, possibly difficult to assess Government willingness to take on future premiums? Yes, PCS principle Consistency of provision of subsidy with risklayering principles Advisory group consultations Aligned with IGP M&E indicator (adoption of a comprehensive DRF strategy) Potential redundancy with improved risk financing criterion Yes, could include rating based on assessment of DRF context No, if outcome indicators of prevented asset loss, malnutrition, loss of life and food security are included 28 ODI Advisory report Proposed value criteria Source Completeness Avoid redundancy Operational Preference independence Impact of the subsidy on risk taking (moral hazard) and/or risk reduction Advisory group consultations Aligned with SMART PCS resilience and sustainability principles No redundancy Could be included as qualitative indicator if PCS provision contingent on risk taking or risk reduction, otherwise difficult to establish Yes Reduced opportunity cost to government Key informant interviews (risk pool member country) Aligned with IGP M&E indicator (efficacy in support of vulnerable countries) Potential redundancy with two criteria: expected cumulative discounted premium reduction and increased affordability of CDRI products by reducing the cost of the insurance premium Yes, based on value of premium subsidy Yes, considering climate change attribution perspective Reduced disaster response cost/ reduced cost of humanitarian response to affected government and donors Key informant interviews (risk pool and risk pool member countries) Aligned with IGP M&E indicator (efficacy in support of vulnerable countries) No redundancy Yes; could build on economics of resilience and early action methodologies, but if cost reductions beyond the value of the payout are based on assumptions related to speed, similar caveats to that of the improved response speed criteria apply No, if outcome indicators of prevented asset loss, malnutrition, loss of life and food security are included 29 ODI Advisory report Proposed value criteria Source Completeness Avoid redundancy Operational Preference independence Increased autonomy by governments to choose coverage and handle payouts† Key informant interview (technical partners) Aligned with SMART PCS principles and supports wider DRR and development targets No redundancy Could involve qualitative judgement, based on stakeholder consultation Yes Enhanced risk ownership through greater risk awareness and assessment Key informant interview (technical partners) Aligned with SMART PCS principles and supports wider DRR and development targets Potential redundancy with impact of the subsidy on risk taking (moral hazard) and/or risk reduction criterion Could be included as qualitative indicator if PCS provision contingent on risk taking or risk reduction, otherwise difficult to establish Yes * ‘All parties paying for pre-arranged financing should have access to adequate information and appropriate financial advice to assess value for money, impact and any risks of the product relative to expectations and needs of the client and relative to other potential feasible options that could be taken to achieve the stated objectives. This will be assessed in the context of the broader disaster risk financing and insurance strategy. Products should be priced based on sound actuarial principles that adequately account for the underlying risks and operating expenses. Any financial advice is delivered with the highest standards of integrity, impartiality, competence, and care.’ (www.globalriskfinancing.org/resource/grif-operations-manual: 33) † However, it should be noted that one risk pool member country highlighted the empowerment they felt from paying the premium themselves – so while subsidies could have an advantage over capital support in a country’s level of autonomy, they also risk undermining ownership and empowerment for some countries. 30 ODI Advisory report